JMeter – Simplistic tests and “closed system” testing

Over the last few years I have used JMeter for some small web performance tests but only recently have I been able to really get to grips with it and the subject of performance testing. As I started learning to program from an early age some 26 years ago, I’ve always kept in mind the performance of the code I write, whether it be COBOL, Java, Pl/SQL, Php or assembly. Now I’ve been given the change to get involved with the performance testing of some new commercial web properties and it just so happens that due to some prior knowledge and other factors we are using JMeter. Since the start we have been using fairly simplistic test scripts which includes multiple thread groups, each running a different ‘test case’, for instance, one group may visit the homepage, carry out a search, view a result, another may visit the homepage, login, browse a list of documents, then logout. These are basic test cases and don’t really come close to simulating a ‘real user’ but instead simply test a simplistic user flow. Each of these thread groups is configured for a certain number of threads and iterations to hopefully produce the required number of transactions per hour – a very inexact method and one which doesn’t align well with the requirements which go along with the system under test, such as 1000 search tests per hour. The problem comes that if you know that a single run of the search thread would take 30 seconds on average including think times, a single thread would be able to loop 120 times, thus to give us our 1000 per hour we would need 8.33 threads, rounded down giving us 8 threads running 120 loops giving us roughly 960 simplistic user journeys per hour – not spot on but quite close. Now let’s say that during your hour test the response times increase due to an issue on the system under test at 30 mins execution time, for simplicity’s sake lets say the remaining 480 tests take 45 seconds each, your execution time extends to 1 hour 15 mins. The problem is that because you are limited by the number of threads in the group the test time can only increase (or indeed decrease in other cases). If you report that response times increased you cannot give a true figure because the tests were not equal – in terms of execution time. This is due to the fact the JMeter testing acts as a “closed system”. Due to the fact that we fix the thread count, if response times increase the load profile being applied to the server changes, ie it changes from 8 thread journeys every 30 seconds to 8 thread journeys every 45 seconds – the test actually lessens the effect of the performance decrease being observed. In an ideal world you would generate 8 thread journeys per 30 seconds whatever the response time is – as new users cant back off from visiting the site due to its performance until they visit the site, ie the number of new visits is not dictated by the system performance – but by marketing etc. Thus as you can see, closed system testing can mask and skew results. So can we model an ‘open system’ in JMeter? – well yes, at least to some degree, and that will be the topic of my next post.

3 comments to JMeter – Simplistic tests and “closed system” testing

  • I am actually thankful to the owner of this website
    who has shared this fantastic post at at this time.

  • It’s really a nice and helpful piece of info. I am glad that you just shared
    this helpful information with us. Please keep us informed like this.
    Thank you for sharing.

  • Nate

    Hey there Roamer, I hope all is well. I have been reading some of your posts and want to know if you’re willing to answer a couple of questions,


A sample text widget

Etiam pulvinar consectetur dolor sed malesuada. Ut convallis euismod dolor nec pretium. Nunc ut tristique massa.

Nam sodales mi vitae dolor ullamcorper et vulputate enim accumsan. Morbi orci magna, tincidunt vitae molestie nec, molestie at mi. Nulla nulla lorem, suscipit in posuere in, interdum non magna.