JMeter – Simplistic tests and “closed system” testing

Over the last few years I have used JMeter for some small web performance tests but only recently have I been able to really get to grips with it and the subject of performance testing. As I started learning to program from an early age some 26 years ago, I’ve always kept in mind the performance of the code I write, whether it be COBOL, Java, Pl/SQL, Php or assembly. Now I’ve been given the change to get involved with the performance testing of some new commercial web properties and it just so happens that due to some prior knowledge and other factors we are using JMeter. Since the start we have been using fairly simplistic test scripts which includes multiple thread groups, each running a different ‘test case’, for instance, one group may visit the homepage, carry out a search, view a result, another may visit the homepage, login, browse a list of documents, then logout. These are basic test cases and don’t really come close to simulating a ‘real user’ but instead simply test a simplistic user flow. Each of these thread groups is configured for a certain number of threads and iterations to hopefully produce the required number of transactions per hour – a very inexact method and one which doesn’t align well with the requirements which go along with the system under test, such as 1000 search tests per hour. The problem comes that if you know that a single run of the search thread would take 30 seconds on average including think times, a single thread would be able to loop 120 times, thus to give us our 1000 per hour we would need 8.33 threads, rounded down giving us 8 threads running 120 loops giving us roughly 960 simplistic user journeys per hour – not spot on but quite close. Now let’s say that during your hour test the response times increase due to an issue on the system under test at 30 mins execution time, for simplicity’s sake lets say the remaining 480 tests take 45 seconds each, your execution time extends to 1 hour 15 mins. The problem is that because you are limited by the number of threads in the group the test time can only increase (or indeed decrease in other cases). If you report that response times increased you cannot give a true figure because the tests were not equal – in terms of execution time. This is due to the fact the JMeter testing acts as a “closed system”. Due to the fact that we fix the thread count, if response times increase the load profile being applied to the server changes, ie it changes from 8 thread journeys every 30 seconds to 8 thread journeys every 45 seconds – the test actually lessens the effect of the performance decrease being observed. In an ideal world you would generate 8 thread journeys per 30 seconds whatever the response time is – as new users cant back off from visiting the site due to its performance until they visit the site, ie the number of new visits is not dictated by the system performance – but by marketing etc. Thus as you can see, closed system testing can mask and skew results. So can we model an ‘open system’ in JMeter? – well yes, at least to some degree, and that will be the topic of my next post.

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

  

  

  


*

Get a free giffgaff Sim