Preface:

Oftentimes, we have observed that the Jmeter samples of load scenarios don’t match with previous runs or don’t match among different servers in standalone and client server environment (distributed) etc. The results/samples depend on the many parameters like SUT version, time of execution, data in  back end, pacing of requests, think time between requests and iterations and jmeter server/client performance etc. But these situations are known and in general, load test engineers will try to setup and keep everything else consistent during all the test execution. However, of if you still notice differences in samples of hits per seconds, response time etc. during test execution from different systems,   then you should first verify your load test setup. This blog will will help you to identify the initial steps for handling situations where you notice such differences:

Reduce sampling impact on load generation:

The load generation process can be impacted highly (or negligible) based on the usage of the memory, cpu, disk and network by the jmeter server and/or client. In all of these scenarios, the network traffic can impact load generation heavily if we are using distributed load execution scenario and system is sending all the samples to client immediately, depending on the network and network adapter traffic handling capacity as well as on the client system performance (which is obvious in all of the cases). Even jmeter sends the samples back to client as they are generated and client writes it to file on  receipt  as  a default behavior means it uses network resources and client machine memory , disk and cpu for same. So, if client machines network or other resources performance are faulty, then it will impact on server’s load generation activity adversely. To avoid this, set the sample sending mode appropriately and you will find more on mode as moving forward on this post.

There are total 7 modes for sampling in JMeter:

1. Standard
   
2. Hold
   
3. Batch
   
4. Statistical
   
5. Stripped
   
6. StrippedBatch
   
7. Custom Implementation
   

1. Standard — Default mode

   This will send the samples to client as they are generated and client will write it to file on receipt in distributed environment. Same way it will try to write in file immediately in standalone environment. This will use less memory of load generating system but impact on load generation as the next requests will be sent after the samples sending is completed by network layer. So if the receiving client’s network is slow then the sample sending will be slow and which can impact on requests sending to SUT. Still if the standard mode is required then try to reduce sampling interval means reduce the client — server communication and sampling size. It is advisable to run small test in standalone mode and server mode and if you found difference then try to check the network and system performance.
   

2. Hold mode

   In this mode, the sample data is held in an array in the memory and will be saved to file at the end of run by client after sending this data to client in distributed mode while it get saved at last in standalone environment (not advisable to use this mode in standalone if disk IO is performing well). But as you must have already guessed, it is a memory consuming mode and it can be a major cause for system  crash . So, it is advisable tthink thrice before using this mode. For small load test it is among best load generation modes. Run small tests and watch the increase in memory usage to understand how risky it is to use this mode and how best to use this mode.
   

3. Batch mode  

   The system send samples in batch as per the defined threshold of samples or time. So set following property appropriately:
   

             num_sample_threshold – number of samples in a batch (default 100)
   

             time_threshold – number of milliseconds to wait (default 60 seconds)
   

   But sometimes it impact advertly if client network and system performance is not good.
   

4. Statistical mode

   This mode is mainly for summarized sampling and do not samples all the fields. Also, the sampling rate is depends on the properties described with batch mode. The samples would be grouped as per thread group name and sample label. It accumulates only following fields and other fields that changes between samples will be ignored:
   

   Fields which will be accumulated are: 1. Elapsed time, 2. Latency, 3. Bytes, 4. Sample count and 5. Error count.
   

   This mode somewhat reduces the samples data impact over the network and will use very less resources of client as well in distributed environment. So it is advisable to setup effiecient thresholds after consideration of client system performance, network performance etc.
   

5. Stripped mode

   This mode is nothing but standard mode without response data of successful samples.
   

6. StrippedBatch mode

   This is nothing but the Batch mode without response data of successful samples.

7. Custom implementation mode

   We can write our own custom sample sender class to customize the sampling behavior and set the mode parameter to this custom sample sender class name.
   
   This must implement the interface SampleSender and have a constructor which takes a single parameter of type RemoteSampleListener.
   
   For example:

   public class PdfResponseSampleSender implements SampleSender, Serializable { //The code will just show the basic implementation and may be missing some required implementation to fit in standard practice. private static final Logger log = LoggingManager.getLoggerForClass(); //SampleSender have a constructor which takes a single parameter of type RemoteSampleListener. So following will be used for that. private final RemoteSampleListener listener; PdfResponseSampleSender(RemoteSampleListener listener) { this.listener = listener; } //It will be fired while sample started and stopped public void sampleOccurred(SampleEvent event) { //Basically it is advicable to create enum of mime types. but here we just using it directoly in string variable String pdfType="application/pdf"; SampleResult result = event.getResult(); //Empty the response data if the media type is not pdf. if(result.getMediaType().compateTo(pdfType)!=0) { result.setResponseData(new byte[0]); } try { listener.sampleOccurred(event); } catch (RemoteException e) { log.error("Error sending sample result over network ",e) } } } 

   We can set mode=org.example.load.PdfResponseSampleSender into jmeter.properties and make sure that this class is reachable to jmeter.

Conclusion:

I would say that we can resolve any load generation issue easily as jmeter is very flexible and supports customization for most of the activities of L & P testing.

If you have any questions, please post those in the comments below, and I will try to answer those as soon as I can!

PS: All opinions in this post are my personal and it may or may not be of Apexon. Also, I have taken help of JMeter user manual and source code for this blog post.