Contents: Stress testing, load testing One size doesn't fit all If it were only that easy... A rich feature set The final word Resources About the authors Rate this article Related content: Eye on performance series Stress testing Web services Proofing Web applications for performance and scalability IBM developer kits for Java (downloads) Subscriptions: dW newsletters dW Subscription (CDs and downloads)

Jack Shirazi (mailto:jack@JavaPerformanceTuning.com?cc=&subject=A load of stress), Director, JavaPerformanceTuning.com
Kirk Pepperdine (mailto:kirk@JavaPerformanceTuning.com?cc=&subject=A load of stress), CTO, JavaPerformanceTuning.com

28 Oct 2003

Intrepid optimizers Jack Shirazi and Kirk Pepperdine, Director and CTO of JavaPerformanceTuning.com, follow performance discussions all over the Internet. A recent stop on TheServerSide.com message boards revealed some questions on stress testing and load testing. Jack and Kirk examine this subject in detail and discuss how the right tool can make a load of difference in your results.

TheServerSide.com discussion boards are usually quite active, so we stopped there this month to see what was happening in the world of performance. Given its name, it should come as no surprise that performance discussions at TheServerSide tend to focus on J2EE systems. Of course, that's a pretty wide-ranging subject, as it encompasses almost everything in the Java platform -- even J2ME systems are often clients to J2EE systems, so you can even get an occasional question about optimizing J2ME systems.

Stress testing, load testing
One of the most common questions asked on the performance lists is: "Is there a tool that can help me to stress test my J2EE application?" Before we can answer that, let's ask ourselves: what is stress testing, and why do these developers want to do it? (I'm sure quite a few of you have been in the stressful situation of absolutely needing to have your tests finished by yesterday, but that's not what we're talking about here.) Stress testing is aimed at discovering under what conditions your application's performance becomes unacceptable. You do this by changing the application inputs to place a heavier and heavier load on the application, measuring how performance changes with the variation in those inputs. This activity is also called load testing, though load testing usually describes a specific type of stress testing -- increasing the number of users to stress test your application.

The simplest way to stress test an application is to manually vary the inputs (number of clients, size of requests, frequency of requests, mix of requests, and so on) and plot how the performance varies. For some applications, this is all you need to do. But if you have many inputs, or a large range of values over which to vary those inputs, you probably need an automated tool. Also, with manual testing, it can be difficult to accurately reproduce the set of tests if you want to re-test your application after making changes. When it comes to having multiple users testing your application, it is almost impossible to run manual tests consistently and, unless you have a lot of unemployed friends, it can be very difficult to scale up the number of users testing the application.

One size doesn't fit all
Unfortunately, there is no generic stress test tool, because every application is different in what inputs it takes and how it deals with them. But for many J2EE applications, communications from the client arrive at the server using the HTTP protocol. Fortunately, there are many load testing tools that can simulate user activity over HTTP in a controlled and reproducible manner. These range from free tools, such as Apache JMeter, The Grinder, and PushToTest, to quite expensive tools such as Mercury Astraload. Typically, you get what what you pay for -- the more expensive the tool, the more it can do. To understand the differences, let's consider first what the most basic type of load test tool does.

If you were building your own load test tool, then you likely would start by writing a program that runs one thread per simulated client. Each thread would need to communicate with the server, probably using the java.net.URL classes. This approach would give you a bare-bones HTTP client simulation that could do GETs and PUTs. All each thread needs to do is send an HTTP request, collect the answer, wait some time (simulating "think time"), and repeat. The sequence of actions can be quite easily extracted into a separate configuration file. And presto, you've got a basic load testing tool. You will probably need to add some configuration options for how many threads (simulated clients) to run and whether they all start together or slowly increase the load. And, of course, you'll want to time the interactions with the server, because this is at the core of what you are trying to measure.

If it were only that easy...
Now, what about handling extended interactions, where one request depends on the results of the previous one? What about handling cookies? Cookies are essential for many session-oriented J2EE systems. How about varying data input? What if your J2EE application client needs to process some JavaScript in order to proceed with the next communication? After you've collected the response time data, how are you going to analyze it? What about other types of monitoring, such as CPU time, network utilization, heap size, paging activity, or database activity?

It's features like these and others, such as tools to facilitate recording of browser sessions and turning them into test scripts, that differentiate the high-end load testing tools from the basic ones. How do you choose the right tool for you? Of course, this depends on your needs, your schedule, and your budget. Most importantly, you need to use a tool that can correctly simulate the client browser features that your application requires. Once the basic features are in place, you can then consider the productivity of the tools. Generally, the more analysis tools that are included, and the more different types of performance data you can record, the more productive you can be -- and the more you are likely to pay. The top-end load testers can simulate multiple browsers, integrate with most application servers, gather performance data from multiple server hosts including operating system, JVM and database statistics, and generate datasets that can be analyzed afterwards using sophisticated analysis tools. On the other hand, the low-end load testers are free. In these budget-constrained days, there's a lot to be said for "free."

Figure 1 illustrates Apache JMeter, one of the free load testers, showing an automatically recorded script.

Figure 1. JMeter showing an automatically recorded script

A rich feature set
Now that we've looked at the basic functionality of stress testing tools, what additional functionality would be nice to have? How do the various load testing tools differ? Your primary requirement must, of course, be that the tool can simulate your application clients. That may rule out a number of candidates if your application uses some unusual combination of browser features or other non-standard client technology. Beyond that, there are other features that make the load testing more productive. The following list may be a useful starting point for making a decision on the appropriate load testing tool for your project:

Simulates your clients

The primary requirement must be that the load tester handles the features and protocols that your application uses.

Run multiple simulated clients

This is the most basic functionality of a load tester, and it helps to determine what is a load tester and what isn't (some frameworks try to masquerade as load testers.).

Scripted execution with ability to edit scripts

If you can't script the interaction between the client and the server, then you cannot handle anything except the most simple clients. The ability to edit the scripts is essential -- minor changes shouldn't require you to go through the process of re-generating a script.

Supports sessions

If a load tester cannot support sessions or cookies, it isn't really much of a load tester, and it won't be able to load-test most J2EE applications.

Configurable numbers of users

The tester should let you specify how many simulated users are running each script, including letting you vary the number of simulated users over time, as many load tests should start with a small number of users and ramp up slowly to higher numbers of users.

Report success, errors, and failures

Each script must have a defined way to identify a successful interaction, as well as failure and error modes (an error would be getting no page back at all, whereas a failure might be getting the wrong data back on the page.)

Page display

It is useful for the load tester to allow you to inspect some of the pages that are being sent to the simulated users, so that you can be confident the test is working correctly.

Export results

You often want to be able to analyze test results with various tools, including spreadsheets and custom analysis scripts that can process the data. While many load-test tools include extensive analysis functionality, being able to export the data gives you more flexibility to analyze and catalog the data in arbitrary ways.

Think times

Real-world users do not request one page immediately after another -- there are generally delays between viewing one page and the next. Think time is the standard term to express adding a delay into a script to more realistically simulate user behavior. Most load testers support randomly generating think times based on a statistical distribution.

Client selection of data from lists

Users don't tend to work with the same set of data; each is usually running a different interaction with the server. The simulated users should also be doing this, and it is easier to make your simulated users appear to be working with varied data if the scripts can select data from lists of data at strategic points in the interaction.

Recording scripts from manually executed sessions

Rather than writing scripts, it is so much easier to just manually run through a session with your browser and have that session recorded for later editing.

JavaScript

Some applications use JavaScript extensively and need it to be supported by the simulated clients. However, the use of client-side JavaScript may increase the requirements for system resources on the test systems.

Analysis tools

Measuring performance is half the story. The other half is analyzing the performance data. Who better to build those analysis tools than the people who built the measurement tools? Well, at least that's the theory. In any case, the more analysis tools your toolkit provides, the better off you are.

Measure server-side statistics

The basic load tester measures client-based response times from client/server interactions. It would also be nice to gather other statistics, such as CPU utilization or page fault rates, as well. The more statistics you can get, the more you can do with your load testing system. If you have this data, you can then do useful things like view client response times in the context of server load and throughput statistics.

The final word
What one can achieve with any tool is often limited by one's skill, knowledge, and imagination. In describing what to look for in a load testing tool, we have also demonstrated what is possible when using such a tool. Now, we leave it up to you to use your imagination to extend the realm of possibilities.

Resources

• Read the complete Eye on performance series by Jack Shirazi and Kirk Pepperdine.
  
  
• The Java Performance Tuning site contains thousands of performance tuning hints and tips.
  
  
• This review of stress testing tools compares several free and commercial tools.
  
  
• Get tips on performance testing SOAP-based applications from Frank Cohen, CEO of PushToTest (developerWorks, November 2001).
  
  
• Learn how to stress test your software without stressing out your testers (developerWorks, February 2001).
  
  
• Web services are at the heart of distributed computing, and interaction between them is often difficult to test. In this article, "Stress testing Web services" (developerWorks, May 2003), Chris Wilkinson shows that stress testing is an efficient method of detecting code defects, but only if the stress systems are designed effectively.
  
  
• "Proofing Web applications for performance and scalability" (developerWorks, June 2001) is a case study in developing a scripting framework for load testing.
  
  
• Apache JMeter is a free load/regression testing tool.
  
  
• The Grinder is a free tool that can orchestrate the activities of a test script across multiple machines using a graphical console application.
  
  
• Check out the Software Q/A Test Resource Center for information on Web site testing and site management tools.
  
  
• Visit the Developer Bookstore for a comprehensive listing of technical books, including hundreds of Java-related titles.
  
  
• Find hundreds of articles about every aspect of Java programming in the developerWorks Java technology zone.
  
  

What do you think of this document? Killer! (5) Good stuff (4) So-so; not bad (3) Needs work (2) Lame! (1) Comments?

developerWorks > Java technology