IBM Skip to main content
Search for:   within 
      Search help  
     IBM home  |  Products & services  |  Support & downloads   |  My account

developerWorks > Java technology
Eye on performance: A load of stress
140 KBe-mail it!
Stress testing, load testing
One size doesn't fit all
If it were only that easy...
A rich feature set
The final word
About the authors
Rate this article
Related content:
Eye on performance series
Stress testing Web services
Proofing Web applications for performance and scalability
IBM developer kits for Java (downloads)
dW newsletters
dW Subscription
(CDs and downloads)
Stress testing and the factors that go into choosing the right tool for your project

Level: Introductory

Jack Shirazi ( load of stress), Director,
Kirk Pepperdine ( load of stress), CTO,

28 Oct 2003

Column iconIntrepid optimizers Jack Shirazi and Kirk Pepperdine, Director and CTO of, follow performance discussions all over the Internet. A recent stop on message boards revealed some questions on stress testing and load testing. Jack and Kirk examine this subject in detail and discuss how the right tool can make a load of difference in your results. discussion boards are usually quite active, so we stopped there this month to see what was happening in the world of performance. Given its name, it should come as no surprise that performance discussions at TheServerSide tend to focus on J2EE systems. Of course, that's a pretty wide-ranging subject, as it encompasses almost everything in the Java platform -- even J2ME systems are often clients to J2EE systems, so you can even get an occasional question about optimizing J2ME systems.

Stress testing, load testing
One of the most common questions asked on the performance lists is: "Is there a tool that can help me to stress test my J2EE application?" Before we can answer that, let's ask ourselves: what is stress testing, and why do these developers want to do it? (I'm sure quite a few of you have been in the stressful situation of absolutely needing to have your tests finished by yesterday, but that's not what we're talking about here.) Stress testing is aimed at discovering under what conditions your application's performance becomes unacceptable. You do this by changing the application inputs to place a heavier and heavier load on the application, measuring how performance changes with the variation in those inputs. This activity is also called load testing, though load testing usually describes a specific type of stress testing -- increasing the number of users to stress test your application.

The simplest way to stress test an application is to manually vary the inputs (number of clients, size of requests, frequency of requests, mix of requests, and so on) and plot how the performance varies. For some applications, this is all you need to do. But if you have many inputs, or a large range of values over which to vary those inputs, you probably need an automated tool. Also, with manual testing, it can be difficult to accurately reproduce the set of tests if you want to re-test your application after making changes. When it comes to having multiple users testing your application, it is almost impossible to run manual tests consistently and, unless you have a lot of unemployed friends, it can be very difficult to scale up the number of users testing the application.

One size doesn't fit all
Unfortunately, there is no generic stress test tool, because every application is different in what inputs it takes and how it deals with them. But for many J2EE applications, communications from the client arrive at the server using the HTTP protocol. Fortunately, there are many load testing tools that can simulate user activity over HTTP in a controlled and reproducible manner. These range from free tools, such as Apache JMeter, The Grinder, and PushToTest, to quite expensive tools such as Mercury Astraload. Typically, you get what what you pay for -- the more expensive the tool, the more it can do. To understand the differences, let's consider first what the most basic type of load test tool does.

If you were building your own load test tool, then you likely would start by writing a program that runs one thread per simulated client. Each thread would need to communicate with the server, probably using the classes. This approach would give you a bare-bones HTTP client simulation that could do GETs and PUTs. All each thread needs to do is send an HTTP request, collect the answer, wait some time (simulating "think time"), and repeat. The sequence of actions can be quite easily extracted into a separate configuration file. And presto, you've got a basic load testing tool. You will probably need to add some configuration options for how many threads (simulated clients) to run and whether they all start together or slowly increase the load. And, of course, you'll want to time the interactions with the server, because this is at the core of what you are trying to measure.

If it were only that easy...
Now, what about handling extended interactions, where one request depends on the results of the previous one? What about handling cookies? Cookies are essential for many session-oriented J2EE systems. How about varying data input? What if your J2EE application client needs to process some JavaScript in order to proceed with the next communication? After you've collected the response time data, how are you going to analyze it? What about other types of monitoring, such as CPU time, network utilization, heap size, paging activity, or database activity?

It's features like these and others, such as tools to facilitate recording of browser sessions and turning them into test scripts, that differentiate the high-end load testing tools from the basic ones. How do you choose the right tool for you? Of course, this depends on your needs, your schedule, and your budget. Most importantly, you need to use a tool that can correctly simulate the client browser features that your application requires. Once the basic features are in place, you can then consider the productivity of the tools. Generally, the more analysis tools that are included, and the more different types of performance data you can record, the more productive you can be -- and the more you are likely to pay. The top-end load testers can simulate multiple browsers, integrate with most application servers, gather performance data from multiple server hosts including operating system, JVM and database statistics, and generate datasets that can be analyzed afterwards using sophisticated analysis tools. On the other hand, the low-end load testers are free. In these budget-constrained days, there's a lot to be said for "free."

Figure 1 illustrates Apache JMeter, one of the free load testers, showing an automatically recorded script.

Figure 1. JMeter showing an automatically recorded script
JMeter showing an automatically recorded script

A rich feature set
Now that we've looked at the basic functionality of stress testing tools, what additional functionality would be nice to have? How do the various load testing tools differ? Your primary requirement must, of course, be that the tool can simulate your application clients. That may rule out a number of candidates if your application uses some unusual combination of browser features or other non-standard client technology. Beyond that, there are other features that make the load testing more productive. The following list may be a useful starting point for making a decision on the appropriate load testing tool for your project:

Simulates your clients
The primary requirement must be that the load tester handles the features and protocols that your application uses.

Run multiple simulated clients
This is the most basic functionality of a load tester, and it helps to determine what is a load tester and what isn't (some frameworks try to masquerade as load testers.).

Scripted execution with ability to edit scripts
If you can't script the interaction between the client and the server, then you cannot handle anything except the most simple clients. The ability to edit the scripts is essential -- minor changes shouldn't require you to go through the process of re-generating a script.

Supports sessions
If a load tester cannot support sessions or cookies, it isn't really much of a load tester, and it won't be able to load-test most J2EE applications.

Configurable numbers of users
The tester should let you specify how many simulated users are running each script, including letting you vary the number of simulated users over time, as many load tests should start with a small number of users and ramp up slowly to higher numbers of users.

Report success, errors, and failures
Each script must have a defined way to identify a successful interaction, as well as failure and error modes (an error would be getting no page back at all, whereas a failure might be getting the wrong data back on the page.)

Page display
It is useful for the load tester to allow you to inspect some of the pages that are being sent to the simulated users, so that you can be confident the test is working correctly.

Export results
You often want to be able to analyze test results with various tools, including spreadsheets and custom analysis scripts that can process the data. While many load-test tools include extensive analysis functionality, being able to export the data gives you more flexibility to analyze and catalog the data in arbitrary ways.

Think times
Real-world users do not request one page immediately after another -- there are generally delays between viewing one page and the next. Think time is the standard term to express adding a delay into a script to more realistically simulate user behavior. Most load testers support randomly generating think times based on a statistical distribution.

Client selection of data from lists
Users don't tend to work with the same set of data; each is usually running a different interaction with the server. The simulated users should also be doing this, and it is easier to make your simulated users appear to be working with varied data if the scripts can select data from lists of data at strategic points in the interaction.

Recording scripts from manually executed sessions
Rather than writing scripts, it is so much easier to just manually run through a session with your browser and have that session recorded for later editing.

Some applications use JavaScript extensively and need it to be supported by the simulated clients. However, the use of client-side JavaScript may increase the requirements for system resources on the test systems.

Analysis tools
Measuring performance is half the story. The other half is analyzing the performance data. Who better to build those analysis tools than the people who built the measurement tools? Well, at least that's the theory. In any case, the more analysis tools your toolkit provides, the better off you are.

Measure server-side statistics
The basic load tester measures client-based response times from client/server interactions. It would also be nice to gather other statistics, such as CPU utilization or page fault rates, as well. The more statistics you can get, the more you can do with your load testing system. If you have this data, you can then do useful things like view client response times in the context of server load and throughput statistics.

The final word
What one can achieve with any tool is often limited by one's skill, knowledge, and imagination. In describing what to look for in a load testing tool, we have also demonstrated what is possible when using such a tool. Now, we leave it up to you to use your imagination to extend the realm of possibilities.


About the authors
Jack Shirazi is the Director of and author of Java Performance Tuning (O'Reilly). In addition to his performance tuning focus, Jack also develops intelligent agent technology. Contact Jack at

Kirk Pepperdine is the Chief Technical Officer at and has been focused on Object technologies and performance tuning for the last 15 years. Kirk is a co-author of the book ANT Developer's Handbook. Contact Kirk at

140 KBe-mail it!

What do you think of this document?
Killer! (5) Good stuff (4) So-so; not bad (3) Needs work (2) Lame! (1)


developerWorks > Java technology
  About IBM  |  Privacy  |  Terms of use  |  Contact