Description

Apache JMeter is open source software, a 100% pure Java desktop application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.

Apache JMeter may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. You can use it to make a graphical analysis of performance or to test your server/script/object behavior under heavy concurrent load.

Overview

Pros

Free & Open Source
Distributed testing

Cons

A little buggy
Reporting is lacking

Pricing

Free and Open Source
Programmed in Java

Ease of Use

Ease of Installation

Install Java, download the zip file and unzip it. Run the .bat file to run the program and you’re set.

Setting up a Test

Add a thread group to the test plan. Configure the thread group by determining ramp-up period and how many threads you want running at the end (see Figure 1). Set up a proxy to record your test (Right-click Workbench à Add à Non-test elements à HTTP Proxy server). After the proxy is set up you can tell your browser to route requests through the proxy (default localhost:8080). The proxy will record all requests and store them in the Thread Group as samplers. Add a listener to record your test results and you’re good to go. Overall it’s a very straightforward, well-documented process.

Figure 1 - Create thread group

Running a Test

Run à Start or Ctrl+R. Bam!

Technical Knowledge Needed

Very little technical knowledge is needed. It does require the ability to learn how to do something from the documentation since it is not very intuitive.

Platforms

JMeter is a Java program, and thus runs wherever Java can. It requires Java 1.4 or greater.

Functionality

Recording a test

JMeter provides a proxy server through which you can record tests (see Figure 2). It’s fairly simple to set up, and it does a good job recording every request sent through it. You can choose which kinds of files to include or exclude in the recording, which is great because you probably don’t need to record all .gif, .png, etc. files as separate requests. Set it up by right-clicking the Workbench and select Add à Non-test elements à HTTP Proxy Server. It can then be configured to group the samplers however you want, and add the instructions for which URL patterns to include or exclude. See Figure 2 for an example set up. Once it is set up, just click start and tell your web browser to direct traffic through the proxy. The samplers will be created and you are well on your way to having your dream test.

Because it uses a proxy, it cannot detect HTTPS traffic, so any secure requests will have to be created manually.

Figure 2 - Set up a Proxy Server

Scripting

JMeter supports BeanShell scripting. I don’t know anything about it, so I can’t really rate its ability, but supposedly it exists.

Reporting

There are several reports that can be viewed within JMeter that provide a moderate level of detail. The data listed below can be recorded in a CSV file after each test. The built-in reports you can view in the GUI during test runs are lacking. Some of them don’t work. Some are useful (see Figure 3). The list below includes all information available in the CSV file:

Start time (ms)
Load time (ms)
Sampler name
Response code
Response message
Thread name
Content type (e.g. text, bin)
Success
Size in bytes
Latency

Figure 3 - Distribution Graph

Trend Reporting

JMeter’s listeners do several things, including trend graphing (see Figure 4). In addition, the exported CSV file contains detailed information for each request, including the time that the request started and how long it took. From this, with some Excel magic, it would be possible to create a trend graph.

Figure 4 - Trend graph

Integration

The only integration JMeter supports is exporting test results to CSV.

Data Functions

Using a CSV Data Set config element, you can easily create user defined variables using the information from a CSV file (See Figure 5).

Figure 5. CSV Data Set Config

Load Generation / Distributed Load Generation

JMeter offers distributed testing. I have not tested it, but from reviews it appears that in order to use multiple load generation tools, the GUI must be used. However, it is possible to run tests on one computer without the GUI and just record the results to a file. This minimizes system resource use and allows you to generate a higher load from one machine.

Documentation

The documentation for JMeter is extensive. The JMeter website has a ton of documentation, great tutorials, a great user manual, and clear Javadocs. There are also many other website with great tutorials and documentation for getting started with JMeter.

User Interface

Typical Java Swing GUI. Ugly but functional.

Performance v. Stress Testing

JMeter offers both constant-load performance testing and ramp-up stress testing. The tests are set up to run for a certain number of repetitions.

Bugs/Weirdness

Excel problem with JMeter CSV files: When saving test results to a csv, Excel likes to show big numbers (like the start time) in engineering notation (i.e., 1.24E10). If you open the file and save it, it will save it in engineering notation. When you open the file again, several significant digits will be lost, making the start time data essentially useless. This can be overcome by changing the stuff from a CSV to XLSX or by formatting the number as a regular number.
You can’t save both the Test Plan and WorkBench at the same time. You have to save each separately. I also couldn’t get them to open together. I could open either the workbench or the test plan file, but if I tried to do both, the second one I opened would overwrite the first.

Project Info

History

Stefano Mazzocchi of the Apache Software Foundation was the original developer of JMeter. He wrote it primarily to test the performance of Apache JServ (a project that has since been replaced by the Apache Tomcat project). JMeter has been redesigned to enhance the GUI and to add functional-testing capabilities.

Frequency of Releases

JMeter has a new release approximately 1-2 times per year. Most of these releases are minor bug-fix versions. The most recent version (2.3.4) was released in June, 2009.

Ohloh.net says that JMeter is experiencing decreasing year-over-year development activity.

QA Tool Review: JMeter