Pivotal Engineering Journal

Technical articles from Pivotal engineers.

Half a Million Concurrent/Distributed JMeter Users with BOSH ... in 10 Minutes

This post details the steps of utilizing BOSH to setup half a million (potentially much more) concurrent and distributed Apache JMeter threads for load testing.

Posted on by
Categories:   BOSH    Apache JMeter    Distributed Load Testing    Distributed Stress Testing   
Edit this post on GitHub.

BOSH is an open source tool for release engineering, deployment, lifecycle management, and monitoring of distributed systems. BOSH can provision and deploy software over hundreds of VMs. It also performs failure recovery and software updates with zero-to-minimal downtime.

Apache JMeter™ is open source software, a 100% pure Java application designed to load test functional behavior and measure performance. It was originally designed for testing web applications but has since expanded to other test functions. JMeter may be used to test performance both on static and dynamic resources, web dynamic applications. It can be used to simulate a heavy load on a server, group of servers, network or object to test its strength or to analyze overall performance under different load types.

Tornado for Apache JMeter™, is a BOSH release for Apache JMeter ™. It simplifies the usage of JMeter™ in distributed mode, making it easier to perform distributed load/stress testing on multiple IAAS offerings.



1- Choose a Load Testing Target

2- Add an Additional VM Type to the Cloud Config

We will be using m4.large instances for the JMeter Worker VMs. Add this snippet under the vm_types section in the cloud-config, and update it.

- name: default
    instance_type: m4.large

3- Modify the Public IPv4 Addressing Attribute for Your Subnet

The Subnet where all the VMs will come up need to enable the Enable auto-assign public IPv4 address. This will allow your VM to have access to the internet when load testing an external target. You’ll also need to configure the route table and internet gateway accordingly in your AWS account settings.

4- Deploy

Deploy the following manifest (Change the target URL in the jmeter_storm job). It deploys Tornado for Apache™ JMeter™ in STORM mode. It creates 100 worker VMs, each with a JMeter instance that starts and waits for an execution plan to be delivered to it. Each JMeter instance will start 5000 threads. This number is usually a large one for an individual JMeter instance, but we’ll stick with it for this test. See the note at the end of this post.

A quick look at the jobs properties, each thread will make a GET HTTP call each 5 seconds, for 3 minutes , with a rampup time of 2 minutes. The deploy will take few minutes; it is creating 100 VMs.

name: jmeter

- name: jmeter-tornado
  version: "2.1.1"
  url: https://bosh.io/d/github.com/jamlo/jmeter-bosh-release?v=2.1.1
  sha1: c47ae7cd0094c5d9275ec083a8cdeef894f0064a

- alias: default
  os: ubuntu-trusty
  version: latest

  canaries: 50
  max_in_flight: 50
  canary_watch_time: 5000-60000
  update_watch_time: 5000-60000

- name: storm-workers
  azs: [z1]
  instances: 100
  - name: jmeter_storm_worker
    release: jmeter-tornado
        xms: 1024m
        xmx: 6144m
  vm_type: default
  stemcell: default
  - name: default

- name: storm
  lifecycle: errand
  azs: [z1]
  instances: 1
  - name: jmeter_storm
    release: jmeter-tornado
          users: 5000
          ramp_time: 120
          duration: 180
            constant_delay_offset: 5000
            deviation: 5000
        - name: An Example Test
          url: "http://<a target that you have permission to load test>"
          http_method: GET
  vm_type: small
  stemcell: default
  - name: default

At the end of the deploy run bosh instances, you should get something similar to this:

[jamil@ra3id: bosh-1 ]$ bosh instances
Using environment 'X.X.X.X' as client 'admin'

Task 216. Done

Deployment 'jmeter'

Instance                                            Process State  AZ  IPs
storm-workers/012c4e52-ef5a-4f7b-805c-4294206f49f1  running        z1
storm-workers/01dbdf14-545a-4d66-8998-7971f5dd6a25  running        z1
storm-workers/03be756f-e259-42ae-9f53-b897655cdeeb  running        z1
storm-workers/097af91e-9af3-4cdb-b46e-eb76ccefb3b1  running        z1
storm-workers/10741b24-40bc-4b47-b851-e25b76a0a56a  running        z1
storm-workers/1220f5b1-23d6-48a1-be89-756dc5840a4d  running        z1
storm-workers/17b8b1ad-a155-459f-ae44-74c09582bd6a  running        z1
storm-workers/1880b213-dc82-4fda-80b2-6f9ed34f9014  running        z1
storm-workers/1b4bd5ae-7bed-468b-b20c-5d59882bf249  running        z1
storm-workers/1b92e5e5-5d14-44b1-9f12-ffc604ba6177  running        z1
storm-workers/1b962c14-6f91-4729-b3ce-4e444e9395f9  running        z1
storm-workers/1dbcc612-63d6-4bcd-a0aa-910f11718f46  running        z1
storm-workers/1dc6cff0-783a-40d4-97ab-b1db78d0965a  running        z1
storm-workers/1f7c4c0e-8c31-4d21-9722-2f9030b9cd40  running        z1
storm-workers/26de8fa7-470e-436d-b2a2-dd7648f0be12  running        z1
storm-workers/2ae361a4-9a72-42f2-8ad7-9f0c064f6e8a  running        z1
storm-workers/2ae98c98-f5d5-4f37-9525-aebb1874ca57  running        z1
storm-workers/2de84ea4-c5ed-44ee-91b2-3f0eafcc3976  running        z1
storm-workers/2ee299c5-1071-4840-8f5e-2f82e1e29e5c  running        z1
storm-workers/2fc857b8-1aeb-4d69-8400-e42eed41c9ae  running        z1
storm-workers/30bbc8a0-fb0e-4c99-9f89-54049292d413  running        z1
storm-workers/342eaa4e-ea4c-43b5-88e9-88bf9b689015  running        z1
storm-workers/3580e767-55f4-467c-9ff0-db7f355341d6  running        z1
storm-workers/3852d3b9-7021-4c13-b2dd-9d2f32f90a92  running        z1
storm-workers/3924d891-3b87-4c55-ad4d-df7f376568ed  running        z1
storm-workers/3d8cdf2e-c781-4059-bba0-b14ad3a12f12  running        z1
storm-workers/3fc86cf9-4475-44e7-a0a5-851c26280096  running        z1
storm-workers/43bd0db6-82d6-40f2-b5a2-5d87686ca4b7  running        z1
storm-workers/460a2a16-28f3-4c8f-9cec-6259470f52d8  running        z1
storm-workers/47ce2858-0cf2-472a-abf4-c62dd7da4332  running        z1
storm-workers/508d80f6-79fc-4270-8075-a3f12bb769f2  running        z1
storm-workers/54be2d45-0c0c-4b96-89e9-76159b99eb43  running        z1
storm-workers/5a2b34b9-0e73-4c17-816a-9cd93b0fdddb  running        z1
storm-workers/5c35db4d-effb-4086-86eb-b276f659afbe  running        z1
storm-workers/5c824ca8-0eb2-40c8-a601-ee8925301f67  running        z1
storm-workers/5f551106-2147-4936-88cb-fd77fca74774  running        z1
storm-workers/5fa68c1b-23c0-4fad-8413-4a9bc183facf  running        z1
storm-workers/60365918-0f0f-4108-8113-636d3bb24be7  running        z1
storm-workers/60feda1f-1aff-4053-9d0c-a300afd5c450  running        z1
storm-workers/61f98b1e-2dd4-4a7b-8db5-906dc7f91fbd  running        z1
storm-workers/63504bca-4946-4a73-ae94-c796590b0c0d  running        z1
storm-workers/6388be98-be90-4888-88b3-64cf8114dd98  running        z1
storm-workers/6605ed80-ea0f-48c3-b373-d1c6cef10437  running        z1
storm-workers/6980dd56-68a9-4ccc-9ce7-599b73dd401a  running        z1
storm-workers/6b705cff-5f41-432d-8685-7130faa92a92  running        z1
storm-workers/6f445301-a145-4677-91f8-430952029458  running        z1
storm-workers/6f74b098-1f05-4095-9875-9fc072202bbc  running        z1
storm-workers/76f5b4c0-22ce-458b-99ca-4db70726d04f  running        z1
storm-workers/7f1a788f-c686-4293-b9db-f975b11987a1  running        z1
storm-workers/84fa5894-f438-4608-ac0b-a3b9f2d6c03f  running        z1
storm-workers/890b879a-1bf0-4dbb-9e55-b5f360f09567  running        z1
storm-workers/8e39ca72-4668-4689-87fb-020149fee2ae  running        z1
storm-workers/9611dabd-84c4-473e-8fc5-909b309c7a3d  running        z1
storm-workers/97dec8fb-e075-451f-b4b9-3cc6a7ba50f1  running        z1
storm-workers/9984a17e-633a-49fc-b49e-7930118212e1  running        z1
storm-workers/9b6c5661-0b2b-4c92-84af-9b0804880d8b  running        z1
storm-workers/9bbcf7f1-c3fd-4508-ba52-e7fa5c666961  running        z1
storm-workers/a005cbfc-4558-4353-8d61-a181755cd21b  running        z1
storm-workers/a680d32f-251f-43b1-bce8-ce14f4aa6875  running        z1
storm-workers/a932e157-f9da-4757-bd22-449bac79beae  running        z1
storm-workers/a9457e0b-1d46-43f4-a5f8-5e5e0f6fe73b  running        z1
storm-workers/a9ffcfa6-d3a9-4155-99b0-c57cba2bdad8  running        z1
storm-workers/aa777614-b306-40df-ba95-8262fc9278cb  running        z1
storm-workers/aaee289c-bb4f-4f3d-82cf-909a95116ae7  running        z1
storm-workers/ab4a5729-c1f6-471a-8a67-d61167f34ae7  running        z1
storm-workers/aefff130-1205-4e50-86de-b652722eb645  running        z1
storm-workers/b144772c-0702-40e6-8070-dfbac47adca1  running        z1
storm-workers/b66ea281-6705-4ec6-b5eb-d1667c943d2d  running        z1
storm-workers/b6d81f58-3d3f-481a-ab1a-2a369f973207  running        z1
storm-workers/b86f4d93-d7d8-42b3-b6ed-430a7f3de4a6  running        z1
storm-workers/bb83b13b-2bfa-4ca7-b4ac-30451246a493  running        z1
storm-workers/c0fe0a7b-22e2-4b0c-9a6b-a2ca14b89545  running        z1
storm-workers/c3791e89-544e-48c5-9955-5f538dd532f9  running        z1
storm-workers/c3e4ef40-66c4-4dd4-9808-1ef4cee451e2  running        z1
storm-workers/c6246836-56f3-46e1-afd4-985b421cde85  running        z1
storm-workers/c9634d5f-1503-45af-bb7f-14a4c2d7ce05  running        z1
storm-workers/cb7b6c75-c581-43d0-96a6-4ff241ac196e  running        z1
storm-workers/cc205ae0-b11a-4f6e-80c9-254eab2e4a56  running        z1
storm-workers/cdc96956-57e8-4a1a-ba7c-87a2fca4de16  running        z1
storm-workers/ce2f5d05-e989-44e1-b599-116e621cce10  running        z1
storm-workers/ceb33e8f-8a87-4d8e-b25f-b967711c8489  running        z1
storm-workers/d0cebb00-74c7-45df-affa-399330295de7  running        z1
storm-workers/d7853cda-5161-451a-af26-cd9757fd80c8  running        z1
storm-workers/d7b492ee-7657-4f65-8434-e17f7cbe1035  running        z1
storm-workers/de87cab5-fab8-497e-9b29-7359b3246792  running        z1
storm-workers/e497884c-e57d-420d-a7d6-5202f1611e8f  running        z1
storm-workers/e506a559-c03a-4d6a-91c4-a1c9bc46e2b8  running        z1
storm-workers/e61c5bd3-d37d-4fed-8d0b-c1f577bdbb19  running        z1
storm-workers/e6302b6d-22b5-4b1e-a541-b93211e7a79e  running        z1
storm-workers/e69121ae-0a15-45d6-9b77-3fa23a5e03ee  running        z1
storm-workers/e823047c-1c9c-4c25-b693-8b7cb52db3d5  running        z1
storm-workers/ea9211b6-2f13-4a96-bca1-2c2e02be1662  running        z1
storm-workers/eaa1f261-6e95-4d1b-a26d-09185208aa0c  running        z1
storm-workers/ec4aa7fb-d950-4ba1-94bd-5cf4be148a20  running        z1
storm-workers/ed194209-e517-41c8-b357-24a170a1b734  running        z1
storm-workers/ef9b1acd-2547-4c70-ab22-639eba25da7d  running        z1
storm-workers/f0199888-3d9f-4643-89a8-cb6ecb603aba  running        z1
storm-workers/f04f19f6-b141-4f79-b3e4-cd2ebfd9a051  running        z1
storm/2d73d1d5-84f4-49eb-b123-03d1dbe3667e          -              z1

99 instances


5- Open the Flood Gates

Since we are using STORM mode, we’ll need to run a BOSH errand that will trigger all the JMeter™ workers. Run :

bosh run-errand -d jmeter storm --download-logs

in your terminal. This will create the errand VM that runs JMeter™ in client mode, distributes the JMX plan to all workers, waits for them to finish, aggregates the logs, and then download the tests result (including a statically generated dashboard) to your local machine.

6- Check the Results

After the errand has finished and downloaded the results onto your local machine, extract the tarball. You’ll find all the test run logs, including the JTL file. This can be used to generate the corresponding metrics.

In-addition to the log files, a dashboard has also been generated. In the extracted directory, navigate to dashboard folder and double click index.html. You’ll find detailed graphs for the test run result. An example is shown below.

7- Congrats

Congrats! You have just simulated 0.5 million users. Make sure to delete the deployment after the test run if you are not using it, or else the IAAS bill will be interesting.


In our test, each thread will make an HTTP call every 5 seconds. If you want to lower that number, the number of threads per every JMeter instance needs to be lowered down, too.

Note: Apache™, Apache JMeter™, and JMeter™ are trademarks of the Apache Software Foundation (ASF).