BOSH is an open source tool for release engineering, deployment, lifecycle management, and monitoring of distributed systems. BOSH can provision and deploy software over hundreds of VMs. It also performs failure recovery and software updates with zero-to-minimal downtime.
Apache JMeter™ is open source software, a 100% pure Java application designed to load test functional behavior and measure performance. It was originally designed for testing web applications but has since expanded to other test functions. JMeter may be used to test performance both on static and dynamic resources, web dynamic applications. It can be used to simulate a heavy load on a server, group of servers, network or object to test its strength or to analyze overall performance under different load types.
Tornado for Apache JMeter™, is a BOSH release for Apache JMeter ™. It simplifies the usage of JMeter™ in distributed mode, making it easier to perform distributed load/stress testing on multiple IAAS offerings.
In this internal test, we used a service with the necessary permission (that can also handle this load). I repeat, DO NOT choose a target before getting the corresponding approval.
We will be using
m4.large instances for the JMeter Worker VMs. Add this snippet under the
vm_types section in the cloud-config, and update it.
- name: default cloud_properties: instance_type: m4.large
The Subnet where all the VMs will come up need to enable the
Enable auto-assign public IPv4 address. This will allow your VM to have access to the internet when load testing an external target. You'll also need to configure the route table and internet gateway accordingly in your AWS account settings.
Deploy the following manifest (Change the target URL in the
jmeter_storm job). It deploys Tornado for Apache™ JMeter™ in STORM mode. It creates 100 worker VMs, each with a JMeter instance that starts and waits for an execution plan to be delivered to it. Each JMeter instance will start 5000 threads. This number is usually a large one for an individual JMeter instance, but we'll stick with it for this test. See the note at the end of this post.
A quick look at the jobs properties, each thread will make a GET HTTP call each 5 seconds, for 3 minutes , with a rampup time of 2 minutes. The deploy will take few minutes; it is creating 100 VMs.
--- name: jmeter releases: - name: jmeter-tornado version: "2.1.1" url: https://bosh.io/d/github.com/jamlo/jmeter-bosh-release?v=2.1.1 sha1: c47ae7cd0094c5d9275ec083a8cdeef894f0064a stemcells: - alias: default os: ubuntu-trusty version: latest update: canaries: 50 max_in_flight: 50 canary_watch_time: 5000-60000 update_watch_time: 5000-60000 instance_groups: - name: storm-workers azs: [z1] instances: 100 jobs: - name: jmeter_storm_worker release: jmeter-tornado properties: jvm: xms: 1024m xmx: 6144m vm_type: default stemcell: default networks: - name: default - name: storm lifecycle: errand azs: [z1] instances: 1 jobs: - name: jmeter_storm release: jmeter-tornado properties: wizard: configuration: users: 5000 ramp_time: 120 duration: 180 gaussian_random_timer: constant_delay_offset: 5000 deviation: 5000 targets: - name: An Example Test url: "http://<a target that you have permission to load test>" http_method: GET vm_type: small stemcell: default networks: - name: default
At the end of the deploy run
bosh instances, you should get something similar to this:
[jamil@ra3id: bosh-1 ]$ bosh instances Using environment 'X.X.X.X' as client 'admin' Task 216. Done Deployment 'jmeter' Instance Process State AZ IPs storm-workers/012c4e52-ef5a-4f7b-805c-4294206f49f1 running z1 10.0.1.55 storm-workers/01dbdf14-545a-4d66-8998-7971f5dd6a25 running z1 10.0.1.7 storm-workers/03be756f-e259-42ae-9f53-b897655cdeeb running z1 10.0.1.8 storm-workers/097af91e-9af3-4cdb-b46e-eb76ccefb3b1 running z1 10.0.1.9 storm-workers/10741b24-40bc-4b47-b851-e25b76a0a56a running z1 10.0.1.56 storm-workers/1220f5b1-23d6-48a1-be89-756dc5840a4d running z1 10.0.1.10 storm-workers/17b8b1ad-a155-459f-ae44-74c09582bd6a running z1 10.0.1.57 storm-workers/1880b213-dc82-4fda-80b2-6f9ed34f9014 running z1 10.0.1.58 storm-workers/1b4bd5ae-7bed-468b-b20c-5d59882bf249 running z1 10.0.1.59 storm-workers/1b92e5e5-5d14-44b1-9f12-ffc604ba6177 running z1 10.0.1.11 storm-workers/1b962c14-6f91-4729-b3ce-4e444e9395f9 running z1 10.0.1.100 storm-workers/1dbcc612-63d6-4bcd-a0aa-910f11718f46 running z1 10.0.1.60 storm-workers/1dc6cff0-783a-40d4-97ab-b1db78d0965a running z1 10.0.1.61 storm-workers/1f7c4c0e-8c31-4d21-9722-2f9030b9cd40 running z1 10.0.1.12 storm-workers/26de8fa7-470e-436d-b2a2-dd7648f0be12 running z1 10.0.1.62 storm-workers/2ae361a4-9a72-42f2-8ad7-9f0c064f6e8a running z1 10.0.1.63 storm-workers/2ae98c98-f5d5-4f37-9525-aebb1874ca57 running z1 10.0.1.6 storm-workers/2de84ea4-c5ed-44ee-91b2-3f0eafcc3976 running z1 10.0.1.13 storm-workers/2ee299c5-1071-4840-8f5e-2f82e1e29e5c running z1 10.0.1.64 storm-workers/2fc857b8-1aeb-4d69-8400-e42eed41c9ae running z1 10.0.1.14 storm-workers/30bbc8a0-fb0e-4c99-9f89-54049292d413 running z1 10.0.1.15 storm-workers/342eaa4e-ea4c-43b5-88e9-88bf9b689015 running z1 10.0.1.16 storm-workers/3580e767-55f4-467c-9ff0-db7f355341d6 running z1 10.0.1.65 storm-workers/3852d3b9-7021-4c13-b2dd-9d2f32f90a92 running z1 10.0.1.17 storm-workers/3924d891-3b87-4c55-ad4d-df7f376568ed running z1 10.0.1.18 storm-workers/3d8cdf2e-c781-4059-bba0-b14ad3a12f12 running z1 10.0.1.66 storm-workers/3fc86cf9-4475-44e7-a0a5-851c26280096 running z1 10.0.1.19 storm-workers/43bd0db6-82d6-40f2-b5a2-5d87686ca4b7 running z1 10.0.1.20 storm-workers/460a2a16-28f3-4c8f-9cec-6259470f52d8 running z1 10.0.1.21 storm-workers/47ce2858-0cf2-472a-abf4-c62dd7da4332 running z1 10.0.1.22 storm-workers/508d80f6-79fc-4270-8075-a3f12bb769f2 running z1 10.0.1.67 storm-workers/54be2d45-0c0c-4b96-89e9-76159b99eb43 running z1 10.0.1.23 storm-workers/5a2b34b9-0e73-4c17-816a-9cd93b0fdddb running z1 10.0.1.24 storm-workers/5c35db4d-effb-4086-86eb-b276f659afbe running z1 10.0.1.68 storm-workers/5c824ca8-0eb2-40c8-a601-ee8925301f67 running z1 10.0.1.69 storm-workers/5f551106-2147-4936-88cb-fd77fca74774 running z1 10.0.1.70 storm-workers/5fa68c1b-23c0-4fad-8413-4a9bc183facf running z1 10.0.1.25 storm-workers/60365918-0f0f-4108-8113-636d3bb24be7 running z1 10.0.1.101 storm-workers/60feda1f-1aff-4053-9d0c-a300afd5c450 running z1 10.0.1.71 storm-workers/61f98b1e-2dd4-4a7b-8db5-906dc7f91fbd running z1 10.0.1.26 storm-workers/63504bca-4946-4a73-ae94-c796590b0c0d running z1 10.0.1.72 storm-workers/6388be98-be90-4888-88b3-64cf8114dd98 running z1 10.0.1.27 storm-workers/6605ed80-ea0f-48c3-b373-d1c6cef10437 running z1 10.0.1.28 storm-workers/6980dd56-68a9-4ccc-9ce7-599b73dd401a running z1 10.0.1.29 storm-workers/6b705cff-5f41-432d-8685-7130faa92a92 running z1 10.0.1.30 storm-workers/6f445301-a145-4677-91f8-430952029458 running z1 10.0.1.73 storm-workers/6f74b098-1f05-4095-9875-9fc072202bbc running z1 10.0.1.31 storm-workers/76f5b4c0-22ce-458b-99ca-4db70726d04f running z1 10.0.1.32 storm-workers/7f1a788f-c686-4293-b9db-f975b11987a1 running z1 10.0.1.74 storm-workers/84fa5894-f438-4608-ac0b-a3b9f2d6c03f running z1 10.0.1.33 storm-workers/890b879a-1bf0-4dbb-9e55-b5f360f09567 running z1 10.0.1.34 storm-workers/8e39ca72-4668-4689-87fb-020149fee2ae running z1 10.0.1.75 storm-workers/9611dabd-84c4-473e-8fc5-909b309c7a3d running z1 10.0.1.76 storm-workers/97dec8fb-e075-451f-b4b9-3cc6a7ba50f1 running z1 10.0.1.35 storm-workers/9984a17e-633a-49fc-b49e-7930118212e1 running z1 10.0.1.36 storm-workers/9b6c5661-0b2b-4c92-84af-9b0804880d8b running z1 10.0.1.77 storm-workers/9bbcf7f1-c3fd-4508-ba52-e7fa5c666961 running z1 10.0.1.78 storm-workers/a005cbfc-4558-4353-8d61-a181755cd21b running z1 10.0.1.37 storm-workers/a680d32f-251f-43b1-bce8-ce14f4aa6875 running z1 10.0.1.79 storm-workers/a932e157-f9da-4757-bd22-449bac79beae running z1 10.0.1.80 storm-workers/a9457e0b-1d46-43f4-a5f8-5e5e0f6fe73b running z1 10.0.1.81 storm-workers/a9ffcfa6-d3a9-4155-99b0-c57cba2bdad8 running z1 10.0.1.82 storm-workers/aa777614-b306-40df-ba95-8262fc9278cb running z1 10.0.1.83 storm-workers/aaee289c-bb4f-4f3d-82cf-909a95116ae7 running z1 10.0.1.38 storm-workers/ab4a5729-c1f6-471a-8a67-d61167f34ae7 running z1 10.0.1.84 storm-workers/aefff130-1205-4e50-86de-b652722eb645 running z1 10.0.1.4 storm-workers/b144772c-0702-40e6-8070-dfbac47adca1 running z1 10.0.1.39 storm-workers/b66ea281-6705-4ec6-b5eb-d1667c943d2d running z1 10.0.1.40 storm-workers/b6d81f58-3d3f-481a-ab1a-2a369f973207 running z1 10.0.1.41 storm-workers/b86f4d93-d7d8-42b3-b6ed-430a7f3de4a6 running z1 10.0.1.85 storm-workers/bb83b13b-2bfa-4ca7-b4ac-30451246a493 running z1 10.0.1.86 storm-workers/c0fe0a7b-22e2-4b0c-9a6b-a2ca14b89545 running z1 10.0.1.87 storm-workers/c3791e89-544e-48c5-9955-5f538dd532f9 running z1 10.0.1.88 storm-workers/c3e4ef40-66c4-4dd4-9808-1ef4cee451e2 running z1 10.0.1.42 storm-workers/c6246836-56f3-46e1-afd4-985b421cde85 running z1 10.0.1.43 storm-workers/c9634d5f-1503-45af-bb7f-14a4c2d7ce05 running z1 10.0.1.89 storm-workers/cb7b6c75-c581-43d0-96a6-4ff241ac196e running z1 10.0.1.44 storm-workers/cc205ae0-b11a-4f6e-80c9-254eab2e4a56 running z1 10.0.1.45 storm-workers/cdc96956-57e8-4a1a-ba7c-87a2fca4de16 running z1 10.0.1.102 storm-workers/ce2f5d05-e989-44e1-b599-116e621cce10 running z1 10.0.1.90 storm-workers/ceb33e8f-8a87-4d8e-b25f-b967711c8489 running z1 10.0.1.91 storm-workers/d0cebb00-74c7-45df-affa-399330295de7 running z1 10.0.1.46 storm-workers/d7853cda-5161-451a-af26-cd9757fd80c8 running z1 10.0.1.47 storm-workers/d7b492ee-7657-4f65-8434-e17f7cbe1035 running z1 10.0.1.92 storm-workers/de87cab5-fab8-497e-9b29-7359b3246792 running z1 10.0.1.93 storm-workers/e497884c-e57d-420d-a7d6-5202f1611e8f running z1 10.0.1.94 storm-workers/e506a559-c03a-4d6a-91c4-a1c9bc46e2b8 running z1 10.0.1.95 storm-workers/e61c5bd3-d37d-4fed-8d0b-c1f577bdbb19 running z1 10.0.1.48 storm-workers/e6302b6d-22b5-4b1e-a541-b93211e7a79e running z1 10.0.1.49 storm-workers/e69121ae-0a15-45d6-9b77-3fa23a5e03ee running z1 10.0.1.96 storm-workers/e823047c-1c9c-4c25-b693-8b7cb52db3d5 running z1 10.0.1.97 storm-workers/ea9211b6-2f13-4a96-bca1-2c2e02be1662 running z1 10.0.1.98 storm-workers/eaa1f261-6e95-4d1b-a26d-09185208aa0c running z1 10.0.1.50 storm-workers/ec4aa7fb-d950-4ba1-94bd-5cf4be148a20 running z1 10.0.1.99 storm-workers/ed194209-e517-41c8-b357-24a170a1b734 running z1 10.0.1.51 storm-workers/ef9b1acd-2547-4c70-ab22-639eba25da7d running z1 10.0.1.52 storm-workers/f0199888-3d9f-4643-89a8-cb6ecb603aba running z1 10.0.1.53 storm-workers/f04f19f6-b141-4f79-b3e4-cd2ebfd9a051 running z1 10.0.1.54 storm/2d73d1d5-84f4-49eb-b123-03d1dbe3667e - z1 10.0.1.5 99 instances Succeeded
Since we are using STORM mode, we'll need to run a BOSH errand that will trigger all the JMeter™ workers. Run :
bosh run-errand -d jmeter storm --download-logs
in your terminal. This will create the errand VM that runs JMeter™ in client mode, distributes the JMX plan to all workers, waits for them to finish, aggregates the logs, and then download the tests result (including a statically generated dashboard) to your local machine.
After the errand has finished and downloaded the results onto your local machine, extract the tarball. You'll find all the test run logs, including the JTL file. This can be used to generate the corresponding metrics.
In-addition to the log files, a dashboard has also been generated. In the extracted directory, navigate to dashboard folder and double click
index.html. You'll find detailed graphs for the test run result. An example is shown below.
Congrats! You have just simulated 0.5 million users. Make sure to delete the deployment after the test run if you are not using it, or else the IAAS bill will be interesting.
In our test, each thread will make an HTTP call every 5 seconds. If you want to lower that number, the number of threads per every JMeter instance needs to be lowered down, too.
Note: Apache™, Apache JMeter™, and JMeter™ are trademarks of the Apache Software Foundation (ASF).