Performance testing for high-load and distributed systems

Midair runs the same load on the same bench each time. Every run records its measurements in the log and stays a separate record.

A performance number means little on its own. It means something next to the same number from the previous build, measured the same way.

Start your pilot See an example comparison

perf/http_latency · build 2.4.1Example

load

wrk · 4 threads · 256 connections · 60 s

requests

41 230 req/s

latency

p50 1.9 ms · p99 9.1 ms

energy

408 J over 60 s · avg 6.8 W · peak 9.4 W

bench

lab-2 · 3 hosts · lock 9c41e07

record

run 312 · kept with its configuration

What performance testing is.Numbers under a known load.

Performance testing measures how a system behaves under load: its throughput, its latency, the requests it serves per second and the energy it uses.

Load testing checks the load the system is expected to carry. Stress testing goes past that point to find where the system breaks and how.

On devices and networks, the bench matters as much as the code. The same build gives different numbers on another host, link or power supply.

What a run measures.Choose a kind of measurement.

Throughput

Bits or packets per second through a link, a device or a service. iperf and iperf3 sit behind one interface, so a test can switch tools without a rewrite.

Tools

iperf, iperf3, netperf, TRex

In the run log

throughput  mean 912 Mbit/s  min 884  max 931
iperf3 · TCP · 8 streams · 60 s · Example

Latency

Round-trip time and request latency, recorded as a median and percentiles. The tail usually moves before the median does.

Tools

wrk, ping, netperf, sfnt-pingpong

In the run log

rtt  median 0.41 ms  p99 1.90 ms  stdev 0.22 ms
sfnt-pingpong · UDP · 64 bytes · Example

Requests

Requests and connections per second against HTTP servers, Redis and memcached.

Tools

wrk, redis-benchmark, memtier, memaslap

In the run log

rps  41 230 req/s    cps  2 950 conn/s
wrk · 256 connections · 60 s · Example

Energy

Energy over the test window in joules and milliwatt-hours, with average and peak power. It works with SCPI bench instruments and the PPK2, Joulescope, Otii and Monsoon profilers.

Tools

tsf-power

In the run log

energy  408 J (113.3 mWh) over 60 s
power   avg 6.8 W  peak 9.4 W
tsf-power · Joulescope · Example

A test writes each value into the Test Environment log as a measurement, with its unit and how it was aggregated: minimum, mean, median, percentile or a full series.

Compare two builds.Same load, same bench, one change.

An example chart drawn from two recorded runs of the same test. Move the slider or point at the chart to read each load step.

p99 latency by concurrent connections build 2.4.0 build 2.4.1 Example

Load step

Connections

256

build 2.4.0

4.8 ms

build 2.4.1

9.1 ms

Change

+90%

Midair has no screen that compares two runs. Each run is a separate record, with its measurements in its own log. If you publish runs to Bublik, its history view charts a measurement across runs.

Show the numbers

Connections	build 2.4.0, ms	build 2.4.1, ms	Change
32	2.1	2.0	-5%
64	2.2	2.1	-5%
96	2.4	2.3	-4%
128	2.6	2.5	-4%
160	2.9	2.9	0%
192	3.3	3.6	+9%
224	3.9	5.4	+38%
256	4.8	9.1	+90%
288	6.2	14.8	+139%
320	8.4	22.6	+169%

What keeps two runs comparable.Five things each record pins down.

1. The code

   builder.conf.lock records the exact commit of the suite and of every library it builds.

2. The bench

   Per-bench env and run.conf files name the hosts, interfaces and instruments the run uses.

3. The expected result

   trc.xml holds the expected verdict for each test. A test fails when its measurement crosses the limit it checks.

4. Several hosts, one run

   Test Environment coordinates load generators and receivers on several hosts in one run, with one log.

5. A new record every time

   Every start is a new record with its own configuration. Midair never writes over an earlier run.

Related capabilities.Where the load comes from.

Routers and gatewaysNetwork hardware testingTraffic, packet loss and link faults against devices on a bench. Every buildTest automationRepeatable runs across devices and machines. On the deviceEmbedded systems testingTest levels from one workstation to a fleet. The whole pictureSoftware testingWhat Midair does from the code to the running device.

Further reading.From our engineering work.

BlogWhat is the fastest programming language?Why the usual speed ranking has no useful answer, and what decides the speed of a program. Embedded testing diaryReproducible testsHow resets, device state, network isolation and run fingerprints make two runs comparable.

Questions.Short answers.

What is performance testing?

Performance testing measures how a system behaves under load: its throughput, its latency and the resources it uses as the load grows.

What is the difference between load testing and stress testing?

Load testing checks how the system performs under the traffic it is expected to carry. Stress testing pushes past that point to find where the system breaks and how it behaves while it breaks.

Which load tools can a test drive?

Test Environment has ready wrappers for iperf, iperf3, netperf, TRex, wrk, redis-benchmark, memtier, memaslap, sfnt-pingpong and stress. A test starts them on the agent hosts and reads their results.

Can Midair compare two builds?

Not on one screen. You run the same suite on both builds, and each run keeps its measurements in its own record. If you publish runs to Bublik, its history view charts a measurement across runs.

Can we measure energy use?

Yes. tsf-power measures the energy a device uses over a test window, with SCPI bench instruments or USB profilers such as PPK2, Joulescope, Otii and Monsoon.

Does it fit our CI?

Yes. Runs start from your CI, for example with the Jenkins plugin. An AI assistant can start runs and read failures through MCP.

Start with one workload on one bench.Then measure every build the same way.

Compare the plans on the pricing page. Prices are monthly, in euros.

Start your pilot