Performance testing for high-load and distributed systems

Reproduce performance degradation under controlled load, and keep the exact workload, environment and runtime data attached to the result.

What you get

A performance result is only useful if you can produce it again on demand.

•

Repeatable load scenarios

The same workload, defined once and run again whenever it is needed

•

Distributed execution

Scenarios that span several hosts, nodes or devices at once

•

Runtime resource evidence

CPU, memory, system events and logs recorded while the load was applied

•

Comparison across releases

The same workload on two builds, with both sets of evidence side by side

•

Centralized results

One place for the runs, instead of logs spread across the hosts that produced them

Why performance failures are hard to reproduce

Performance is a property of the whole setup, so it moves when any part of the setup moves.

Throughput drops only at a particular load

Below that point everything looks fine, and the graph gives no warning

Latency changes with configuration or topology

The same binary, a different arrangement of nodes, a different answer

One host or device behaves differently from another

Which makes the average useless and the outlier the thing worth reading

Resource use grows over long runs

Nothing fails in ten minutes, and something does after ten hours

A new release behaves differently under the same scenario

Which is only visible if the previous run is still there to compare against

Performance scenarios Midair can coordinate

The load generators stay whatever the project already uses. Midair drives the scenario and keeps what it ran against.

Load testing

Behaviour under the sustained workload the system is expected to carry.

Stress testing

Behaviour near the expected limit and past it, where the failure mode is the result.

Long-running testing

Degradation over time, where the interesting data is the slope rather than the peak.

Distributed performance testing

Scenarios involving several hosts, nodes or devices, coordinated as one run.

Midair

Performance testing in Midair

Execution and observation carry this work. Code analysis joins in only when the cause looks like it is in the code.

Execute

TS Factory

Coordinates the scenario across machines and devices, and records the environment it ran against.

Observe

Delta

Collects runtime resource data, logs and system events during the run, per host.

Compare

Midair

Puts the same workload on two versions or two configurations next to each other.

When degradation looks connected to software logic rather than to the environment, Visao can be brought in to analyze the code paths involved.

Compare the same workload before and after a change

A performance regression is a difference between two runs. That only works if the two runs are actually comparable.

Run A

Release A

load X, configuration Y

Runtime evidence recorded per host

Run B

Release B

the same load X, the same configuration Y

Runtime evidence recorded per host

Only the release changed, so the difference in the resource data belongs to the release.

Where this workflow fits

High-load backend systems

Distributed services

Network infrastructure

Multi-host environments

Systems whose performance depends on configuration or topology

FAQ

What is performance testing?

Performance testing evaluates how a system behaves under different levels of load: what its throughput and latency are, and how its resource use changes as the workload grows.

What is the difference between load and stress testing?

Load testing checks how the system performs under the traffic it is expected to carry. Stress testing pushes it past that point, to find where it breaks and how it behaves while breaking.

What does Midair add to performance testing?

It preserves the test configuration and the runtime evidence, so the same scenario can be repeated and compared across versions rather than described from memory.

Start your pilot

Bring one workload into Midair, run it against the configuration you care about, and keep the resource data with the result.