Test automation for complex and distributed systems

Describe a test in plain words. TS Factory writes it into your Test Environment suite and pushes it to a branch. Midair runs the suite pinned to that branch.

The tests run across devices, hosts and services. Each run keeps its configuration, logs and verdicts.

Join the waitlist See the example

assistant · MCP · run 518 Example

mcp     midair_start_run  wifi-ts @ ts/ping-latency
midair  run 518 started on node lab-2
test    wifi/connect_ping     PASSED
test    wifi/ping_latency     FAILED  expected PASSED
verdict 5% of the packets to the gateway were lost
mcp     midair_run_failures  run 518
mcp     midair_get_log  wifi/ping_latency  page 1
log     Sent 20, received 19, lost 5%
midair  run 518 saved: configuration, logs, verdicts

What test automation is.For systems with many parts.

Test automation uses software to prepare, run and check tests that people would otherwise repeat by hand.

In a complex system, the test is the small part. The hosts, the devices, the network and their order have to be ready before it runs, and recorded after it.

Test Environment does that work: agents on several hosts run one scenario together. TS Factory adds the writing: it turns a plain description into tests in the same suite.

From prompt to branch.Four steps, one example.

This example adds one test to a Wi-Fi suite. Step through it, or press Play.

Midair · Test suites

Write the spec in plain words

Open the suite on the Test suites screen and press Run. Type what the change should do, and pick Claude Code or the Codex CLI as the tool.

Example

Suitewifi-ts

Branchts/ping-latency

ToolClaude Code

Spec prompt

Add a test to the wifi package. Join the bench network and ping the gateway 20 times. Fail on any packet loss, or if the average round trip is over 50 ms. Make the count and the limit test parameters.

Run

TS Factory

TS Factory edits the suite

Midair sends the prompt to the tool with the context of the Test Environment framework. The tool clones the suite repository and changes it in place. The screen shows its output while it runs.

Example

ts factory · wifi-ts · claude code
clone   wifi-ts  →  ts/ping-latency
read    wifi/connect_ping.c      bench settings, ping
write   lib/wifi_bench.c         join steps, shared
write   wifi/ping_latency.c      new test
edit    wifi/package.xml         count, max_avg_ms
edit    trc.xml                  expected PASSED
commit  wifi: add ping_latency
4 files changed

wifi/ping_latency.c

Read the generated test

The result is an ordinary C test on the Test Environment API. Your engineers review it like any other change.

Example

/** @page wifi-ping_latency Ping the gateway over WiFi */
#define TE_TEST_NAME "wifi/ping_latency"

#include "wifi_bench.h"
#include "tapi_ping.h"

int
main(int argc, char **argv)
{
    unsigned int count;
    double max_avg_ms;
    wifi_bench bench = WIFI_BENCH_INIT;
    tapi_ping_app *ping = NULL;
    tapi_ping_report report;

    TEST_START;
    TEST_GET_UINT_PARAM(count);
    TEST_GET_DOUBLE_PARAM(max_avg_ms);

    TEST_STEP("Join the bench network");
    CHECK_RC(wifi_bench_join(&bench));

    TEST_STEP("Ping the gateway %u times", count);
    CHECK_RC(wifi_bench_ping(&bench, count, &ping, &report));

    TEST_STEP("Check loss and round trip");
    if (report.lost_percentage != 0)
        TEST_VERDICT("%u%% of the packets to the gateway were lost",
                     report.lost_percentage);
    if (report.rtt.avg > max_avg_ms)
        TEST_VERDICT("Average round trip %.1f ms is over %.1f ms",
                     report.rtt.avg, max_avg_ms);

    TEST_SUCCESS;

cleanup:
    if (ping != NULL)
        CLEANUP_CHECK_RC(tapi_ping_destroy(ping));
    wifi_bench_leave(&bench);
    TEST_END;
}

Midair · Runs

Pushed to a branch, run on that branch

The tool pushes the change to the branch the suite record names. A run pinned to that branch starts from a template, from CI or through MCP. The main branch stays as it was.

Example

git push origin ts/ping-latency
branch    ts/ping-latency   4 files changed
run 519   template wifi-nightly
          suite branch ts/ping-latency
          node lab-2 · env.lab-2

wifi/connect_ping    PASSED   expected PASSED
wifi/ping_latency    PASSED   expected PASSED
                     avg 7.4 ms, 0% lost
2 as expected · main unchanged

What is worth automating.And what can wait.

Automate first

• Checks you repeat on every build

  Boot, connect, pass basic traffic. They run most often, so they pay back first.

• Scenarios that need several hosts

  A device, its peers and a traffic source, started in the right order. Agents keep the order the same in every run.

• Failures you had to reproduce by hand

  Write the steps down as a test once. The next regression then shows up in a run, with its log.

• Long and nightly sessions

  Planned runs start a template on a schedule, for example every weekday at 02:00.

Leave manual for now

• Tests whose expected result is still unclear

  Agree on what should happen first. A generated test cannot settle a question nobody has answered.

• One-off checks during bring-up

  A board that is on the desk for one day does not need a suite yet.

• Decisions about what a finding means

  The attack vector loop marks each finding as shown by a run or only suspected. A person confirms it.

Use what you already have.Suites, scripts, pipelines and logs.

1. Existing Test Environment suites

   Midair runs suites built on Test Environment as they are. TS Factory changes them through ordinary commits on a branch.

2. Scripts and specs as source data

   A suite record can name a read-only source repository. The generator reads it as the source for the change, next to the prompt.

3. Your CI

   Runs start from Jenkins with the plugin, from the API or from an AI assistant through MCP.

4. Logs from another system

   Import a raw Test Environment log into a run record. Midair reads it the same way as a log it collected.

Questions.Short answers.

What is test automation?

Test automation uses software to prepare, run and check tests that people would otherwise repeat by hand. For complex systems, it also prepares the hosts, devices and network the test needs.

What is TS Factory?

TS Factory is the test suite generator in Midair. It passes a spec prompt in plain words to Claude Code or the Codex CLI. The tool changes a Test Environment suite and pushes the change to a branch.

Do we have to accept generated tests as they are?

No. The change arrives as commits on a branch. You review it like any other change, and you decide when to merge it.

Can an AI assistant run the tests too?

Yes. Through MCP, an assistant can create suites, start runs pinned to a branch, and read the failures and the log pages.

What is an attack vector loop?

It runs the generator in a loop: analyse, plan, write tests, run, judge. The loop never changes the code under test, and a person confirms each finding.

Can TS Factory test physical devices?

Yes. The tests reach a device through a host next to it, which controls power, flashing and the serial console. The device needs no agent.

Related pages.Guides and reading.

GuideSoftware testing for complex systemsPick your kind of system and see which tests matter for it. GuideEmbedded systems testingTest levels from one workstation to a fleet of devices. GuideFirmware testing servicesFlashing, boot, power loss and the bench under test control. GuideNetwork hardware testingDistributed scenarios across routers, gateways and device labs. GuidePerformance testingSustained and high-load scenarios across complex systems. Embedded testing diaryDevice fleetsHow tests with several agents handle mixed setups, shared resources and changing configurations. Embedded testing diaryReproducible testsWhat has to be recorded around a run so that a failure can be reproduced. BlogFast AI development is changing how software gets testedWhy code written faster needs tests written faster too.

Start with one test you repeat by hand.Write it once, then run it on every build.

Compare the plans on the pricing page. Prices are monthly, in euros.

Join the waitlist