Network hardware testing for real-world conditions

Midair tests routers, gateways and connected devices from test hosts placed around them. You choose the network conditions, and every run keeps its evidence.

A network device rarely fails on a clean lab link. It fails when a link drops, packets go missing or many clients connect at once. The bench creates each of these on purpose.

Join the waitlist See the conditions

What network hardware testing is.The device, tested from both sides.

Network hardware testing checks how a router, gateway or connected device handles real traffic. Test hosts sit on its WAN side and its LAN side, send traffic through it and check what comes out.

The device under test needs no test agent. Agents run on the hosts around it, and Test Environment coordinates them in one run.

A result depends on the firmware, the configuration and the state of the network. Midair stores all three with the verdict.

Network conditions.Choose one to see the test it runs.

The drawing is an example bench. Each condition lights up the links it changes and shows an example log.

Example Test Environment loss 2% +150 ms down WAN WAN host agent · netem Router no agent device under test Wi-Fi client agent · tsf-wifi LAN client agent LAN clients ×16 agent · netns

A Wi-Fi client joins

What the bench does
The Wi-Fi client host joins the network named in the bench settings. It gets an address by DHCP and pings a host on the WAN side.
Test
wifi/connect_ping
Fails when
The client does not associate within 30 seconds, gets no address, or loses any of the four pings.
Example log · run 214 · wifi/connect_ping
stepBring wlan0 up
stepDescribe 'bench-5g' in the WiFi subtree of Agt_WiFi
ringAssociated after 4 second(s)
ringAddress on wlan0 is 192.168.8.23
stepPing 192.0.2.1 through wlan0
ringSent 4, received 4, lost 0%
ringRound trip time min/avg/max/mdev = 1.91/2.34/3.05/0.42 ms
testPASSED

Packet loss on the WAN

What the bench does
The WAN host adds 2% packet loss on its link to the router with netem. A LAN client then downloads a file through the router.
Test
wan/http_download
Fails when
The download stalls, or the transfer rate falls below the limit set in the test.
Example log · run 221 · wan/http_download · loss=2%
netemwan0: loss 2%
stepDownload 64 MiB from 192.0.2.1 on Agt_LAN
ring64 MiB in 11.8 s, 45.5 Mbit/s
checkrate 45.5 Mbit/s, limit 20 Mbit/s
testPASSED

High latency

What the bench does
netem adds 150 ms of delay and 30 ms of jitter on the WAN link. A LAN client sends TCP traffic to the WAN host with iperf3.
Test
wan/tcp_throughput
Fails when
Throughput falls below the limit set in the test, or the connection resets.
Example log · run 222 · wan/tcp_throughput · delay=150ms
netemwan0: delay 150 ms 30 ms
iperf3Agt_LAN -> Agt_WAN, TCP, 4 streams, 30 s
ringreceiver 38.2 Mbit/s, 412 retransmits
checklimit 50 Mbit/s
testFAILED  expected PASSED
verdictThroughput 38.2 Mbit/s is below 50 Mbit/s

WAN link down

What the bench does
The WAN host takes its interface down for 20 seconds and brings it back. LAN clients keep pinging the router and a WAN address.
Test
wan/link_flap
Fails when
LAN traffic stops while the WAN is down, or WAN traffic does not return within 60 seconds.
Example log · run 223 · wan/link_flap · down=20s
linkwan0 down
ringLAN ping to the router: 20 of 20 replies
linkwan0 up
ringWAN traffic back after 7 s, limit 60 s
testPASSED

Many clients at once

What the bench does
Sixteen LAN clients run in network namespaces on one host. They get addresses by DHCP and send traffic through the router at the same time.
Test
lan/many_clients
Fails when
A client gets no address, or the total throughput falls below the limit set in the test.
Example log · run 224 · lan/many_clients · clients=16
dhcp16 of 16 clients got an address
iperf316 clients -> Agt_WAN, TCP, 60 s
ringtotal 902 Mbit/s, lowest client 51.8 Mbit/s
checklimit 800 Mbit/s
testPASSED

What the bench controls.Built into Test Environment.

netem
Delay, jitter, packet loss, duplication and reordering on a test host interface.
tbf
A rate limit on a link, to test a slow uplink.
Interfaces
Links up and down, addresses, routes and network namespaces on the test hosts.
Services
DHCP and PPPoE servers on a test host, so the device meets the upstream it expects.
Wi-Fi
Access point and client settings through tsf-wifi, on OpenWrt (UCI) or with hostapd and wpa_supplicant.
Traffic
iperf and iperf3 behind one interface, plus netperf, wrk and TRex. The test reads their results and decides the verdict.
Capture
Packet capture on the test hosts while the test runs.
Switches, power
A switch-ctl agent for managed switches, and power control to cut and restore power to the device.

Reproduce a failure.From a red verdict to a second run.

Example
1 / 4

A run fails

The failures list shows the test, its parameters and the verdict text that the test wrote.

run 417 · bench lab-net · fw 3.1.2 · failures
wifi/connect_ping   FAILED  expected PASSED
verdict  25% of the packets to 192.0.2.1 were lost
1 unexpected of 18
2 / 4

Read what the run used

The record keeps the bench settings, the configuration and the exact commits of the suite and its libraries.

run 417 · configuration
env.lab-net         WIFI_SSID=bench-5g
                    WIFI_SECURITY=wpa2  WIFI_STANDARD=ax
                    TA_WIFI_A_IF=wlan0
builder.conf.lock   suite 9c41e07  tsf-wifi 4e1c9a2
firmware            3.1.2
3 / 4

Start it again

Start the same suite with the same configuration. Midair creates a new record and keeps the first one as it was.

run 418 · same configuration as run 417
wifi/connect_ping   FAILED  expected PASSED
verdict  25% of the packets to 192.0.2.1 were lost
run 417 is kept as it was
4 / 4

Read both records

The same verdict on the same configuration means the failure reproduces. A pass on the second run means it is intermittent, so the test needs more repeats.

wifi/connect_ping · two records
run 417   FAILED  25% lost   fw 3.1.2  lock 9c41e07
run 418   FAILED  25% lost   fw 3.1.2  lock 9c41e07
same verdict, same configuration: it reproduces

Further reading.From our engineering work.

Embedded testing diaryReproducible testsHow resets, device state, network isolation and run fingerprints make two runs comparable. Embedded testing diaryDevice fleetsWhat changes when one test bench becomes many devices. GuideSoftware testing for complex systemsWhy failures appear between components, and how to test for them.

Questions.Short answers.

What is network hardware testing?

Network hardware testing checks how routers, gateways and connected devices handle real traffic and real network faults. Test hosts on each side of the device send traffic through it and check the result.

Does the device need a test agent?

No. Agents run on hosts next to the device. The device needs only its normal network interfaces. A serial console and a controlled power outlet help, but they are optional.

Which network conditions can the bench create?

With netem on a test host: delay, jitter, packet loss, duplication and reordering. The bench can also limit the rate of a link, take a link down and run many clients at once.

Which traffic tools can tests use?

Test Environment has ready wrappers for iperf, iperf3, netperf, wrk and TRex. A test reads their results and decides the verdict.

Can we test Wi-Fi?

Yes. tsf-wifi configures access points and clients on OpenWrt or with hostapd and wpa_supplicant. The example suite joins a network, gets an address and pings through it.

Does it fit our CI?

Yes. Runs start from your CI, for example with the Jenkins plugin. An AI assistant can start runs and read failures through MCP.

Start with one device and one condition.Then add the rest of the bench.

Compare the plans on the pricing page. Prices are monthly, in euros.

Join the waitlist