IoT device testing for connected products and device fleets

Midair tests connected devices on the bench and checks the fleet in the field. Each device is compared with a baseline and with the devices of the same model.

Delta runs on the device and reports what differs. Test Environment recreates the network conditions of a real day in the lab.

Join the waitlist See the fleet

What IoT device testing is.The device, its network and its fleet.

IoT device testing checks a connected device together with its firmware, its network and the services it talks to.

One device on a desk can pass every test. A thousand devices in the field run different firmware versions, lose their network and change over time.

So IoT testing has two halves. In the lab, tests recreate field conditions on purpose. In the field, checks find the device that no longer looks like the others.

The fleet view.Find the device that differs.

An example fleet of 40 devices in three models. Filter by model or firmware, then choose a device to see how it differs from its baseline.

delta --fleet · 40Example

Model

Firmware

• Matches the baseline
• Drift since the last check
• Fleet outlier

40 / 40 shown · outliers: 4 · with drift: 3

delta · h1-07 · H1 · fw 3.2.0Example

Fleet outlier

1 of 16 H1 hubs runs a telnet server. No other hub in the cohort has it.

Appeared

• proc:telnetd
• port:23/tcp/listen

Missing

• nothing

delta · h1-12 · H1 · fw 3.2.1Example

Drift

After the update to 3.2.1, one SSH server package replaced another. The baseline still lists the old one.

Appeared

• pkg:dropbear@2022.83

Missing

• pkg:openssh-server@9.6p1

delta · s2-05 · S2 · fw 3.1.0Example

Fleet outlier

The SSH configuration file has a checksum that no other S2 sensor has.

Appeared

• file:/etc/ssh/sshd_config@9be2c41f

Missing

• file:/etc/ssh/sshd_config@a41c07d3

delta · s2-11 · S2 · fw 3.2.0Example

Fleet outlier

This sensor sends its default route through a tunnel interface. The rest of the cohort does not.

Appeared

• netif:tun0
• route:0.0.0.0/0->10.8.0.1

Missing

• route:0.0.0.0/0->192.168.1.1

delta · s2-14 · S2 · fw 3.2.1Example

Drift

Since the last check, port 1883 is open on this sensor and an MQTT broker listens on it.

Appeared

• port:1883/tcp/listen
• proc:mosquitto

Missing

• nothing

delta · g3-03 · G3 · fw 3.2.0Example

Drift

A cron file appeared between two checks. Its mode lets any user change it (ATT&CK T1053.003).

Appeared

• file:/etc/cron.d/backup mode 0666
• proc:backup.sh

Missing

• nothing

delta · g3-06 · G3 · fw 3.2.1Example

Fleet outlier

Every other G3 gateway runs the watchdog daemon. This one does not.

Appeared

• nothing

Missing

• proc:watchdogd

delta · Choose a device in the grid.Example

Matches the baseline

No differences from the baseline.

Appeared

• nothing

Missing

• nothing

• Delta collects a baseline from each device: processes, ports, packages, files, interfaces and routes.
• For outliers, Delta compares devices of the same model. A feature that few devices have shows up under Appeared. A feature that almost all devices have, and this one lacks, shows up under Missing.
• For drift, Delta checks each device on a timer and reports what changed since the last check.
• In Midair, an expected change can be accepted into the baseline. After that, it no longer shows up.

A day in the life of one device.Five conditions to recreate in the lab.

Choose a moment in the day of an example device. The panel shows the condition, how Test Environment recreates it and what the test checks.

1. 00
2. 06
3. 12
4. 18
5. 24

02:00Update window

In the field

The device downloads a firmware update over a slow link. The link drops halfway through.

In the lab

netem delay 400ms loss 2% · link down at 60 %

The test checks

The device ends on one complete firmware version and keeps its configuration.

07:30Router restart

In the field

The Wi-Fi access point restarts and is gone for 90 seconds.

In the lab

access point interface down for 90 s, then up

The test checks

The device associates again within 30 seconds, gets an address over DHCP and answers four pings with no loss.

12:30Busy network

In the field

Midday traffic adds delay and jitter, and some packets arrive out of order.

In the lab

netem delay 250ms jitter 80ms reorder 5%

The test checks

Telemetry reaches the broker in order, with no duplicates and no gaps.

18:00Weak signal

In the field

The signal gets weak in the evening. Packets are lost in bursts for ten minutes.

In the lab

netem loss 15% correlation 50%

The test checks

The device waits longer between attempts and reconnects. It does not flood the server with new sessions.

23:59Token expiry

In the field

The access token the device got in the morning expires at midnight.

In the lab

tsf-time: the client runs under a clock 24 h ahead

The test checks

The device gets a new token before the old one expires, and no data is lost.

What an IoT test covers.From the first boot to a year in the field.

1. Provisioning and first connection

   The device joins the network, gets an address and registers with its service. The test checks each step against a time limit.

2. Reconnection

   Links drop, access points restart and addresses change. The test checks that the device comes back on its own.

3. Firmware updates

   An update can stop halfway or reach a device with old settings. The test checks the version and the configuration after each attempt.

4. Traffic to and from services

   Test modules speak MQTT, CoAP and HTTP to the device, or stand in for its server. So the test sees both sides of the exchange.

5. Long runs

   Delta checks the device on a timer and reports drift between checks: new processes, open ports, packages and changed files.

Questions.Short answers.

What is IoT device testing?

IoT device testing checks a connected device with its firmware, its network and the services it uses. It covers tests on the lab bench and checks across the fleet in the field.

How does Midair find an outlier in a fleet?

Delta collects a baseline from each device. It compares devices of the same model and reports features that few devices have, or that this device lacks.

What is drift?

Drift is a change on one device between two checks. Delta runs on a timer and reports new or removed processes, ports and packages, and changed file checksums.

Does Delta change anything on the device?

No. Delta reports the differences and changes nothing on the device. It is a static binary of about 104 KB for embedded Linux on ARM, RISC-V, MIPS and x86_64.

Can the lab recreate network problems?

Yes. Test Environment sets delay, jitter, loss, duplication and reordering on a host interface. It can also take links down and up during a test.

Does the device need a test agent?

No. Tests reach the device through a host next to it. Delta runs on the device only when you want the device itself inspected, and Midair deploys it over SSH.

Related pages.Capabilities and reading.

All test levelsEmbedded systems testingFrom a workstation to a fleet, with one record per run. One firmware buildFirmware testing servicesFlashing, boot, power loss and the bench under test control. Firmware meets hardwareHardware-software integration testingHardware-in-the-loop tests and the boot sequence. Gateways and routersNetwork hardware testingRouters, gateways and tests that need many hosts. SecurityEmbedded security testingSecurity-sensitive behavior in firmware, devices and their interfaces. Embedded testing diaryDevice fleetsMixed device setups, several test hosts and devices that two tests want at once. Embedded testing diaryReproducible testsHow resets, device state, network isolation and run fingerprints make two runs comparable.

Start with one device model.Then add the fleet.

Bring one device, one firmware build and one field failure that is hard to reproduce. Compare the plans on the pricing page. Prices are monthly, in euros.

Join the waitlist