Embedded testing diary #2: Reproducible tests
A test result only means something if you know what changed between two runs. Here is how reproducibility works in embedded testing: resets, device state, network isolation, and run fingerprints.
A test result only means something if you know what changed between two runs.
If a test passes once and fails the next time even though the firmware is identical, the cause may be elsewhere: the host, network, power, device state or physical conditions around the bench. Until the complete setup can be returned to a known state, a failure does not point reliably to the firmware. It only tells you that something moved.
This is how testing turns into debugging the lab instead of the product.
For embedded systems, reproducibility means being able to restore the same combination of firmware, host environment, network, power, physical state and inputs — and obtain the same verdict. If green turns red, you should be able to identify the variable that changed.
Most of the work lies in controlling the things that drift quietly between runs.
Reset only as far as necessary
A reset is not one operation. It can mean a power cycle, a soft reboot, a factory reset, a full reflash or complete reprovisioning of the device.
Each step restores more state, but also makes the run slower. Reflashing adds another cost: flash memory has a limited number of program and erase cycles, so unnecessary writes shorten its useful life.
The right choice is the least expensive reset that restores the starting conditions required by the test.
One rule makes a large difference across a device fleet: do not flash a device that already has the target version. Check the installed version first and flash only when it differs. If test hosts boot from SD cards, as Raspberry Pi fleets often do, treat those cards as another limited-life component. Netboot or a RAM disk can be a better option where the setup allows it.
Firmware is only one part of device state
A factory reset clears only the state the vendor designed it to clear. The variables that break reproducibility often live elsewhere:
Secrets and pairing: BLE bonding keys, stored Wi-Fi credentials and provisioned certificates can make the next run begin from a different state.
Time: the real-time clock and NTP synchronisation matter when behaviour depends on timestamps, expiry or scheduled events.
Charge and temperature: battery level and thermal history can affect power management, radio behaviour and timing.
Persistent configuration and calibration: NVRAM, configuration partitions and calibration data may survive the reset used by the test.
The host drifts as well. USB devices re-enumerate, so /dev/ttyUSB0 may refer to a different board after a reboot. Devices should be identified by serial number or another stable identity, not by whichever path the operating system assigned first.
Processes may also retain adapters or locks. DHCP leases, ARP caches and firewall rules can carry state from one run into the next. Re-imaging the host before every test is rarely practical, but the environment still needs to reset the parts that can affect the result.
Keep a hardware recovery path
Software control is not enough when the device or operating system stops responding. A controllable power switch for each device — usually a relay board or managed PDU — provides a hardware recovery path and a known power-on sequence.
The power setup itself needs control. A brown-out during a write can corrupt flash and create intermittent failures that are difficult to diagnose, so stable power and a UPS are worth having. Powering on an entire fleet at once can also create enough inrush current to pull down the rail. Sequencing devices avoids turning the recovery mechanism into another source of failures.
Control the network and startup order
A shared network introduces traffic and state that do not belong to the test. An isolated test LAN with predictable addressing makes it easier to know which packets, services and devices were present during a run.
The order of events matters just as much. Many apparently flaky tests are races in the setup: the device starts before the host is listening, or the test begins before the device has finished booting.
Fixed delays hide this problem rather than solve it. Replace sleep 30 with a real readiness check. Wait until the host has opened the connection, the device responds and the required service is healthy — not until the clock says they probably should be.
Record the fingerprint of every run
Every run needs a fingerprint: firmware version or build, host image, device serial number, relevant configuration, network setup, charge, temperature and the exact test inputs.
This record is part of the test result. Without it, reproducing a failure means guessing what the bench looked like at the time. With it, the environment can be restored, a failed run can be compared with a successful one and the changed variable can be traced.
Where this leaves us
This is ordinary work: version checks, power switches, stable device identities, isolated networks, readiness gates and run metadata. Across a fleet, it becomes too many moving parts to manage reliably by hand.
That is why the testing environment needs to own the setup around the test, not only execute the test script. It should restore host and device state, coordinate startup, verify readiness and record what actually ran.
When a run fails, the evidence should point towards the product — not send you back into the lab to work out which part of the bench drifted.