Hardware-software integration testing for embedded systems

Midair tests firmware where it meets the board: power, buses, sensors and the boot sequence. Each run keeps the console output, the verdicts and the bench setup.

A test host next to the device switches power, drives lines and buses, and reads the serial console. The device itself needs no test agent.

Join the waitlist Inject a fault

What hardware-software integration testing is.Software, checked against the real board.

Hardware-software integration testing checks that firmware, drivers and applications work with the hardware around them. That means the board, its buses, its sensors and its power supply.

Hardware-in-the-loop (HIL) testing is the usual way to do it. A test host plays the outside world, changes conditions on purpose and records how the device reacts.

Most integration failures depend on timing, order and state. So a useful run keeps the build, the bench and the console output next to each verdict.

Hardware-in-the-loop testing.Inject a fault, follow the signal.

Choose a fault. The diagram shows the path of the signal between the test host, the bench and the device. The panel shows what the test records.

Bench: peripherals and environment PDU outlet power Serial console UART GPIO · I2C · SPI sensor and reset lines CAN bus other nodes Test host unix agent power-ctl agentrun record Device under test no agent on the device PDU UART GPIO·I2C CAN Test hostunix agentpower-ctl agent Device under test no agent on the device Bench: peripherals and environment

Power cut

The host does
The power-ctl agent switches the outlet off in the middle of a flash write. Two seconds later it switches the outlet on again.
Signal path
  1. power-ctl agent
  2. PDU outlet
  3. device
  4. serial console
  5. test host
The test records 
bench hil-1 · run 518 · Examplepower    outlet 3: off during flash write
power    outlet 3: on after 2.0 s
console  U-Boot 2024.01
console  ubifs: recovery needed, replaying journal
test     power/cut_during_write    PASSED  expected PASSED

Bus error

The host does
The unix agent puts 500 unexpected frames on the CAN bus in one second. The device has to keep sending its heartbeat.
Signal path
  1. unix agent
  2. CAN bus
  3. device
  4. serial console
  5. test host
The test records 
bench hil-1 · run 518 · Examplecan      can0: 500 frames, id 0x0F0, in 1.0 s
can      heartbeat 0x701: gap of 1.8 s, limit 0.5 s
console  canbridge: rx queue full, frames dropped
test     can/burst_keeps_heartbeat FAILED  expected PASSED

Late sensor

The host does
The host holds the data-ready line of the sensor low and raises it 250 ms late. The device has to report the timeout.
Signal path
  1. unix agent
  2. GPIO line
  3. device
  4. serial console
  5. test host
The test records 
bench hil-1 · run 518 · Examplegpio     line 17 (DRDY): low, raised after 250 ms
console  sensor: read 0x0000, value accepted
test     sensor/late_ready_reported FAILED  expected PASSED
         no timeout reported by the device

Reset

The host does
The host pulls the reset line low for 100 ms. The test measures how long the service needs to come back.
Signal path
  1. unix agent
  2. reset line
  3. device
  4. serial console
  5. test host
The test records 
bench hil-1 · run 518 · Examplegpio     line 5 (nRESET): low for 100 ms, then high
console  U-Boot 2024.01
console  Starting kernel ...
test     reset/service_back_in_10s  PASSED  7.4 s

The boot sequence.Six stages, five hand-overs.

Integration failures gather where one stage hands over to the next. Choose a point on the timeline to see what fails there and how a test catches it.

Power on · 0.0 s

Slow power rail

What fails
The supply rises slowly, and the processor starts before its power controller is ready. Some boots hang and some do not.
What the console shows
(nothing: the console stays silent)
How the test catches it
The power-ctl agent cycles the outlet 50 times. Each cycle must reach the U-Boot banner on the console within two seconds.
Bootloader · 0.6 s

Wrong device tree

What fails
The bootloader loads the device tree of another board revision. The kernel then looks for peripherals at the wrong addresses.
What the console shows
Loading fdt: board-rev-a.dtb
How the test catches it
The test compares the device tree name on the console with the board revision in the bench configuration.
Kernel · 2.4 s

Root file system not found

What fails
After an update, the kernel cannot mount the new root partition and stops.
What the console shows
Kernel panic - not syncing: VFS: Unable to mount root fs
How the test catches it
Test Environment watches the console for the panic pattern. It stops the test, saves the console log and power-cycles the board.
Drivers · 3.8 s

Driver before its regulator

What fails
A sensor driver starts before the voltage regulator of the sensor is on. The sensor stays missing until the next reboot.
What the console shows
tmp102 1-0048: error -110 reading temperature
How the test catches it
The test lists the expected devices after each of 20 cold starts. One missing sensor in one start fails the test.
Services · 5.1 s

Service starts too early

What fails
A service starts before the CAN interface is up. It exits with an error and does not start again.
What the console shows
canbridge.service: Failed with result exit-code
How the test catches it
The test checks the state of each service after boot and saves the service log when one has failed.
Application · 8.0 s

Watchdog reset

What fails
The application takes longer to start than the watchdog timeout. The board resets and starts again, in a loop.
What the console shows
Reset cause: WDOG
How the test catches it
After boot, the test reads the boot status of the watchdog. A watchdog reset during the last boot fails the test.

What a run keeps.The bench is part of the result.

Test Environment reads the bench from configuration files in the suite. Midair stores every start as a new record and never writes over an old one.

conf/rcf.conf
The test agents and the hosts they run on, including the power-ctl agent.
conf/env.<bench>
The values of one bench: addresses, serial ports and outlet numbers. They come from environment variables.
conf/run.conf.<bench>
Which tests and parameters run on that bench.
conf/cs.conf.*
The setup of hosts and interfaces for the Configurator, in YAML.
conf/trc.xml
The expected verdict of every test, so a regression stands out.
conf/builder.conf.lock
The exact commits of the suite and of its libraries.

Next to these files, the record keeps the console log, the kernel log, the result tree and the list of failures.

Questions.Short answers.

What is hardware-software integration testing?

Hardware-software integration testing checks that firmware and software work with the physical hardware around them. It covers buses, sensors, power, timing and the boot sequence.

What is hardware-in-the-loop testing?

Hardware-in-the-loop (HIL) testing runs the real device in a loop with a test host. The host drives the inputs of the device, changes its power and buses, and records how it responds.

How is HIL different from integration testing in general?

Integration testing is the wider practice. HIL is one way to do it, with the real board on a bench. Tests in an emulator such as QEMU and tests across several devices on a network are other ways.

Does the device need a test agent?

No. Test agents run on a host next to the device. The host reaches the device through switched power, the serial console, GPIO, I2C, SPI, CAN, JTAG or the network.

Which faults can a test inject?

A test can cut power through outlets on PDUs, relays or USB hubs. It can drive GPIO lines, run I2C and SPI transfers, send CAN frames and reset the board. The serial console shows how the device reacts.

Can we start before the board is ready?

Yes, in part. The same suite can run on QEMU with snapshots, and Visao analyzes the firmware source without running it. Tests of real peripherals wait for the board.

Start with one board and one fault.Then add the rest of the bench.

Bring one firmware build and one integration failure that is hard to reproduce. Compare the plans on the pricing page. Prices are monthly, in euros.

Join the waitlist