Embedded systems validation for complete system behavior

Embedded system validation checks that the finished device does what its requirements and users need, on real hardware and in its real environment.

Midair runs the validation suite on your bench and keeps every run as a record. Each result links back to the requirement it checks.

Join the waitlist See the traceability graph
validation run 57 · bench-b · fw 3.1.0 Example 
midair  run 57 · builder.conf.lock 3f9c2e1
te      --tester-req="REQ-UPD-03|REQ-PWR-02"
test    update/wifi_credentials_kept
        REQ-UPD-03  FAILED  expected PASSED
console wpa_supplicant: no network configured
test    power/cut_during_update
        REQ-UPD-03 REQ-PWR-02  PASSED
power   outlet 2: off 2 s, on
test    power/cold_boot_x50
        REQ-PWR-02  PASSED
midair  run 57 saved: 3 tests, 1 unexpected

What embedded system validation is.And how it differs from verification and testing.

Validation asks one question: does the finished product do the job it was built for? For an embedded system, the answer comes from the real device, on real hardware, with the network, power and other devices around it.

Verification asks a different question: does each part match its specification? A code review, a static analysis run or a unit test verifies. A field-like scenario on the assembled device validates.

Testing is how you get the evidence for both. A test runs the product, compares the result with an expected verdict and records what happened.

Verification

Did we build the product right?

Checks against
The specification and the design: interfaces, coding rules, register maps, protocol details.
Example on a Wi-Fi gateway
Visao reports an integer overflow in otp.cpp before the code reaches the board.
Leaves behind
Findings, review notes and unit test results for each part.

Validation

Did we build the right product?

Checks against
The requirements and the conditions of real use: the network, power, the devices around it.
Example on a Wi-Fi gateway
The gateway joins the Wi-Fi network within 30 s after a power cut, on the production board.
Leaves behind
A record per requirement: which tests ran, on which build and bench, with which result.

Testing

What does the product actually do?

Checks against
An expected verdict for each test, written down before the run.
Example on a Wi-Fi gateway
wifi/connect_ping associates, gets an address by DHCP and sends four pings. Any loss fails it.
Leaves behind
Verdicts, logs and printouts. They are the evidence that verification and validation both use.

Requirements, tests, evidence.Follow one requirement through a run.

Choose a requirement. The graph shows the tests that carry its tag and the evidence those tests left in the run record.

validation run 57 · bench-b · fw 3.1.0Example

Requirements

Tests and checks

  • wifi/connect_ping PASSED
  • wifi/reconnect_after_ap_restart PASSED
  • update/wifi_credentials_kept FAILED · expected PASSED
  • power/cut_during_update PASSED
  • power/cold_boot_x50 PASSED
  • delta/listening_ports PASSED
  • delta/binary_hardening FAILED · expected PASSED
  • visao/new_findings FAILED · expected PASSED

Evidence

  • Run recordRun 57: result tree, verdicts against trc.xml, parameters
  • Printout0% of the packets to 192.168.4.1 were lost
  • Console logwpa_supplicant: no network configured, after the update
  • Power logOutlet 2 off for 2 s, then on; current trace from a PPK2
  • DeltaListening ports match the baseline
  • DeltaBinary /usr/bin/agent lacks required mitigation: stack canary
  • Visaootp.cpp:20:20 arithm/integer/overflow (MISRA 12.4)
  • Visaootp.cpp:31:5 cpp/string/no_null_terminator (MISRA 21.17)

REQ-NET-01 Joins the configured Wi-Fi network within 30 s

Covered. Every result is as expected.

Tests
2
As expected
2
Unexpected
0
Evidence
2

wifi/connect_ping PASSED
wifi/reconnect_after_ap_restart PASSED

REQ-UPD-03 Keeps Wi-Fi credentials after a firmware update

Covered. Some results are not as expected.

Tests
2
As expected
1
Unexpected
1
Evidence
3

update/wifi_credentials_kept FAILED
power/cut_during_update PASSED

REQ-PWR-02 Boots and reconnects after a power cut

Covered. Every result is as expected.

Tests
2
As expected
2
Unexpected
0
Evidence
2

power/cut_during_update PASSED
power/cold_boot_x50 PASSED

REQ-SEC-04 Listens only on the documented ports

Covered. Every result is as expected.

Tests
1
As expected
1
Unexpected
0
Evidence
1

delta/listening_ports PASSED

REQ-SEC-05 Ships binaries built with a stack canary

Covered. Some results are not as expected.

Tests
1
As expected
0
Unexpected
1
Evidence
1

delta/binary_hardening FAILED

REQ-SW-06 OTP code has no overflow or unterminated strings

Covered. Some results are not as expected.

Tests
1
As expected
0
Unexpected
1
Evidence
2

visao/new_findings FAILED

REQ-LOG-07 Logs the reason for every reboot

Not covered. No test in the suite carries this tag.

Tests
0
As expected
0
Unexpected
0
Evidence
0

Where the links come from

  1. Each test in the suite carries the IDs of the requirements it checks, as a tag in package.xml.
  2. Test Environment can run only the tests for one requirement, for example after a change that touches it.
  3. trc.xml holds the expected verdict of every test. The run record keeps the obtained verdict next to it.
  4. Bublik, the results viewer Midair can publish to, filters results by requirement.

Your team keeps the requirement list itself, in the tool you already use. Midair does not replace a requirements management tool.

tests/update/package.xml<script name="wifi_credentials_kept">
    <req id="REQ-UPD-03"/>
</script>
<script name="cut_during_update">
    <req id="REQ-UPD-03"/>
    <req id="REQ-PWR-02"/>
</script>./run.sh --cfg=bench-b --tester-req=REQ-UPD-03

What one validation run answers.Six questions from a release review.

Every start in Midair is a new record with its own configuration. Midair never writes over a run, so the answers stay true later.

Which build did we validate?
builder.conf.lockbuilder.conf.lock pins the exact commit of every component in the build.
On which bench, with which settings?
env.bench-bThe bench values come from env.<bench> and run.conf.<bench>, and the run keeps them.
What did we expect?
trc.xmltrc.xml lists the expected verdict of each test, so a known failure does not hide a new one.
What happened?
result treeThe result tree shows each test with its parameters, obtained verdict, error lines and printouts. The console log is saved with it.
Did the device itself change?
DeltaDelta compares the device with a golden baseline: processes, ports, files, packages and binary hardening.
Which tests exist at all?
te-docte-doc builds an HTML or PDF catalogue of the suite and a report of the results.

When to run a validation.Five common moments.

  1. Before a release, after the components have passed their own tests.
  2. After a large firmware change, to see whether behavior changed elsewhere.
  3. After a new board revision or a new configuration, with the same suite.
  4. Before a larger roll-out, when more devices and sites depend on the result.
  5. When a regression is hard to explain, to rerun the old record next to the new one.

Questions.Short answers.

What is embedded system validation?

Embedded system validation checks that the finished device meets its requirements and its users' needs in real conditions. It runs on the real hardware, with the network, power and devices the product will meet in use.

What is the difference between verification and validation in embedded systems?

Verification checks that each part matches its specification, for example with code review, static analysis or unit tests. Validation checks that the assembled product does the right job in its real environment. Testing produces the evidence for both.

What is program validation and testing in an embedded system?

Program validation runs the embedded software on its target hardware and checks it against the requirements. Testing is the method: each test has an expected verdict, and the run records what actually happened.

Does Midair trace requirements to tests?

Partly. Tests in a Test Environment suite carry requirement tags, and a run can select tests by requirement. The run record keeps verdicts and evidence per test. Your team keeps the requirement list in its own tool.

Does Midair certify my product?

No. Midair produces evidence: run records, logs, Delta reports and Visao findings. A certification body or your own process decides what that evidence proves.

What do I need to start?

One build, one bench with the device, and a short list of requirements you want to check before a release. TS Factory can draft the first tests from a short description, and you add the requirement tags.

Start with one requirement.Then add the next.

Bring one build, one bench and the requirement you worry about most. Midair records every run, so the evidence is there at release time.

Join the waitlist