The Build Passed, So Why Doesn't It Run — Automating Firmware Tests on Real Hardware

TL;DR

• A passing west build doesn't mean the firmware runs on hardware — catching silent failures manually doesn't scale.


• Combining Zephyr Twister's --device-testing mode with a self-hosted runner gives you automated serial-log-based testing on real boards with every push.


• All you need to start is a Raspberry Pi (or even a Windows PC) and a J-Link.

The most deflating moment in firmware development is when west build passes cleanly, you flash the board, and nothing happens. No serial output, no LED activity — just a Hard Fault dump, or worse, dead silence.

A successful build means "the code has no syntax errors." It does not mean "the firmware behaves as intended on hardware." Verifying that gap requires plugging in a J-Link, flashing, opening a serial terminal, and reading the logs by hand. Repeat this for every change, and eventually you start cutting corners: "I'll just spot-check this one." Those shortcuts compound, and regression bugs creep in.

This post documents how I automated that manual verification. On every push, the firmware is automatically flashed to a real board, serial logs are captured, and pass/fail is determined — a HIL (Hardware-in-the-Loop) CI (Continuous Integration) pipeline. I built it with Zephyr's reported that 45% of developers spend more time debugging AI-generated code than writing it themselves.

Second, manual testing doesn't scale. Fixing one feature can break another. In web development, automated test suites catch these regressions. My firmware workflow had no such safety net. Testing every feature manually after every change is impractical, so I'd only verify "the part I just touched." That's exactly how regression bugs get in.

Third, repetition creates friction. Plug in J-Link, wait for flash, open serial monitor, scan for log patterns, record the result. Two to three minutes each time, but across dozens of iterations per day with the AI loop, the cumulative cost adds up. The real damage, though, is the temptation to skip it. And the code you skip testing on is the code that causes problems later.

The first three stages of the loop — research, planning, execution — were already efficient thanks to AI collaboration. But as long as the last stage was manual, it bottlenecked the entire pipeline. The missing piece was test automation. And in embedded, test automation means putting real hardware in the loop — HIL.

Wait — Can't I Just Test on the Board Locally?

"I already have a board on my desk and I'm running west flash — why bother with CI?" I thought the same thing at first.

Local testing and HIL CI perform the same physical actions (flash, check serial logs), but the implications differ:

It's the same difference as running npm test locally versus having GitHub Actions run it on every PR. Local testing is a snapshot of "what I verified right now." CI testing is a gate that "every change must pass before merge."

This difference matters especially for firmware because regressions take far longer to surface. A web service shows error rate spikes on a monitoring dashboard immediately after deploy. Firmware can silently malfunction — intermittent BLE disconnects, failing to wake from sleep under specific conditions. You may not find out until a customer reports it. CI verifying basic behavior on every commit means that at minimum, you can git bisect to find "when exactly did it break."

Embedded CI Is Not Web CI

Web backend CI is comparatively straightforward. Push code, a cloud VM spins up, installs dependencies, runs tests, reports results. The VM starts clean every time, so environment-related flaky tests are relatively rare.

Embedded CI is fundamentally different.

The Limits of QEMU

Zephyr has a built-in test runner called (an open-source hardware emulator). Testing without a physical board, straight from a CI server — that's appealing. The Zephyr project itself runs thousands of QEMU-based Twister tests.

But QEMU's coverage has hard limits:

QEMU's driver model doesn't cover every edge case — certain behaviors are considered unnecessary in an emulated environment. The core functionality of most product firmware sits in the right column. The reality is: "Most firmware is too tightly coupled with hardware for emulation to be the only path forward — at some point, the dev board is the only way to make progress."

. But no matter how advanced emulators get, reproducing BLE RF paths or real sensor analog characteristics remains fundamentally difficult.

Variables That Only Physical Hardware Creates

Real-board testing introduces variables that don't exist in emulation:

• 
  Timing: Virtual time in an emulator and physical time on real hardware flow differently. A 100ms timeout can pass in QEMU and fail on the board.


• 
  Power: Unstable USB hub power can reset the board or interrupt flashing mid-process. The CI log just says "connection lost."


• 
  RF environment: BLE tests are affected by ambient Wi-Fi interference. The same code can pass at the office and fail in the server room.

These variables create flaky tests. In web CI, flaky tests are mostly async timing issues fixable by code changes. In embedded CI, flaky tests are often caused by the physical environment — no amount of code changes will eliminate them.

That's the reality. Embedded CI is not a world where "correct code guarantees passing tests." But it's still better than manual testing. "Imperfect but automated verification" is more reliable in practice than "thorough but human-dependent verification." I decided to build a HIL CI pipeline.

Pipeline Design — How Far to Automate

Self-hosted Runner: The Common Pattern for Connecting Physical Boards to CI

,

Self-hosted runner bridges the cloud CI platform and the physical board

I chose a Raspberry Pi 4 as the runner. The reason is simple: low power consumption for 24/7 operation, four USB ports for connecting multiple boards, and ARM Linux where the Zephyr toolchain runs natively. [TBD: Need to add actual Raspberry Pi performance/stability experience after use]

You Don't Need a Raspberry Pi

"Do I have to buy a Raspberry Pi?" No. A self-hosted runner is any machine that can run the CI agent software. A Linux desktop, a macOS laptop, even a Windows PC works.

Using a Windows PC as a runner:

, and all officially support Windows. Install

The complete sequence from a single git push through build, flash, test, and verdict

Breaking it down:

Steps 1-3: Cloud. The developer pushes code. The CI platform reads the YAML, finds a matching runner, and dispatches the job. At this point, the code only exists in the cloud.

Steps 4-5: Runner build. The runner checks out the source and cross-compiles with west build. Build logs are generated here. If the build fails, it stops and the error log is uploaded to the cloud. In the split Docker architecture, this step runs on a cloud runner (amd64).

Steps 6-8: Physical interaction with the board. On a successful build, the runner uses nrfjprog to flash the firmware via USB/J-Link. The board resets, boots the new firmware, and outputs logs through the UART serial port. This log capture is the core of HIL — the runner opens the board's serial port (/dev/ttyACM0 or COM3 on Windows) and reads the output in real time.

Step 9: Verdict. Twister matches the captured serial log against regex patterns defined in testcase.yaml. If "Feature initialized successfully" appears within the timeout, it's a pass. Otherwise, fail.

Steps 10-11: Reporting. The runner uploads the verdict and log files to the cloud. The CI platform marks the PR with a check (pass or fail). On failure, serial logs are attached as artifacts for the developer to download and analyze.

Where logs are generated:

Reproducing the Build Environment with Docker

The most common CI failure is "it works on my PC but not in CI." The standard solution for Zephyr/NCS projects is Docker.

Nordic provides an official Docker image called (nordicplayground/nrfconnect-sdk). It contains every dependency needed to run west commands — Zephyr SDK, Python venv, west manifest. You pull this image and use it as the build environment; you're not uploading your code to Docker Hub. It's the same idea as apt install for the compiler.

One caveat: this official image is amd64 (x86_64) only. A Raspberry Pi is ARM64 and can't run this image directly. So the CI pipeline splits into two stages:

's west.yml, so running west init and west update inside the Docker image reproduces the exact same environment as my dev PC. Accessing USB devices from inside a Docker container requires the --device flag, and its behavior varies subtly across platforms — which is another reason I chose the split architecture.

HIL CI Works Without T2 Topology Too

The example above assumes T2 topology (a west.yml manifest at the project root). But HIL CI itself doesn't require T2. All you need is "a buildable project" and "a board to flash."

The build method in CI varies by project structure:

The simplest way to put a freestanding project into CI is to pre-install the NCS SDK on the runner machine and set ZEPHYR_BASE:

# Freestanding project CI example (GitHub Actions)

jobs:

  hil-test:

    runs-on: self-hosted  # runner with pre-installed SDK

    env:

      ZEPHYR_BASE: /home/runner/ncs/v2.9.0/zephyr

    steps:

      - uses: actions/checkout@v4

      - run: west build -b nrf52dk/nrf52832

      - run: west twister --device-testing --hardware-map hardware-map.yml -T tests/

The downside: the SDK version is tied to the runner machine. Updating the runner's SDK affects every project. That's exactly why T2 topology uses west.yml to pin SDK versions independently per project. But if you have a single project and just want to get CI running, freestanding is enough. You can upgrade the structure later.

Precedent: Golioth's Implementation

The implementation I referenced most while designing this pipeline was periodically logs each thread's stack usage:

[00:00:05.000] <inf> thread_analyzer:  main    : STACK: unused 512 usage 1536 / 2048 (75 %); CPU: 12 %

[00:00:05.000] <inf> thread_analyzer:  ble_rx  : STACK: unused 128 usage 896 / 1024 (87 %); CPU: 3 %

"unused 128" means only 128 bytes of stack headroom remain. You can pattern-match this and fail when headroom drops below a threshold — catching stack growth early as the AI adds code.

What this approach can't catch

Serial log pattern matching only verifies "logs I predicted in advance." Unexpected failures — BLE disconnecting after 30 minutes, sensor values drifting at certain temperatures — won't be caught unless you build tests that reproduce those specific conditions.

Real-time interactive debugging is also outside CI's scope. "Watch serial output while pressing a button at a specific moment" is still a desk job. CI's role is "automatically re-verify known correct behavior on every commit," not "discover new problems." When you do discover a new problem, you write a test for it and add it to CI — that's how test suites naturally grow thicker over time.

Automatable Tests vs. Non-automatable Tests

Not everything can be automated with HIL. Drawing the boundary clearly matters.

Automatable:

• UART/RTT log output verification (string pattern matching)


• State machine transition checks (log state changes, verify sequence)


• Boot time measurement (timestamp-based)


• I2C/SPI device response checks (when sensors are physically connected)


• Memory usage reports (parsing the .map file generated at build time)

Difficult or impossible to automate:

• BLE RF performance (RSSI, packet error rate) — requires dedicated test equipment


• Analog sensor accuracy — requires a reference input source


• Power consumption measurement — requires a current probe (Zephyr 4.2 added a power measurement harness to Twister, but it needs physical measurement hardware)


• Long-duration stress tests — hits CI execution time limits


• UI/display output — camera-based verification is possible but complex ( produces this workflow:
  
  
  

  : "AI's accuracy is highest when analyzing logs." Logs are factual data, which leaves little room for hallucination. The same applies to CI-captured serial logs. Hand the AI a Hard Fault register dump, stack trace, and error codes, and it provides reasonably accurate analysis: "this address corresponds to this function at this offset, and the probable cause is X."
  
  

  
  
  
  
  

  CODE Kopieren

  # Workflow example: save logs on CI failure (GitHub Actions)
  
  - name: Save failure logs
  
    if: failure()
  
    run: |
  
      cp twister-out/*/handler.log artifacts/
  
      cp twister-out/*/device.log artifacts/
  
  
  
  - name: Upload artifacts
  
    if: failure()
  
    uses: actions/upload-artifact@v4
  
    with:
  
      name: failure-logs
  
      path: artifacts/
  
  

  
  
  
  
  
  
  
  

  Feeding the saved logs to Claude Code:
  
  

  
  
  
  
  

  CODE Kopieren

  # Request AI analysis of failure logs locally
  
  claude "This Twister test failed in CI. Analyze device.log." \
  
    @artifacts/device.log
  
  

  
  
  
  
  
  
  
  

  This loop isn't fully automated yet. There's manual intervention between CI failure, log download, and handing it to the AI. Tools like — a script that greps build/zephyr/.config and Kconfig sources to catch nonexistent symbols when a .conf file is modified — also works in CI.

  
  
  

  The approach is straightforward. Include the hook script in the repo and run it before the build step in the CI workflow:
  
  

  
  
  
  
  

  CODE Kopieren

  # Run Kconfig validation hook in CI
  
  - name: Validate Kconfig
  
    run: |
  
      west build -b nrf52dk/nrf52832
  
      ./scripts/validate_kconfig.sh prj.conf build/zephyr/.config
  
  

  
  
  
  
  
  
  
  

  Claude Code's skill fires when the AI modifies a .conf file; the CI validation catches it when a human edits .conf manually too. The same validation logic, running at two points. Tools created during AI collaboration naturally extending into CI infrastructure — that's the compounding effect of the pipeline built across this series.

  
  
  
  
  
  

  
  
  
  Remaining Gaps and Next Steps
  

  
  
  

  
  
  
  What HIL CI Still Can't Catch
  

  
  
  

  I need to be honest. Adding HIL CI doesn't mean every hardware problem is automatically caught:

  
  
  
  
  
  • 
    RF performance: BLE connection stability, RSSI, and packet error rate require measurement equipment (sniffer, spectrum analyzer). Serial logs only tell you "connection succeeded/failed," not "why it failed."
  
  
  • 
    Long-term stability: Memory leaks and stack overflows only surface after hours or days of operation. CI workflows typically run for minutes to tens of minutes — too short to catch these.
  
  
  • 
    Power consumption: Current profiles of sleep/wake cycles can't be measured without a current probe.
    Development environment
    
    
    2
    
    AI tooling
    
    
    4
    Research → Plan → Execute → Test Loop
    AI workflow
    
    
    5
    HIL CI (this post)
    Automated verification