Testing firmware tooling without hardware

If you write a tool that talks to firmware, you have a testing problem that application developers do not. Your code under test is only half the system. The other half is a device, and a device needs a board, a cable, a power supply and somebody to plug it in.

That makes continuous integration awkward and it makes edge cases expensive. You can plug in one board. You cannot plug in one board per buffer size.

We publish smp-server-fixtures to solve that for our own tools. They are SMP servers that run as ordinary host processes — some simulated, some emulated — so a client test suite can launch one as a subprocess and drive it over a real transport, with no hardware involved.

This post covers what they do, what they do not replace, and the handful of details that turned out to matter. The protocol here is SMP, but the shape of the problem is the same for any tool that talks to a device.

What is SMP?

SMP is the Simple Management Protocol, the thing MCUmgr speaks. It is how you manage a microcontroller remotely: read its state, move files on and off it, and upload new firmware to it. In Zephyr it is the standard route for an over-the-air update.

The protocol is split into command groups — os for echo, reset and device info, img for firmware upload, fs for file transfer, stat, shell, settings, enum and Zephyr's own zbasic. A device enables some of them and not others.

We maintain two clients for it, smpclient and smpmgr — described in full under smpclient and smpmgr. Those are the things under test.

Why testing a client against real hardware is hard

The interesting failures are not in the happy path. They are in fragmentation, reassembly and buffer arithmetic, and those depend on numbers fixed when the server is built.

The server advertises a buffer size. The client has to split a message into fragments that fit, send them, and have the server put them back together. Get the arithmetic wrong by one and it works at 512 bytes and fails at 400.

So the thing you want to test against is not "a device". It is a device at buf_size 96, and another at 128, 256, 400, 512, 1024 and 2048. Four hundred is in there because it is not a multiple of the client's default 128-byte line length, and 96 because it is below it. Those are the two cases where off-by-one errors live.

That is seven boards for one dimension of one transport, before you have tested IPv6, or the shell transport, or what happens when the device refuses to say what its buffer size is.

Running one

A native_sim fixture is a Linux executable. It prints its pseudo-terminal on boot and then speaks SMP down it:

$ ./zephyr_*_native_sim_serial.exe
uart connected to pseudotty: /dev/pts/7
$ smpmgr --port /dev/pts/7 os echo hello

The UDP fixtures bind a host socket instead, at 127.0.0.1:1337, or [::1] for the IPv6 build. QEMU fixtures take a command line with the image as a loader. That is the whole interface: a test suite starts the process, talks to it, and kills it when the test ends.

What native_sim and QEMU each give you

Zephyr's native_sim target compiles the firmware as a native executable for the host. It is not emulation; it is the same application code linked against a host implementation of the OS layer. It is fast, and it is where most of the matrix lives.

We build it 32-bit on purpose. On Zephyr v4.4.0 the 64-bit native_sim pseudo-terminal UART does not deliver received bytes to the MCUmgr transport, so the fixture boots and then never answers. The fixtures themselves track a post-v4.4.0 Zephyr commit, pinned by SHA in west.yml and carried in every artifact's filename, because MCUboot's raw serial recovery had not landed in the release.

QEMU targets run the real binary under emulation, so the instruction set is a Cortex-M one and the timing is paced by an emulated UART rather than by how fast the host can write to a pipe. qemu_cortex_m0 is an nRF51 with 16 KB of RAM, which is enough for a merged MCUboot plus application — so it is the fixture that exercises a genuine firmware upload — and not enough for much else.

The fixtures

Two of those put the client in a position it would otherwise never reach, and a third works around the simulator.

serial_noparams turns off the MCUmgr params command, so the client cannot read the server's buffer size and has to fall back to its defaults. It makes the client take a branch nothing else takes. You cannot easily build that situation on a bench; you can build it as a fixture once.

serial_line512 accepts 512-byte frame lines, so a client configured with a line length other than its 128-byte default has something to talk to.

serial_bigrx is the simulator one. native_sim's pseudo-terminal has no baud pacing, so fragments arrive back to back, overrun the default two-by-128-byte receive pool, and the server logs uart_mcumgr: Insufficient buffers, fragment dropped. A 32-by-512 pool lets it receive a full multi-fragment message. That config is folded into most of the native_sim serial fixtures, not only the one named after it. It is a property of the simulator rather than of the protocol, and without it the drop looks like a client bug.

There is a matching trap on the QEMU side, which is why the launch commands use a TCP socket rather than a pseudo-terminal: QEMU's emulated UART holds the final byte of a frame over a PTY, stalling reassembly part way through. A -chardev socket does not. The manifest carries the socket form, and the client mirrors it.

Why a Cortex-M3 with four megabytes of SRAM

native_sim is fast but bursty, so a test that passes there has not proved much about timing. qemu_cortex_m0 paces properly but cannot hold a large buffer and a file system at once.

mps2/an385 is QEMU's Cortex-M3 development board model, and the machine comes with 4 MB of SRAM. That is the fixture that round-trips a real buffer-filling message at 2048 bytes, three ways — an os echo, an fs upload, and an img upload — on a target that paces the link.

Having the room is what makes the do-everything image possible. A pair of devicetree overlays — one for MCUboot, one for the application, which have to agree — carve the SRAM into four regions: 256 KB for MCUboot while it runs, 2 MB for the flash simulator's backing store, about 1.7 MB for the application to execute from, and 1 KB at the top for retained memory. It also lays an MCUboot partition table and a littlefs partition into the simulated flash.

The boot path is not how a real product boots, so here it is in full. The flash simulator is not memory-mapped for execute-in-place, so MCUboot uses RAM_LOAD: it copies slot 0 into SRAM and runs it from there. Signing is off on both emulated targets, though the artifact is still named .signed.bin. QEMU starts it with two loaders — MCUboot's code-only hex, plus the application dropped straight into slot 0's backing store at 0x20050000.

Once running, the app serves SMP on uart0 and logs on uart1, so uart0 stays a clean channel while uart1 shows the whole lifecycle. Sending os reset boot_mode=1 writes a boot mode to retained memory and drops back into MCUboot's serial recovery on the next boot. An upload done in recovery survives the soft reset, because the slots live in a RAM-backed flash simulator that is not erased at boot. One image, both halves of the update path.

How a test suite finds the fixtures

Every release ships a manifest.json — one entry per fixture, so a client builds its test registry rather than hardcoding rows that go stale.

Each entry carries the filename, the target and config, the transport, the IP family, the app's buf_size and buf_count, the bootloader's advertised recovery_buf_size and recovery_buf_count where it has them, the maximum serial line length, the UDP port, the enabled command groups, whether the image carries MCUboot and whether it is a recovery build, and the command to launch it — run for a native_sim executable, qemu_cmd for an emulated one.

The bootloader's buffer size and the application's buffer size are different numbers, and a client reads a different one depending on whether the device is in recovery. The manifest keeps them in separate fields for that reason. A test registry that conflates them will have a test that passes for the wrong reason.

Releases are tagged by commit SHA, and a CI check refuses to publish unless every asset carries that build's SHA and every image has a manifest entry. A release cannot mix builds or ship a fixture the manifest does not describe.

What emulation does not cover

Everything physical, which is most of what goes wrong in the field.

It will not find a marginal signal, a brown-out part way through a flash write, a USB-to-serial adapter that drops bytes under load, a cable, a connector, or a device that resets while you are talking to it. It will not tell you anything about power. And on the emulated recovery fixture the flash, the signing and the loader are all stand-ins — the Cortex-M instruction set is the only part that is real.

What it does cover is the protocol logic, the buffer arithmetic, the fragmentation and reassembly, the fallback paths, and the state machine around a firmware update. Those are cheap to get wrong, expensive to debug on a bench, and perfectly deterministic once you can run them a thousand times in CI.

We still test on hardware. The fixtures mean we are testing the right things on it.

The fixtures are at smp-server-fixtures and the clients at smpclient and smpmgr, all open source. If you are building device tooling and wondering how much of it you can test without a bench, the answer is usually more than you think.