Production, Provisioning, and the Field

Firmware that works on your desk is about a third of the job. The rest happens when the board is new and nobody knows whether the hardware is right, when ten thousand units go through a factory that has 40 seconds per unit, when every device needs its own serial number and keys, and when a fleet runs for years and you must diagnose a fault on a device you will never touch. This lesson covers hardware bring-up, manufacturing test firmware, factory programming and provisioning, locking devices, release engineering, OTA fleet management, field diagnostics, regulatory compliance, long-term support, and post-mortems.

Hardware bring-up with the EE

The first prototype boards arrive. Some fraction of them, and some fraction of the design, is wrong. Bring-up is the process of finding out which parts work, and firmware engineers do much of it, together with the electrical engineer (EE) who designed the board.

Reading schematics and datasheets efficiently

You do not need to design circuits to bring up a board, but you must read the schematic fluently. Find, for every peripheral you will touch: which MCU pin it uses, what is between the pin and the device (series resistors, level shifters, pull-ups), how the device is powered and reset, and which pins are strapping or boot pins.

Datasheets are long. Read them in an order that finds problems fastest:

1. The errata sheet first. Silicon bugs that affect your design are listed there, with workarounds. Reading it after a week of debugging is a rite of passage you can skip. The STM32F407 errata sheet, for example, documents I2C and other peripheral limitations that explain otherwise baffling behaviour.
2. The block diagram and pinout, to orient yourself.
3. Electrical characteristics: supply range, current consumption, input thresholds, timing parameters. This is where “it works at room temperature but not in the cold chamber” is explained.
4. The register map and functional description of only the blocks you use, when you write the driver.

The bring-up checklist

Work bottom-up, never skipping a step, because every later step depends on the earlier ones:

Step	Check	Tool
1. Power	Every rail at the right voltage, within tolerance, no excessive current at power-on, correct sequencing	Bench supply with current limit, multimeter, scope
2. Clocks	Crystal oscillates (HSE), correct frequency on MCO pin	Scope (probe the MCO output, not the crystal: the probe’s capacitance can stop it)
3. Reset	NRST released, reset supervisor behaves, no reset loops	Scope on NRST
4. Debug	SWD connects, can halt, read IDCODE, read and write RAM, program flash	Probe, OpenOCD
5. UART	Console prints a banner at the right baud rate	USB-serial adapter
6. Peripherals	One at a time: GPIO, then each bus, then each external chip	Logic analyzer, scope, firmware tests

The first firmware you load on a new board is not the product firmware. It is a minimal bring-up image: clocks, a UART console, and a shell of test commands you can run one at a time. Product firmware on an unproven board turns every hardware fault into a confusing software symptom.

Finding board errors with firmware

Firmware can find assembly faults that are hard to see. A pin test drives each pin in turn and checks that its neighbours do not follow, which finds solder bridges between adjacent pins:

pin_test.c

#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include "gpio.h"

typedef struct { gpio_port_t port; uint8_t pin; } pin_ref_t;

/* Pins in physical package order, from the board's pin map. */
static const pin_ref_t pins_under_test[] = {
    { GPIO_PORT_E, 2 }, { GPIO_PORT_E, 3 }, { GPIO_PORT_E, 4 },
    { GPIO_PORT_E, 5 }, { GPIO_PORT_E, 6 }, { GPIO_PORT_C, 13 },
};
#define N_PINS (sizeof pins_under_test / sizeof pins_under_test[0])

/* Returns the number of shorts found; reports each pair through cb. */
size_t pin_test_run(void (*cb)(size_t driven, size_t follower))
{
    size_t faults = 0;
    for (size_t i = 0; i < N_PINS; i++) {
        gpio_init(pins_under_test[i].port, pins_under_test[i].pin, GPIO_IN_PULLDOWN);
    }
    for (size_t d = 0; d < N_PINS; d++) {
        gpio_init(pins_under_test[d].port, pins_under_test[d].pin, GPIO_OUT_PUSHPULL);
        gpio_write(pins_under_test[d].port, pins_under_test[d].pin, true);
        for (volatile uint32_t settle = 0; settle < 1000u; settle++) { }
        for (size_t f = 0; f < N_PINS; f++) {
            if (f != d && gpio_read(pins_under_test[f].port, pins_under_test[f].pin)) {
                cb(d, f);
                faults++;
            }
        }
        gpio_init(pins_under_test[d].port, pins_under_test[d].pin, GPIO_IN_PULLDOWN);
    }
    return faults;
}

All pins start as inputs with pull-downs. Each pin in turn drives high; any other pin that reads high is shorted to it (or to a supply rail, which a second pass driving low would distinguish). The settle loop gives the internal pull-down (around 40 kΩ) time to discharge pin capacitance; its volatile counter keeps the compiler from deleting it. Only pins that are free to be driven may be listed: a pin connected to another chip’s output, or to something that must not be toggled (a motor driver enable), must be excluded or tested in a fixture.

Manufacturing test firmware

Every unit off the production line must be tested before it ships. The factory needs a fast, automatic, unambiguous answer: pass or fail, and if fail, why.

What to test

Test every interface that the assembly process could break: every external chip responds and reports the right ID, every bus works, every LED lights, every button reads, every connector carries signals, the crystal runs at the right frequency, the flash can be erased and programmed, radios transmit at the right power and frequency, and current consumption in each power mode is within limits (a high sleep current often means a misplaced or wrong-value component).

A self-test command table

The firmware side is a set of self-tests behind a command interface. A table of named tests keeps it extensible and gives the test station a stable, machine-readable protocol:

selftest.c

#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include "selftest.h"
#include "board.h"

typedef enum { ST_PASS, ST_FAIL, ST_SKIP } st_result_t;
typedef st_result_t (*selftest_fn)(char *detail, size_t cap);

typedef struct {
    const char *name;
    selftest_fn fn;
    bool        needs_operator;    /* true: requires a human or fixture action */
} selftest_t;

#define UID_BASE   0x1FFF7A10u
#define RCC_CR     (*(volatile uint32_t *)0x40023800u)
#define RCC_CFGR   (*(volatile uint32_t *)0x40023808u)

static st_result_t st_uid(char *d, size_t cap)
{
    const volatile uint32_t *uid = (const volatile uint32_t *)UID_BASE;
    snprintf(d, cap, "uid=%08lX%08lX%08lX",
             (unsigned long)uid[2], (unsigned long)uid[1], (unsigned long)uid[0]);
    return ST_PASS;
}

static st_result_t st_clock(char *d, size_t cap)
{
    bool hse_ok = (RCC_CR & (1u << 17)) != 0u;          /* HSERDY */
    bool pll_ok = (RCC_CR & (1u << 25)) != 0u;          /* PLLRDY */
    bool on_pll = ((RCC_CFGR >> 2) & 3u) == 2u;         /* SWS = PLL */
    snprintf(d, cap, "hse=%d pll=%d sws_pll=%d", hse_ok, pll_ok, on_pll);
    return (hse_ok && pll_ok && on_pll) ? ST_PASS : ST_FAIL;
}

static st_result_t st_accel(char *d, size_t cap)
{
    uint8_t id = 0;
    bool ok = lis3dsh_read_reg(0x0F, &id) && id == 0x3F; /* WHO_AM_I */
    snprintf(d, cap, "who_am_i=0x%02X", id);
    return ok ? ST_PASS : ST_FAIL;
}

static st_result_t st_codec(char *d, size_t cap)
{
    uint8_t id = 0;
    bool ok = cs43l22_read_reg(0x01, &id) && (id >> 3) == 0x1Cu; /* chip ID */
    snprintf(d, cap, "addr=0x4A id=0x%02X", id);
    return ok ? ST_PASS : ST_FAIL;
}

static st_result_t st_flash(char *d, size_t cap)
{
    uint32_t bad = flash_test_scratch_sector();          /* erase, pattern, verify */
    snprintf(d, cap, "sector=11 bad_words=%lu", (unsigned long)bad);
    return (bad == 0u) ? ST_PASS : ST_FAIL;
}

static st_result_t st_button(char *d, size_t cap)
{
    bool pressed = button_wait_press_ms(5000);
    snprintf(d, cap, "pressed=%d", pressed);
    return pressed ? ST_PASS : ST_FAIL;
}

static const selftest_t tests[] = {
    { "uid",    st_uid,    false },
    { "clock",  st_clock,  false },
    { "accel",  st_accel,  false },
    { "codec",  st_codec,  false },
    { "flash",  st_flash,  false },
    { "button", st_button, true  },
};
#define N_TESTS (sizeof tests / sizeof tests[0])

static bool run_one(const selftest_t *t)
{
    char detail[64] = "";
    st_result_t r = t->fn(detail, sizeof detail);
    const char *word = (r == ST_PASS) ? "PASS" : (r == ST_FAIL) ? "FAIL" : "SKIP";
    printf("SELFTEST %s %s %s\r\n", t->name, word, detail);
    return r != ST_FAIL;
}

/* Shell command: "selftest <name>", "selftest auto", or "selftest all" */
void selftest_cmd(const char *arg)
{
    size_t passed = 0, run = 0;
    bool all  = (strcmp(arg, "all") == 0);
    bool autm = (strcmp(arg, "auto") == 0);

    for (size_t i = 0; i < N_TESTS; i++) {
        bool selected = all || (autm && !tests[i].needs_operator) ||
                        strcmp(arg, tests[i].name) == 0;
        if (selected) {
            run++;
            passed += run_one(&tests[i]) ? 1u : 0u;
        }
    }
    if (run == 0u) {
        printf("SELFTEST %s UNKNOWN\r\n", arg);
    } else if (all || autm) {
        printf("SELFTEST SUMMARY %s %lu/%lu\r\n",
               (passed == run) ? "PASS" : "FAIL",
               (unsigned long)passed, (unsigned long)run);
    }
}

The table maps names to functions and marks tests that need an operator or fixture action, so selftest auto runs everything that can run unattended. Every result line starts with the same token and has the same shape (SELFTEST <name> <PASS|FAIL|SKIP> <detail>), which makes the station’s parser a few lines of Python and keeps the format stable across firmware versions. The detail field records the measured values, not just pass or fail: when the factory reports a 2% failure rate on codec, the logged IDs tell you whether the chip is missing (no ACK, id=0x00) or a different part was fitted (unexpected ID).

The accelerometer and codec are the Discovery board’s on-board LIS3DSH (WHO_AM_I register 0x0F reads 0x3F) and CS43L22 (I2C address 0x4A, chip ID in the upper five bits of register 0x01). The codec only answers once its reset pin (PD4 on the Discovery) is driven high, which is the sort of detail a self-test catches on the first board where someone forgot it. printf is fine here because this runs from the main loop in a test image, with _write retargeted to the UART as The C Standard Library on a Microcontroller showed. The driver helpers (lis3dsh_read_reg, flash_test_scratch_sector, and so on) come from earlier lessons and are declared in board.h.

Test fixtures

A test fixture holds the board and connects to it. A bed-of-nails fixture presses spring-loaded pogo pins onto test pads on the PCB, giving access to power, SWD, UART, and signals without connectors. The fixture’s PC programs the board, runs the self-tests over UART, measures currents and voltages with its own instruments, stimulates inputs (presses a button with a solenoid, injects a signal), records results against the unit’s serial number, and prints a label. Design the test pads into the PCB from the first prototype: adding them later costs a board revision.

Programming at the factory

Firmware gets into the flash in one of three ways:

• SWD programming on the line. A programmer connects through test pads or a connector and writes the image. Gang programmers program several boards in parallel to meet line takt time. Programming 512 KB over SWD takes several seconds; budget for it.
• Pre-programmed chips. The distributor or a programming house programs MCUs before assembly. Fast on the line, but the image is frozen weeks before production, and a firmware change means scrapping or reprogramming stock.
• Bootloader first. Only the bootloader is programmed by SWD (or pre-programmed); the application is loaded through the bootloader over UART, USB, or the product’s normal update path during test. The application can then change up to the last minute, and the factory exercises the same update path the field will use.

Many products combine them: bootloader pre-programmed or written by SWD, then test firmware loaded and run, then the product application loaded through the bootloader at the end of test.

Provisioning

Provisioning gives each unit its identity and per-unit data: serial number, hardware revision, calibration values, and cryptographic keys or certificates.

Unique IDs

Every STM32F4 has a 96-bit unique device ID at 0x1FFF7A10, readable as three 32-bit words (the st_uid test above prints it). It is programmed by ST during manufacturing and encodes the lot, wafer, and position on the wafer. It is unique but neither secret nor sequential, so use it as a stable hardware identifier and as an input for deriving per-device values, never as a key by itself.

Serial numbers come from the factory database, not from the chip: the test station requests the next serial number, writes it into the device’s provisioning record, prints it on the label, and records the mapping between serial number, UID, test results, and firmware version. That database is how support later answers “which units have the codec from the bad batch”.

Calibration data

Sensors, ADC references, and oscillators vary from unit to unit. The fixture measures known references, computes corrections, and writes them into the device. Calibration data is as important as firmware: a device with lost calibration produces plausible but wrong data, which is worse than no data.

Per-device keys and certificates

Connected devices need a unique identity for the cloud: typically a private key and a certificate. The secure pattern is to generate the key pair on the device (using the hardware random number generator), send only the public key or a certificate signing request to the factory station, have it signed by the manufacturer’s certificate authority (kept in an HSM, never on the line PC), and write the certificate back. The private key never leaves the device. Secure elements such as the ATECC608 do this internally, and Firmware Security covers the details.

Where to store it

Per-unit data must survive firmware updates and must not be overwritten by accident:

Location	Properties	Good for
OTP area (STM32F407: 512 bytes in 16 blocks of 32 bytes at 0x1FFF7800, lock bytes at 0x1FFF7A00)	Programmable once per bit, lockable per block, survives everything	Serial number, hardware revision, key hashes
A dedicated flash sector, excluded from updates	Rewritable, so can be recalibrated; needs protection	Calibration, certificates
Secure element	Keys never readable, hardware crypto	Private keys

Store provisioning data in a versioned record with a magic number and CRC, just like config records:

provision.h

#ifndef PROVISION_H
#define PROVISION_H

#include <stdint.h>

#define PROV_MAGIC   0x50524F56u      /* "PROV" */
#define PROV_VERSION 2u

typedef struct {
    uint32_t magic;
    uint16_t version;
    uint16_t length;                  /* sizeof(provision_t) when written */
    char     serial[16];              /* NUL-terminated, from factory DB */
    uint8_t  hw_rev;
    uint8_t  reserved[3];
    int16_t  temp_offset_centi;       /* calibration */
    uint16_t vref_mv;                 /* measured on the fixture */
    uint8_t  cert_sha256[32];         /* hash of the device certificate */
    uint32_t crc;                     /* CRC-32 over all preceding bytes */
} provision_t;

_Static_assert(sizeof(provision_t) == 68, "provision_t layout changed");

#endif

The explicit reserved bytes make the padding visible and the _Static_assert catches accidental layout changes that would make old provisioning records unreadable. The version and length fields let newer firmware read older records and fill defaults for fields that did not exist. Application firmware treats a missing or invalid provisioning record as a serious fault: it should refuse to connect to the cloud and report itself as unprovisioned rather than run with default calibration silently.

Locking the device

The last step on the line, after every test has passed and all provisioning is written, is to lock the device: enable readout protection and write any option bytes that disable debug access or protect sectors.

On the STM32F4, readout protection is set in the option bytes through the flash interface (base 0x40023C00). The RDP field occupies bits 15:8 of FLASH_OPTCR (offset 0x14): 0xAA is level 0 (no protection), 0xCC is level 2, and any other value is level 1. Option bytes are unlocked by writing two keys to FLASH_OPTKEYR (offset 0x08), changed in OPTCR, and committed by setting OPTSTRT (bit 1) and waiting for BSY (bit 16 of FLASH_SR):

lock.c

#include <stdint.h>

#define FLASH_OPTKEYR (*(volatile uint32_t *)0x40023C08u)
#define FLASH_SR      (*(volatile uint32_t *)0x40023C0Cu)
#define FLASH_OPTCR   (*(volatile uint32_t *)0x40023C14u)

#define OPTCR_OPTLOCK (1u << 0)
#define OPTCR_OPTSTRT (1u << 1)
#define OPTCR_RDP_POS 8u
#define SR_BSY        (1u << 16)

#define RDP_LEVEL1    0xBBu           /* any value except 0xAA and 0xCC */

int lock_set_rdp_level1(void)
{
    FLASH_OPTKEYR = 0x08192A3Bu;      /* option byte unlock sequence */
    FLASH_OPTKEYR = 0x4C5D6E7Fu;
    if (FLASH_OPTCR & OPTCR_OPTLOCK) {
        return -1;                    /* unlock failed */
    }
    while (FLASH_SR & SR_BSY) { }

    uint32_t optcr = FLASH_OPTCR;
    optcr &= ~(0xFFu << OPTCR_RDP_POS);
    optcr |= (RDP_LEVEL1 << OPTCR_RDP_POS);
    FLASH_OPTCR = optcr;
    FLASH_OPTCR = optcr | OPTCR_OPTSTRT;
    while (FLASH_SR & SR_BSY) { }

    FLASH_OPTCR |= OPTCR_OPTLOCK;
    return 0;                         /* takes effect after reset/power cycle */
}

The read-modify-write preserves the other option bits (watchdog mode, brown-out level, write protection) while changing only RDP. In practice the factory station usually sets option bytes with the programmer tool (OpenOCD’s stm32f2x driver commands or ST’s programming tools) rather than from firmware, but the sequence is the same. With level 1 set, a debugger can still connect but cannot read flash; regressing to level 0 is possible but mass-erases the whole flash, keys and all.

Danger

RDP level 2 (0xCC) is permanent. It disables the debug port forever and blocks all further option byte changes. A device locked at level 2 with broken firmware, or with a bootloader that cannot update itself, is scrap. Only set level 2 when your update path is proven, and test the exact factory sequence on sacrificial units first.

Release process

A firmware release is not “the build on Alice’s laptop that seemed to work”. It is a controlled, repeatable process:

• Version numbers follow semantic versioning and are stamped into the image from the git tag, with the commit hash and a build ID, as Build Systems, Toolchain Flags, and CI described.
• Release builds differ from debug builds in defined ways only: optimization level, assertions’ production reaction, log levels, disabled debug shells or semihosting, enabled readout protection checks. Test the release build itself, never only the debug build.
• Release notes list user-visible changes, fixed bugs (with ticket IDs), known issues, compatibility (minimum bootloader, protocol versions, host app versions), and the image hashes.
• Signing happens in CI or a dedicated signing service with the key in an HSM or cloud KMS. No developer machine ever holds the production signing key.
• Artifact retention. Archive the .elf (with debug symbols), .bin, .map, the signed image, the SBOM, and test reports for every release, forever. Without the exact .elf, a crash address from the field cannot be decoded.
• Reproducibility. Anyone can rebuild the release from the tag with the archived toolchain image and get a bit-identical binary.
• Tagging. The release is a signed git tag on a reviewed commit; release branches receive only fixes.

OTA fleet management

Shipping an update to every device at once is how one bug bricks a fleet. Fleet updates are rolled out in stages:

1. Internal devices (your own test fleet) get the release first.
2. Canaries: a small percentage of field devices, ideally a representative mix of hardware revisions, regions, and usage patterns.
3. Staged rollout: 1%, then 10%, then 50%, then 100%, with a hold period at each stage.

At each stage, rollback telemetry decides whether to continue: update success rate, boot success and rollback counts from the bootloader’s trial-boot mechanism, crash and watchdog reset rates compared with the previous version, connectivity, and battery drain. Define the thresholds before the rollout (“halt if reset rate exceeds 1.5x baseline”) so the decision is not made under pressure.

Plan compatibility windows: how many old versions the cloud and host apps support, and how a device that has been offline for two years gets from version 1.2 to version 4.0 (often through a mandatory intermediate version that migrates data formats). Devices must be able to download, verify, and reject an image they cannot run, without bricking.

Field diagnostics

When a device misbehaves in the field, the only information you will ever have is what the firmware recorded. Design that information in from the start.

Crash records capture the fault handler’s register dump and fault status registers in .noinit RAM, survive the reset, and are stored and reported after reboot (Watchdogs, Faults, and Recovering From Them builds this). Event logs record significant events with timestamps: boots with reset cause, errors, state changes, updates. Health counters summarize long-term behaviour cheaply:

health.h

#ifndef HEALTH_H
#define HEALTH_H

#include <stdint.h>

typedef struct {
    uint32_t boots;
    uint32_t resets_por;
    uint32_t resets_watchdog;
    uint32_t resets_software;
    uint32_t resets_brownout;
    uint32_t hard_faults;
    uint32_t comm_crc_errors;
    uint32_t comm_timeouts;
    uint32_t flash_write_errors;
    uint32_t uptime_total_s;          /* across all boots */
    uint32_t uptime_max_s;            /* longest single boot */
    uint16_t stack_min_free_bytes;    /* worst high-water mark seen */
    uint16_t battery_min_mv;
} health_t;

void health_on_boot(void);            /* read reset cause, bump counters */
void health_tick_1s(void);
void health_save_if_due(void);        /* persist hourly, not every change */

#endif

A dashboard of these counters across a fleet is extraordinarily revealing: a firmware version whose watchdog resets per device-day are three times the previous version’s, a hardware batch whose brown-out resets cluster in cold regions, a site where CRC errors spike because of a noisy motor. Persist counters periodically (hourly, or on a clean shutdown), not on every increment, or the counters themselves wear out the flash.

Remote retrieval: the device uploads health counters with its regular telemetry, and supports a “send diagnostics” command that uploads the crash record and recent event log. For devices without connectivity, a shell command or a USB mass-storage file does the same. Package everything into a support bundle: firmware version and build ID, provisioning record (serial, hardware revision), health counters, the last crash record, the last N event log entries, and current configuration. A support engineer who gets a bundle can usually diagnose without a device in hand.

Regulatory and compliance

Firmware engineers are pulled into compliance work more than they expect.

EMC (electromagnetic compatibility: FCC Part 15 in the US, the EMC directive for CE marking in the EU) limits what a device emits and requires it to tolerate interference. Firmware affects emissions directly: GPIO output speed settings (use the slowest OSPEEDR setting that meets timing, since fast edges radiate), clock frequencies and their harmonics, unused peripherals and clock outputs left enabled, and spread-spectrum clocking, which modulates the PLL frequency slightly to spread energy across a band (the STM32F407’s PLL supports this through the RCC_SSCGR register). Firmware also affects immunity: robust handling of glitched inputs, watchdog recovery, and communication retries determine whether the device survives the ESD and burst tests.

Radio products need radio certification, which is one reason to use pre-certified modules. In the EU, the Radio Equipment Directive’s cybersecurity requirements apply to many connected radio products, with EN 18031 as the harmonised standard family describing them: secure update mechanisms, no universal default passwords, protection of stored credentials, and similar. The EU Cyber Resilience Act extends cybersecurity obligations to products with digital elements generally, with requirements phasing in over the next few years. Other markets have their own rules.

SBOMs (software bills of materials) list every component in the firmware, with versions and licenses: RTOS, TCP/IP stack, crypto library, vendor HAL, your own code. Regulators and large customers increasingly require them, and you need one anyway to answer “are we affected by this CVE in mbedTLS?” in minutes. Generate it in CI from the build (formats: SPDX or CycloneDX), and keep it with each release’s artifacts.

Supporting old firmware for years

An industrial or medical product is supported for ten years or more. Planning for that:

• Release branches. Each supported major version has a branch that receives security and critical fixes; fixes are made on the main branch first and backported.
• LTS versions. Declare which versions are long-term supported and for how long, so customers can plan.
• Toolchain archival. The exact compiler version matters: a different GCC release can change code generation, timing, and size. Archive the toolchain as a container image referenced by digest, not by a tag that can move, along with all build tools. Test once a year that you can still rebuild an old release bit-identically.
• Hardware for testing. Keep working units of every hardware revision in a lab, connected to the HIL rig, so fixes to old branches are tested on the hardware they ship to.
• Third-party dependencies. Vendor every dependency into the repository or an internal mirror; the upstream project, download site, or vendor SDK portal may be gone in five years.

Post-mortems

When something goes badly wrong (a fleet-wide outage, a field recall, a security incident, a release that bricked units), run a post-mortem. Its purpose is to prevent recurrence, not to assign blame; people are honest about what happened only if it is safe to be.

A useful post-mortem document answers:

1. What happened, with a timeline from the first trigger to full recovery.
2. Impact: how many devices or customers, for how long, and what they experienced.
3. Root cause, found by repeatedly asking why: the watchdog reset loop was caused by a flash write timeout, which happened because a new SPI clock setting violated the flash chip’s timing at low temperature, which was not caught because HIL tests run only at room temperature.
4. What went well and what went badly in detection and response.
5. Action items, each with an owner and a date: the fix itself, plus the process changes that would have caught it (a cold-chamber test in the release checklist, a timing margin check in review).

The action items are the point. A post-mortem whose only output is the code fix has wasted the incident.

Common mistakes

• Loading product firmware on a board that has never been brought up. Fix: a minimal bring-up image and a bottom-up checklist: power, clocks, reset, debug, UART, peripherals.
• Reading the errata sheet after weeks of debugging. Fix: read it first for every part on the board.
• Self-tests that print only PASS or FAIL. Fix: record measured values in a fixed machine-readable format.
• No test pads designed into the PCB. Fix: pads for power, SWD, UART, and key signals from the first prototype.
• Private keys generated on the factory PC and copied to devices. Fix: generate keys on the device or in a secure element; only public material leaves.
• Setting RDP level 2 before the update path is proven. Fix: level 1 in production until field updates are thoroughly validated, and test the locking sequence on sacrificial units.
• Discarding the .elf of a release. Fix: archive every release’s ELF, map, images, SBOM, and test reports permanently.
• Rolling out an update to the whole fleet at once. Fix: internal, canary, then staged percentages with predefined halt thresholds.
• Persisting health counters on every increment. Fix: keep them in RAM and save periodically.

Exercises

1. Write a bring-up checklist for the STM32F4 Discovery board as if it were a new design, using its schematic: list each power rail, the clock sources, the reset circuit, the debug connection, and the order in which you would test the on-board peripherals. Note which errata items from the STM32F407 errata sheet affect peripherals the course uses.
2. Implement the self-test table on the Discovery board with at least uid, clock, accel, codec, and button. Write a Python station script that runs selftest auto, parses every line, and writes one CSV row per board with the UID and every detail field. Then hold the codec in reset (leave PD4 low) and confirm the station reports the failure with a useful detail.
3. Define a provisioning record for your capstone device, compute its size and padding by hand, and confirm with _Static_assert. Write a host-side tool that builds a record (with CRC) from a serial number and calibration values and outputs a .bin to program at a fixed address. Then make the firmware reject the record when one byte is altered.
4. Implement health_t with health_on_boot() reading RCC_CSR reset flags (as in Watchdogs, Faults, and Recovering From Them) and persisting hourly to the A/B config scheme. Trigger a watchdog reset, a software reset, and a power cycle, and show the counters after each.
5. Write a post-mortem for a bug you have actually hit in this course (a missing volatile, a forgotten clock enable, a stack overflow). Include a timeline, the root cause found by asking “why” at least three times, and two action items that would have caught it earlier.

Check yourself

Question 1:On a brand-new board, SWD cannot connect. What should you check before suspecting firmware?

The UART baud rate Power rails, the reset line, and the debug connection wiring, in that order The linker script The RDP level in the option bytes of a never-programmed chip

Question 2:The factory reports that 3% of units fail the codec self-test. What makes this diagnosable from the station logs?

The test returns a nonzero exit code The detail field records the actual ID read (or 0x00 on no ACK), distinguishing a missing chip, a wrong part, and a held-in-reset chip The test runs twice The firmware version number

Question 3:Why should a connected device's private key be generated on the device rather than on the factory test PC?

The device's random number generator is faster Certificates cannot be created otherwise The private key then never exists outside the device, so a compromised factory PC or database cannot leak it Test PCs cannot run elliptic curve cryptography

Question 4:An STM32F4 product ships with RDP level 2. Six months later a critical bug is found. Which statement is true?

A debugger can regress the device to level 0 with a mass erase The device can only be fixed through the firmware's own update path; debug access and option byte changes are permanently disabled Level 2 can be removed by holding BOOT0 high at reset Level 2 only protects the bootloader sector

Question 5:A staged OTA rollout reaches 10% of the fleet. Watchdog resets per device-day are twice the previous version's. What should happen?

Continue to 50%, since 10% is too small to judge Halt the rollout per the predefined threshold, roll back affected devices if possible, and investigate with crash records Disable the watchdog in the next version Push the update to 100% so all devices behave the same

Key takeaways

• Bring-up is bottom-up (power, clocks, reset, debug, UART, peripherals) with a minimal test image, and the errata sheet is read first.
• Manufacturing self-tests come from a command table with stable, machine-readable output that records measured values, not just pass or fail.
• Per-unit identity (serial, calibration, keys) is provisioned into protected storage in versioned, CRC-checked records; private keys are generated on the device.
• Locking (readout protection, option bytes) is the last factory step; RDP level 2 on the STM32F4 is permanent.
• Releases are versioned, signed in CI with keys in an HSM, reproducible, and archived with the .elf, map, SBOM, and test reports.
• OTA updates roll out in stages with predefined telemetry thresholds for halting and rollback.
• Crash records, event logs, health counters, and support bundles are designed in from the start, because they are all you will have in the field.

Next: What Comes Next, the last lesson: interviews, portfolio projects, books, and a study plan.