Filesystems, Logging, and Data on Device

Sooner or later a device has to remember more than a config struct: a week of sensor samples, an event log for field support, certificates, a downloaded firmware image. That data must survive power cuts at the worst moment, must not wear out the flash in a year, and must be readable by a human or a server long after the firmware that wrote it has been replaced. This lesson covers how to choose a storage scheme, how to integrate LittleFS and FatFs, how to design logs that survive power loss, how to encode telemetry, and how to timestamp it honestly.

Choosing a storage scheme

There are three levels of structure you can put on flash, and each one costs more code and RAM than the one below it.

Scheme	What it is	Good for	Cost
Raw sectors	You decide what goes at which address	Firmware images, one fixed config record, a circular log	Zero library code; you own all the power-fail logic
Key-value store	Named values ("wifi.ssid", "cal.offset") in an append-only log	Settings, calibration, small counters	2 to 8 KB of code; values are small
Filesystem	Named files of any size, directories	Many or large blobs, logs you export as files, certificates, data that a PC must read	10 to 30 KB of code, a few KB of RAM

The rule of thumb: use raw sectors when there are one or two well-defined things to store, a key-value store when there are many small things, and a filesystem when the data is file-shaped (variable size, created and deleted at runtime, or needs to be copied to a PC). Do not reach for a filesystem to store 64 bytes of configuration; the A/B config records from Flash, EEPROM, and Non-Volatile Storage do that with less code and stronger guarantees.

A block device abstraction

Every storage library above raw sectors needs the same four operations from the hardware: read, program, erase, and a size description. Define that interface once and every layer above it (LittleFS, a key-value store, your own log) works on internal flash, SPI NOR, or an SD card without change.

blockdev.h

#ifndef BLOCKDEV_H
#define BLOCKDEV_H

#include <stddef.h>
#include <stdint.h>

typedef struct blockdev blockdev_t;

typedef struct {
    int (*read)(blockdev_t *bd, uint32_t addr, void *buf, size_t len);
    int (*prog)(blockdev_t *bd, uint32_t addr, const void *buf, size_t len);
    int (*erase)(blockdev_t *bd, uint32_t block);
    int (*sync)(blockdev_t *bd);
} blockdev_ops_t;

struct blockdev {
    const blockdev_ops_t *ops;
    uint32_t block_size;     /* erase unit in bytes */
    uint32_t block_count;
    uint32_t prog_size;      /* smallest programmable unit */
    void    *ctx;            /* driver private data */
};

#endif

The ops pointer refers to a const table, so the function pointers live in flash and cost no RAM per instance. The geometry fields tell the layer above how big an erase block is, which is the single most important property of any flash device. This is the vtable pattern that Design Patterns for Embedded C develops fully.

SPI NOR as the backing store

A typical external part is a 4 MB SPI NOR flash with 4 KB erasable sectors, 256-byte program pages, and an endurance of about 100,000 erase cycles per sector. Those numbers are much friendlier than the STM32F407’s internal flash (128 KB sectors, 10,000 cycles, and the CPU stalls during erase), which is why data logging almost always goes to external flash. The commands were introduced in Flash, EEPROM, and Non-Volatile Storage: 0x03 read, 0x06 write enable, 0x02 page program, 0x20 4 KB sector erase, 0x05 read status (bit 0 is busy).

Here is the block device implementation, assuming a spi_nor_* driver built on the SPI transaction API from SPI and I2C:

bd_spinor.c

#include "blockdev.h"
#include "spi_nor.h"

#define NOR_PAGE 256u

static int nor_read(blockdev_t *bd, uint32_t addr, void *buf, size_t len)
{
    (void)bd;
    return spi_nor_read(addr, buf, len);         /* 0x03, any length */
}

static int nor_prog(blockdev_t *bd, uint32_t addr, const void *buf, size_t len)
{
    (void)bd;
    const uint8_t *p = buf;
    while (len > 0u) {
        /* A page program must not cross a 256-byte page boundary:
         * the address wraps inside the page and overwrites its start. */
        size_t room  = NOR_PAGE - (addr % NOR_PAGE);
        size_t chunk = (len < room) ? len : room;
        int err = spi_nor_page_program(addr, p, chunk);  /* 0x06, 0x02, wait */
        if (err != 0) {
            return err;
        }
        addr += (uint32_t)chunk;
        p    += chunk;
        len  -= chunk;
    }
    return 0;
}

static int nor_erase(blockdev_t *bd, uint32_t block)
{
    return spi_nor_sector_erase(block * bd->block_size); /* 0x06, 0x20, wait */
}

static int nor_sync(blockdev_t *bd)
{
    (void)bd;
    return 0;                       /* every op already waited for not-busy */
}

static const blockdev_ops_t nor_ops = { nor_read, nor_prog, nor_erase, nor_sync };

blockdev_t g_spinor = {
    .ops = &nor_ops, .block_size = 4096u, .block_count = 1024u,
    .prog_size = 16u, .ctx = NULL
};

The loop in nor_prog is the bug everyone writes once. If you send 300 bytes starting at a page boundary in one page-program command, the chip programs the first 256, then wraps the address back to the start of the same page and programs the remaining 44 bytes over the data you just wrote. Nothing reports an error; you find out when the CRC fails later.

LittleFS

LittleFS is a filesystem designed for microcontrollers on raw flash. Its properties are exactly the ones firmware needs:

• Power-loss resilience. Metadata updates are copy-on-write with CRC-checked commits. If power fails mid-write, the filesystem mounts to the last committed state. A file you were writing keeps its previous contents until you sync or close it.
• Wear leveling. Blocks are allocated dynamically and metadata pairs move after a configurable number of erase cycles, spreading wear across the device.
• Bounded RAM. RAM use is fixed by configuration (a few caches and a lookahead bitmap), not proportional to the size of the device or the number of files.

Configuration

LittleFS talks to your hardware through a struct lfs_config of callbacks and geometry. Here it is wired to the block device above, with all buffers statically allocated so no malloc is needed (build LittleFS with LFS_NO_MALLOC defined):

storage.c

#include <stdint.h>
#include "lfs.h"
#include "blockdev.h"

extern blockdev_t g_spinor;

#define LFS_CACHE_SIZE     256
#define LFS_LOOKAHEAD_SIZE 32

static uint8_t read_buf[LFS_CACHE_SIZE];
static uint8_t prog_buf[LFS_CACHE_SIZE];
static uint8_t lookahead_buf[LFS_LOOKAHEAD_SIZE];

static int lfs_bd_read(const struct lfs_config *c, lfs_block_t block,
                       lfs_off_t off, void *buffer, lfs_size_t size)
{
    blockdev_t *bd = c->context;
    return bd->ops->read(bd, block * c->block_size + off, buffer, size)
           ? LFS_ERR_IO : LFS_ERR_OK;
}

static int lfs_bd_prog(const struct lfs_config *c, lfs_block_t block,
                       lfs_off_t off, const void *buffer, lfs_size_t size)
{
    blockdev_t *bd = c->context;
    return bd->ops->prog(bd, block * c->block_size + off, buffer, size)
           ? LFS_ERR_IO : LFS_ERR_OK;
}

static int lfs_bd_erase(const struct lfs_config *c, lfs_block_t block)
{
    blockdev_t *bd = c->context;
    return bd->ops->erase(bd, block) ? LFS_ERR_IO : LFS_ERR_OK;
}

static int lfs_bd_sync(const struct lfs_config *c)
{
    blockdev_t *bd = c->context;
    return bd->ops->sync(bd) ? LFS_ERR_IO : LFS_ERR_OK;
}

static const struct lfs_config lfs_cfg = {
    .context          = &g_spinor,
    .read             = lfs_bd_read,
    .prog             = lfs_bd_prog,
    .erase            = lfs_bd_erase,
    .sync             = lfs_bd_sync,
    .read_size        = 16,
    .prog_size        = 16,
    .block_size       = 4096,
    .block_count      = 1024,
    .block_cycles     = 500,
    .cache_size       = LFS_CACHE_SIZE,
    .lookahead_size   = LFS_LOOKAHEAD_SIZE,
    .read_buffer      = read_buf,
    .prog_buffer      = prog_buf,
    .lookahead_buffer = lookahead_buf,
};

lfs_t g_lfs;

The fields that matter:

Field	Meaning	Rule
read_size, prog_size	Minimum read and program units	All reads and programs are multiples of these
block_size	Erase unit	Must match the device’s erase sector (4096 here)
block_count	Number of blocks LittleFS may use	1024 blocks of 4 KB = the whole 4 MB device
cache_size	Size of read, program, and per-file caches	Multiple of read_size and prog_size, divides block_size
lookahead_size	Bytes of block allocation bitmap	Multiple of 8; each byte tracks 8 blocks
block_cycles	Erases before metadata is moved to a new block	100 to 1000; lower spreads wear more, costs speed

context carries the block device pointer into each callback, so the same callbacks work for any blockdev_t. The whole configuration is const, so it lives in flash.

Mount, format, read, and write

The canonical first program is a boot counter: a file that is read, incremented, and written back on every boot.

storage.c (continued)

#include <stdbool.h>

static uint8_t file_buf[LFS_CACHE_SIZE];
static const struct lfs_file_config file_cfg = { .buffer = file_buf };

int storage_mount(bool allow_format)
{
    int err = lfs_mount(&g_lfs, &lfs_cfg);
    if (err == LFS_ERR_CORRUPT && allow_format) {
        err = lfs_format(&g_lfs, &lfs_cfg);    /* first boot of a blank chip */
        if (err == LFS_ERR_OK) {
            err = lfs_mount(&g_lfs, &lfs_cfg);
        }
    }
    return err;
}

int storage_bump_boot_count(uint32_t *out)
{
    lfs_file_t f;
    uint32_t count = 0;

    int err = lfs_file_opencfg(&g_lfs, &f, "boot_count",
                               LFS_O_RDWR | LFS_O_CREAT, &file_cfg);
    if (err < 0) {
        return err;
    }
    lfs_ssize_t n = lfs_file_read(&g_lfs, &f, &count, sizeof count);
    if (n != (lfs_ssize_t)sizeof count) {
        count = 0;                              /* new or short file */
    }
    count++;
    if (lfs_file_rewind(&g_lfs, &f) < 0 ||
        lfs_file_write(&g_lfs, &f, &count, sizeof count)
            != (lfs_ssize_t)sizeof count) {
        lfs_file_close(&g_lfs, &f);
        return LFS_ERR_IO;
    }
    err = lfs_file_close(&g_lfs, &f);           /* commit happens here */
    if (err == LFS_ERR_OK) {
        *out = count;
    }
    return err;
}

Three details deserve attention. First, the format-on-corrupt logic is guarded by a flag. Formatting automatically whenever mount fails is a classic way to wipe a device’s entire history because of one transient SPI glitch at boot. Format only when you have positive evidence the chip is blank (a factory flag, or a retry that fails the same way), and log that it happened. Second, each open file needs a cache buffer; with LFS_NO_MALLOC you supply it through lfs_file_opencfg(). Third, the data is not durable until lfs_file_sync() or lfs_file_close() returns. Pull the power between the write and the close, and the next boot sees the old count. That is the guarantee you want: old value or new value, never a torn mix.

Limits

LittleFS is not free. Writes in the middle of large files are slow, because copy-on-write rewrites from the change point to the end of the block chain; append-only logs and small files are fast. Directory listings and lfs_fs_size() walk metadata and take time proportional to the number of files. Every sync performs a metadata commit, so syncing after every 32-byte record multiplies the bytes physically written. And mount time on a large, full device can reach hundreds of milliseconds. Measure these on your hardware before committing to a design.

FatFs on SD cards

FatFs implements the FAT12/16/32 and exFAT filesystems. Choose it when a PC must read the medium directly: a removable SD card that a technician pulls out, or a device that appears as a USB drive (MSC) with files on it. Your board provides five functions (disk_initialize, disk_status, disk_read, disk_write, disk_ioctl) plus get_fattime for timestamps, typically over SPI or the STM32’s SDIO peripheral (4-bit bus, much faster).

sdlog.c

#include "ff.h"

static FATFS fs;
static FIL   log_file;

FRESULT sdlog_open(void)
{
    FRESULT r = f_mount(&fs, "", 1);             /* 1 = mount now */
    if (r != FR_OK) {
        return r;
    }
    return f_open(&log_file, "log.csv", FA_WRITE | FA_OPEN_APPEND);
}

FRESULT sdlog_line(const char *line, UINT len)
{
    UINT written;
    FRESULT r = f_write(&log_file, line, len, &written);
    if (r == FR_OK && written != len) {
        r = FR_DISK_ERR;                        /* disk full */
    }
    if (r == FR_OK) {
        r = f_sync(&log_file);                  /* flush data + directory entry */
    }
    return r;
}

The critical call is f_sync(). FatFs buffers data and the file’s size in the directory entry; until f_sync() or f_close(), a power cut leaves a file whose directory entry says it is shorter than the data written, or a FAT with lost clusters. Even with f_sync(), FAT itself is not power-fail safe: an interrupted update to the FAT table or directory can corrupt the filesystem, and the SD card’s internal controller may be mid-way through its own wear-leveling operation. Products that use SD cards add a supercapacitor or brown-out detection so they can finish writes and close files when power drops, and they run a consistency check on boot.

Key-value stores

Settings are naturally key-value: "wifi.ssid", "sample.period_ms", "cal.temp_offset". A key-value store gives you kv_set(key, data, len) and kv_get(key, buf, cap) over flash. Internally it is the append-only log of records from Flash, EEPROM, and Non-Volatile Storage: each write appends a new record with a key, value, and CRC; reads find the latest valid record; when a sector fills, live records are copied to a fresh sector and the old one is erased. ESP-IDF’s NVS and Zephyr’s settings subsystem are production examples of this design.

Use a key-value store for configuration even if you also have a filesystem: it is smaller, faster for tiny values, and keeps settings separate from bulk data so a full log partition cannot block a config change.

Wear and write amplification

Write amplification is the ratio of bytes physically programmed and erased to bytes the application asked to write. It decides whether your device lasts 50 years or 2 days.

Worked example: a device logs a 32-byte record once per second to a 4 MB SPI NOR with 4 KB sectors rated for 100,000 erase cycles.

Logical data per day     = 32 B x 86,400 s        = 2,764,800 B  (about 2.6 MiB)

Design A: append-only circular log over the whole device (amplification ~1)
  Device fills every       4,194,304 B / 2,764,800 B per day  = 1.52 days
  Each sector erased once per fill, so after 100,000 fills:
  Lifetime                 = 100,000 x 1.52 days  = 152,000 days, about 415 years

Design B: same, but through a filesystem syncing every record (assume 8x amplification)
  Lifetime                 = 415 years / 8         = about 52 years

Design C: read-modify-erase-write of one 4 KB sector per record
  Erases of that sector    = 86,400 per day
  Lifetime                 = 100,000 / 86,400      = 1.16 days

The same logical data rate gives lifetimes from a day to centuries. Design C is not a strawman: it is what happens when someone keeps a “current log page” in one fixed sector and rewrites it in place, or stores a counter in one flash location. The rules that follow:

• Spread writes over as many erase blocks as possible.
• Never rewrite in place; append and garbage-collect.
• Batch small records in RAM and write them in page-sized chunks if you can tolerate losing a few seconds of data on power loss.
• Do the arithmetic during design, with the endurance figure from the datasheet for your temperature range (endurance drops at high temperature).

Power-fail safety patterns

Power can fail between any two instructions. Design every persistent update so that, whatever point it stops at, the next boot finds either the old state or the new state.

• CRC per record. Every record carries a CRC over its contents. A torn write (half programmed) fails the check and is ignored.
• Sequence numbers. Every record carries a monotonically increasing number. At boot, the valid record with the highest number is the newest; there is no “current pointer” to corrupt.
• Append-only. New data never overwrites old data until the old data is provably obsolete, so a failed write cannot destroy the previous good state.
• Two-phase commit. Write the new data completely, verify it, then write a small commit marker (or the sequence number) last. The A/B config scheme is a two-phase commit.
• Journaling. Record the intended change in a journal, apply it, then mark the journal entry complete. On boot, replay or discard incomplete entries. LittleFS’s metadata commits are a form of this.

A circular log on raw flash

Here is a complete circular event log using all of those ideas. It uses the flash_dev_* interface from Testing Embedded Software (erase by 4 KB sector index, program and read by byte address), so it runs against the fake flash in host tests and against SPI NOR on target.

evlog.c

#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include <string.h>
#include "flash_dev.h"
#include "crc32.h"

#define LOG_SECTOR_SIZE  4096u
#define LOG_SECTORS      16u
#define LOG_DATA_MAX     20u
#define REC_SIZE         32u
#define RECS_PER_SECTOR  (LOG_SECTOR_SIZE / REC_SIZE)
#define LOG_RECS         (RECS_PER_SECTOR * LOG_SECTORS)
#define SEQ_BLANK        0xFFFFFFFFu

typedef struct {
    uint32_t seq;
    uint16_t type;
    uint16_t len;
    uint8_t  data[LOG_DATA_MAX];
    uint32_t crc;                    /* CRC-32 over bytes 0..27 */
} log_rec_t;

_Static_assert(sizeof(log_rec_t) == REC_SIZE, "record must be 32 bytes");

static uint32_t next_index;          /* slot the next record goes into */
static uint32_t next_seq;

static void read_slot(uint32_t idx, log_rec_t *r)
{
    if (flash_dev_read(idx * REC_SIZE, (uint8_t *)r, sizeof *r) != 0) {
        memset(r, 0, sizeof *r);     /* unreadable: treat as invalid */
    }
}

static bool rec_valid(const log_rec_t *r)
{
    return r->seq != SEQ_BLANK && r->len <= LOG_DATA_MAX &&
           r->crc == crc32(r, offsetof(log_rec_t, crc));
}

static bool slot_blank(uint32_t idx)
{
    log_rec_t r;
    read_slot(idx, &r);
    const uint8_t *b = (const uint8_t *)&r;
    for (size_t i = 0; i < sizeof r; i++) {
        if (b[i] != 0xFFu) {
            return false;
        }
    }
    return true;
}

void evlog_init(void)
{
    bool found = false;
    uint32_t best_seq = 0, best_idx = 0;

    for (uint32_t i = 0; i < LOG_RECS; i++) {
        log_rec_t r;
        read_slot(i, &r);
        if (rec_valid(&r) && (!found || r.seq > best_seq)) {
            found = true;
            best_seq = r.seq;
            best_idx = i;
        }
    }

    if (!found) {
        next_index = 0;
        next_seq = 1;
        return;
    }
    next_index = (best_idx + 1u) % LOG_RECS;
    next_seq = best_seq + 1u;

    /* Skip slots dirtied by a torn write, up to the next sector boundary,
     * where evlog_append() erases before writing. */
    while (next_index % RECS_PER_SECTOR != 0u && !slot_blank(next_index)) {
        next_index = (next_index + 1u) % LOG_RECS;
    }
}

bool evlog_append(uint16_t type, const void *data, uint16_t len)
{
    if (len > LOG_DATA_MAX) {
        return false;
    }
    if (next_index % RECS_PER_SECTOR == 0u) {
        /* Entering a sector: erase it, discarding its oldest records. */
        if (flash_dev_erase(next_index / RECS_PER_SECTOR) != 0) {
            return false;
        }
    }

    log_rec_t r;
    memset(&r, 0xFF, sizeof r);      /* unused data bytes stay erased */
    r.seq  = next_seq;
    r.type = type;
    r.len  = len;
    if (len > 0u) {
        memcpy(r.data, data, len);   /* memcpy with NULL is UB even for 0 */
    }
    r.crc  = crc32(&r, offsetof(log_rec_t, crc));

    if (flash_dev_program(next_index * REC_SIZE, (const uint8_t *)&r,
                          sizeof r) != 0) {
        return false;
    }
    next_index = (next_index + 1u) % LOG_RECS;
    next_seq++;
    return true;
}

Walk through what happens on each kind of power failure:

• Power fails before flash_dev_program(). Nothing was written. Next boot finds the same newest record and continues.
• Power fails during the program. The slot is partly programmed: its CRC fails, so evlog_init() ignores it as data, and the skip loop moves past it because it is not blank. The record is lost; nothing else is.
• Power fails during the erase. The sector is partly erased. Any records still intact in it are older than the newest record elsewhere, and any half-erased record fails its CRC. Boot resumes from the newest valid record; the next append into that sector region continues the skip and erase logic.

The log keeps between 15 and 16 sectors of history: rotating into a sector costs the 128 oldest records at once. The record layout is checked with _Static_assert because the struct is written to flash byte for byte; that is safe here because the same firmware writes and reads it, but data exported to other machines should be serialized explicitly, as the next section shows. Scanning 2,048 slots at boot over SPI takes tens of milliseconds; a production version reads one record per sector first to find the newest sector, then scans only that one.

Structured logging and telemetry

Data leaves the device eventually: to a support engineer’s laptop, to a cloud service, to a test script. How you encode it decides how much flash and bandwidth it costs and whether next year’s tools can still read it.

Text versus binary

	Text ("T=21.50 H=40\n")	Binary (fixed struct, CBOR, protobuf)
Size	Large: 15 to 50 bytes per sample	Small: 4 to 15 bytes per sample
CPU	snprintf formatting, kilobytes of code	Packing integers, very fast
Human readable	Yes, in any terminal	Needs a decoder
Schema evolution	Ad hoc parsing breaks easily	Designed in (CBOR, protobuf)

The usual professional answer: text for interactive consoles, binary for anything stored or transmitted in volume, with a host-side decoder in the same repository as the firmware.

CBOR

CBOR (Concise Binary Object Representation) is a binary JSON: maps, arrays, integers, strings, byte strings, but encoded compactly and self-describing. The map {"t": 2150} encodes as six bytes:

A1          map with 1 pair
61 74       text string, length 1: "t"
19 08 66    unsigned integer, 2-byte argument: 0x0866 = 2150

A negative value such as -1000 encodes as 39 03 E7 (major type 1 stores -1 minus the value, here 999). Because it is self-describing, a CBOR decoder on the server can read records from firmware versions it has never seen, and extra fields are simply ignored. Small C libraries such as TinyCBOR or zcbor encode into a caller-provided buffer with no heap.

Protocol Buffers with nanopb

Protocol Buffers go a step further: you write a schema, and a generator produces code. nanopb is the protobuf implementation for microcontrollers: it generates plain C structs with fixed-size arrays and needs no heap.

sample.proto

syntax = "proto3";

message Sample {
  uint32 seq         = 1;
  uint32 uptime_s    = 2;
  sint32 temp_centi  = 3;
  uint32 humidity_pm = 4;
  repeated uint32 error_codes = 5;
}

sample.options

Sample.error_codes max_count:4

The options file bounds the repeated field, so nanopb can generate a fixed array and a compile-time maximum encoded size. The generator produces a Sample struct with members seq, uptime_s, temp_centi, humidity_pm, error_codes_count, and uint32_t error_codes[4], plus a field descriptor and a Sample_size macro. Encoding:

telemetry.c

#include <stddef.h>
#include <stdint.h>
#include "pb_encode.h"
#include "sample.pb.h"

size_t telemetry_encode(const Sample *s, uint8_t *buf, size_t cap)
{
    pb_ostream_t os = pb_ostream_from_buffer(buf, cap);
    if (!pb_encode(&os, Sample_fields, s)) {
        return 0;                          /* PB_GET_ERROR(&os) says why */
    }
    return os.bytes_written;
}

Size the caller’s buffer with Sample_size and the encode can never run out of space. Protobuf uses variable-length integers, so a sample with small values encodes in 8 to 12 bytes. sint32 uses zigzag encoding so that small negative temperatures stay small; a plain int32 field would spend 10 bytes on every negative number.

Schema versioning

Telemetry outlives firmware. The rules that keep old and new readers compatible:

• Never reuse or renumber a field. Retire a field by reserving its number.
• Only add new fields; old decoders skip fields they do not know.
• Never change a field’s type or units. A temperature in tenths that becomes hundredths needs a new field.
• Put a schema or firmware version in every log file header or upload, so a decoder can pick the right interpretation of anything that is not self-describing.

Timekeeping

A log entry without a trustworthy time is half as useful. Firmware deals with two kinds of time, and mixing them up causes most timestamp bugs.

• Monotonic time counts up from boot and never jumps: the SysTick millisecond counter, a free-running timer. Use it for timeouts, durations, rate limits, and ordering within one boot.
• Wall-clock time is calendar time, usually as a Unix epoch (seconds since 1970-01-01 UTC). It can jump forwards or backwards when it is synchronized. Use it for humans and for correlating with server data, never for measuring durations.

The STM32F407’s RTC runs from a 32.768 kHz LSE crystal in the backup domain. With a coin cell or supercap on the VBAT pin, it keeps time while main power is off. Without a time source, though, the RTC starts at an arbitrary value on first power-up and drifts by roughly 20 to 100 ppm (about 2 to 9 seconds per day) depending on the crystal and temperature.

Time comes from outside: SNTP over a network, GPS, a phone over BLE (the Current Time Service), a cellular network, or a host tool that sets the clock when it connects. Until the first sync, the device does not know the date.

A robust logging scheme timestamps every record with boot count + monotonic uptime, which is always valid, and writes a special time-sync record whenever wall-clock time becomes known:

seq=1201 boot=37 up=0.004s   BOOT reset_cause=POR
seq=1202 boot=37 up=12.500s  SAMPLE t=21.50
seq=1203 boot=37 up=14.020s  TIME_SYNC epoch=1793092800
seq=1204 boot=37 up=15.000s  SAMPLE t=21.52

A decoder converts every record from boot 37 to wall-clock time using the sync record, including records written before the sync. Records from a boot that never synced are still correctly ordered and spaced. Two more details: store epoch time as an unsigned 32-bit or a 64-bit value (signed 32-bit seconds overflow in January 2038), and log in UTC; time zones are a display concern for the host.

Data integrity: CRC versus hash

Check	Detects	Does not protect against	Where it belongs
CRC-16 / CRC-32	Random bit errors, torn writes, burst errors	Deliberate modification (anyone can recompute it)	Every flash record, every frame
SHA-256 hash	Any change, with overwhelming probability	Deliberate modification if the attacker can also replace the hash	Large images, content identification, deduplication
HMAC or signature	Any change by someone without the key	Key theft	Firmware images, authenticated commands, data that must be trusted

Use a CRC wherever accidental corruption is the threat: it is cheap and the STM32 has hardware for CRC-32. Use a cryptographic check where an adversary is the threat. Firmware Security covers signatures.

Getting data off the device

Data on a device is worthless until someone can read it. Plan the extraction path at design time:

• UART or USB CDC dump. A shell command (log dump) streams records in framed binary or text. Simple and always available on the bench; give it a resume offset so a dropped transfer can continue.
• USB mass storage. The device appears as a drive with a log.csv on it. Convenient for non-technical users, but the PC and the firmware must not both write the filesystem at once: while the host has the drive mounted, firmware must treat the storage as read-only or stop logging.
• Over the air. The device uploads batches (CBOR or protobuf) over MQTT or HTTPS, marks them as sent only after the server acknowledges, and keeps unsent data across reboots. The sequence numbers from your log double as upload cursors.

A word on privacy and retention. Logs collect more than you intend: locations, usage patterns, identifiers, sometimes user content. Regulations such as the GDPR apply to device data as much as to web data. Store only what you need, document what is collected and why, set a retention limit (the circular log enforces one naturally), encrypt personal data at rest when the device can be stolen, and provide a way to erase it on decommissioning or ownership transfer.

Common mistakes

• Formatting the filesystem automatically whenever mount fails. Fix: format only on positive evidence of a blank device, and log it.
• Assuming data is safe after lfs_file_write() or f_write(). Fix: data is durable only after sync or close returns.
• Page program crossing a 256-byte boundary on SPI NOR, silently wrapping. Fix: split programs at page boundaries in the block device driver.
• Rewriting the same sector for every small update. Fix: append-only records spread over many sectors; do the wear arithmetic.
• Trusting SD cards with FAT to survive power loss. Fix: hold-up capacitance, brown-out detection, and a consistency check at boot.
• Writing a packed struct to a file that a server will decode. Fix: use an explicit serialization (CBOR, protobuf, or shift-based packing) with a version.
• Timestamping logs with an unsynchronized RTC and treating it as truth. Fix: boot count plus monotonic uptime on every record, and sync records when wall time is known.
• Measuring durations with wall-clock time. Fix: monotonic counters for durations, wall clock only for display and correlation.

Exercises

1. Compute the lifetime for a device logging a 64-byte record every 10 seconds into the last 512 KB of a 4 MB SPI NOR (100,000 cycles), assuming append-only writes with amplification 2. Then compute it for the same log in STM32F407 internal flash sectors 10 and 11 (two 128 KB sectors, 10,000 cycles). The internal-flash answer should come out shorter by more than a factor of 10.
2. Run evlog.c against the fake flash from Testing Embedded Software. Write Unity tests that append 5,000 records, re-run evlog_init(), and check that next_seq is 5,001 and that the oldest surviving record has the sequence number you expect. Then corrupt one byte of the newest record and confirm evlog_init() resumes after the previous one.
3. Add a “power cut” hook to the fake flash that aborts a program operation after N bytes. Loop N from 0 to 32 for a single append and verify after each that evlog_init() finds a consistent log with no program violations on the next append.
4. Integrate LittleFS on the host with a RAM-backed block device (an array, with the same erase and program rules as the fake flash). Run the boot counter 1,000 times, then write a 100 KB file, and print how many erases each block received. Change block_cycles from 500 to 50 and compare the spread.
5. Define a Sample protobuf schema with nanopb, encode 10 samples, and decode them with Python’s protobuf package on the host. Then add a new field in a second schema version and show that the old Python decoder still reads the new messages. Check the map file to see how much flash nanopb’s encoder adds.

Check yourself

Question 1:Firmware calls lfs_file_write() to append a record and the power fails 1 ms later, before lfs_file_close(). What does the next boot see?

A corrupted filesystem that must be reformatted The file as it was at the last sync or close, without the new record The new record half written The new record, because LittleFS writes synchronously

Question 2:A device stores a 32-bit counter at one fixed address in a 4 KB SPI NOR sector and updates it by erase-and-rewrite once per second. The datasheet says 100,000 cycles. Roughly how long until that sector wears out?

About 3 years About 4 months About 28 hours About 11 days

Question 3:An SPI NOR block device programs 300 bytes starting at address 0x1000 in a single page-program command. What ends up in flash?

All 300 bytes at 0x1000 to 0x112B The command is rejected with an error status The first 256 bytes at 0x1000, then the last 44 bytes programmed over the start of the same page Only the first 256 bytes; the rest are dropped

Question 4:In the circular log, power fails while a record is being programmed. On the next boot, why does evlog_init() skip that slot instead of writing the next record into it?

The slot's sequence number is lower than the newest record The slot fails its CRC so it is not data, but it is not blank either, and programming over it could need 0 to 1 transitions The sector is marked bad The CRC peripheral is disabled after a power failure

Question 5:A device timestamps log records with its RTC. The RTC is set by SNTP after Wi-Fi connects, about 30 seconds after boot. What is the most robust way to timestamp records written before the sync?

Write zero as the timestamp and discard those records on the server Wait for SNTP before logging anything Record boot count and monotonic uptime on every record, and log a sync record mapping uptime to epoch when time becomes known Use the RTC value anyway, since it is only 30 seconds off

Key takeaways

• Use raw sectors for one or two fixed things, a key-value store for settings, and a filesystem only when the data is file-shaped.
• A block device interface (read, program, erase, sync, geometry) lets one storage stack run on internal flash, SPI NOR, or SD.
• LittleFS gives power-loss resilience and wear leveling with bounded RAM; data is durable only after sync or close.
• FAT on SD cards is not power-fail safe; it needs hold-up power and boot-time checks.
• Write amplification decides lifetime: append-only logs across many sectors last centuries, in-place rewrites last days.
• CRC per record plus sequence numbers plus append-only writes makes a log that survives power loss at any instant.
• Timestamp with boot count and monotonic uptime; map to wall-clock time with sync records.

Next: Design Patterns for Embedded C, where the vtables, handles, and callbacks used throughout this lesson become a deliberate toolkit.