Skip to content

deflate compression method (dmcomp, a submodule) and the firmware SAL: FirmwareReady, Write/ReadPersistency - #341

Merged
JohnAmadis merged 4 commits into
developfrom
feat/deflate-and-firmware-sal
Oct 28, 2026
Merged

JohnAmadis merged 4 commits into
developfrom
feat/deflate-and-firmware-sal

Conversation

@JaxCalder

@JaxCalder JaxCalder commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What

Two additions, in two commits, made together so that the tools image (chocotechnologies/dmod) has to be built once.

1. deflate, a compression method of the DMFC files (dmcomp)

Dmod_Compression_* gets a second method, raw DEFLATE (RFC 1951) from choco-technologies/dmcomp (vendored in lib/third-party/dmcomp like FastLZ, with its license; the SDK installs it). The 35 modules of the dmod-boot firmware are 172 kB with it, 215 kB with FastLZ (zlib -9 gives the same as dmcomp).

  • Decoder: 1.7 kB of code on Cortex-M. Its tables (~3 kB) come from Dmod_Malloc (dmcomp_inflate_ws), not from the stack: a module is loaded from whatever thread asks for it, some of them have 4 kB of stack.
  • Encoder (Dmod_Compression_Pack): ~330 kB of malloc, for the host tools. DMOD_USE_DMCOMP_ENCODER is ON when DMOD_USE_STDLIB is, so a firmware (dmod-boot has no stdlib) gets the decoder only without any change in its configuration (+2.6 kB of code; checked by building the firmware). DMOD_USE_DMCOMP (ON) switches the method.
  • Nothing changes by default: fastlz stays the first method, the default of todmfc and of DMOD_COMPRESSION_METHOD – a loader built before this cannot read deflate. Use it with todmfc in.dmf out.dmfc deflate (level 9) or DMOD_COMPRESSION_METHOD=deflate.

2. SAL: Dmod_FirmwareReady, Dmod_WritePersistency, Dmod_ReadPersistency

For a platform with more than one firmware image (banks) and a boot loader that picks one:

  • int Dmod_FirmwareReady(void) – the firmware says that it has started properly; one that is on trial stays only if it does, otherwise the platform falls back to the previous one after a reset. What "properly" means is the firmware's decision (a console, the network, a service ...).
  • int Dmod_WritePersistency(uint32_t Id, const void* Data, size_t Size) – data that survives a reset and a firmware update, found by Id; writing 0 bytes (Data NULL) removes the item.
  • int Dmod_ReadPersistency(uint32_t Id, void* Buffer, size_t BufferSize) – returns the size of the whole item (a NULL buffer asks for the size only), -ENOENT when there is none.

Like Dmod_Reboot, they are implemented by the platform; the default (weak) implementations return -ENOSYS and do nothing (src/system/if/dmod_if_firmware.c).

Tested

  • New unit tests: Dmod_Compression_* with deflate on every level, against fastlz, damaged and cut data, no memory for the decoder (and the tables are given back); DMFC → DMF with both methods; the -ENOSYS defaults. The four tests that fail locally (tests_dmod_ctx, _ldr, _mgr, _dmfc_api: no test module built) fail the same on a clean develop.
  • From a fresh recursive clone of the branch: the build of the Docker image (two passes, examples, cmake --install --component tools), the Make based build of the library in both modes, the CI flow (MODULE then SYSTEM mode, dmod_loader on the example modules), a real module (dmffs) built as DMOD_MODE=DMOD_MODULE with DMOD_COMPRESSION_METHOD=deflate, dmod_loader loading a deflate .dmfc, the dmod-boot firmware for the STM32F746G-DISCO.

Notes

  • The submodule is at 0126ec1, the merge commit of dmcomp#2 on dmcomp's main (dmcomp_inflate_ws, which the decoder needs).
  • New with a submodule: a clone needs --recursive (or git submodule update --init --recursive; the configuration says so when the directory is empty). The CI checkouts take the submodules, Docker/build.sh initializes them before it builds the image (the Dockerfile copies the directory), and a project that has dmod as a submodule (dmod-boot) has to initialize the nested one.
  • Not changed (it was already so): Dmod_FromDMFC only warns when the decompressed size differs from the header's, so a cut fastlz file loads; a cut deflate one is refused.

🤖 Generated with Claude Code

JaxCalder and others added 3 commits October 28, 2026 14:00
Dmod_Compression_* has a second method, "deflate" (raw DEFLATE, RFC 1951), by dmcomp
(choco-technologies/dmcomp, vendored in lib/third-party/dmcomp like FastLZ is). The
modules of the dmod-boot firmware are 172 kB with it, and 215 kB with FastLZ.

* The decoder is 1.7 kB of code. Its tables (~3 kB) come from the heap
  (dmcomp_inflate_ws), not from the stack: a module is loaded from whatever thread asks
  for it, some of them have 4 kB of the stack.
* The encoder (Dmod_Compression_Pack) allocates ~330 kB with malloc and is for the host
  tools. DMOD_USE_DMCOMP_ENCODER is ON when DMOD_USE_STDLIB is, so a firmware (dmod-boot
  has no stdlib) gets the decoder only, with no change in its configuration: +2.6 kB.
* DMOD_USE_DMCOMP (ON) switches the method.
* fastlz stays the first method (the default of todmfc and of DMOD_COMPRESSION_METHOD):
  a loader built before this cannot read deflate. todmfc's default level depends on the
  method: 2 for fastlz, 9 for deflate.

Tests: Dmod_Compression_* with deflate on every level, against fastlz, damaged and cut data,
no memory for the decoder (the tables are given back); DMFC -> DMF with both methods. The
SDK installs the library with its license. Checked in the build of the Docker image
(two passes, examples, tools install), a real module built as DMOD_MODE=DMOD_MODULE with
DMOD_COMPRESSION_METHOD=deflate, and the dmod-boot firmware.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…sistency

A platform with more than one firmware image (banks, with a boot loader that picks
one) needs two things from the firmware, and the firmware needs them from the platform:

* Dmod_FirmwareReady(): the firmware says that it has started properly. A firmware
  that is on trial stays only if it does; otherwise the platform falls back to the
  previous one after a reset. What "properly" means is the firmware's own decision
  (a console, the network, a service ...), so it is the firmware that asks.
* Dmod_WritePersistency(Id, Data, Size) / Dmod_ReadPersistency(Id, Buffer, BufferSize):
  data that survives a reset and an update of the firmware (the settings, the secrets),
  kept apart from the images. An item is found by its Id; writing 0 bytes removes it;
  the read returns the size of the whole item (ask with a NULL buffer for the size).

Like Dmod_Reboot, these are implemented by the platform; the default (weak)
implementations return -ENOSYS and do nothing. Added now, together with the deflate
method, so that the tools image has to be built once.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
lib/third-party/dmcomp is the submodule choco-technologies/dmcomp (7314bf2) instead of
copied files, so the library has one place to be changed and tested. What builds it is
lib/third-party/dmod_dmcomp (the CMakeLists.txt, and the Makefile that it writes; there
are no sources in it): the same library dmod_dmcomp as before, for the CMake and the
Make based builds.

The checkouts of the CI workflows take the submodules, the script that builds the
Docker image initializes them (the Dockerfile copies the directory), and the SDK
installs the sources of the submodule where the build glue finds them. Without the
submodule the configuration says what to do (git submodule update --init --recursive).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@JaxCalder JaxCalder changed the title deflate compression method (dmcomp) and the firmware SAL: FirmwareReady, Write/ReadPersistency deflate compression method (dmcomp, a submodule) and the firmware SAL: FirmwareReady, Write/ReadPersistency Oct 28, 2026
dmcomp#2 (dmcomp_inflate_ws) is merged; the submodule was on its commit 7314bf2. The merge
commit has the same tree, so nothing changes in the build.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@JohnAmadis
JohnAmadis merged commit 8264cbb into develop Oct 28, 2026
6 checks passed
This was referenced Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants