Add production signed WiFi OTA updates

[ODevStudio]

Summary

Adds production signed WiFi pull OTA for v3.1.13+.

• Adds a signed release catalog and on-device release picker for compatible stable upgrades and downgrades.
• Verifies the exact fetched manifest bytes before parsing, then enforces HTTPS validation, GitHub release URL allowlisting, asset size, streamed SHA-256, and partition read-back SHA-256.
• Stages mandatory LittleFS updates after the application reboot and keeps the new application pending until its filesystem succeeds.
• Bounds recovery: target LittleFS is attempted twice, then the application rolls back. If a filesystem write started, the old application restores its matching LittleFS once; failed restore stops at UPDATE ERROR and directs the user to the HDS updater.
• Binds pending filesystem metadata to the running firmware version.
• Caps the signed catalog at 16 releases and 32,768 serialized bytes.
• Verifies each newly generated manifest signature against the public key compiled into the release firmware before publication.

Security Scope

• Signed manifests are mandatory for WiFi pull OTA; there is no unsigned fallback in the pull path.
• The existing ElegantOTA /update endpoint remains a separate LAN push/recovery path and is not covered by that claim.
• The firmware trust anchor is generated from HDS_OTA_MANIFEST_PUBLIC_KEY_PEM; release signing uses HDS_OTA_SIGNING_KEY_PEM.
• Preview, RC, draft, malformed, same-version, and pre-v3.1.13 entries are not selectable.
• Dev builds such as 3.1.13-dev compare using their numeric prefix while catalog releases remain strict stable versions.

Recovery

The current firmware stores signed target and rollback LittleFS metadata before writing the inactive application slot. After reboot, the new firmware gets two bounded target LittleFS attempts and is marked valid only after the filesystem verifies and mounts. Failure rolls the application back. If a target filesystem write began, the old application restores the signed LittleFS matching its version once. Normal weighing does not start while an installation is incomplete.

The single LittleFS partition has no independent rollback slot, so a failed rollback restore stops with UPDATE ERROR / Use HDS updater! rather than running with a partial installation.

Validation

• Native Unity tests cover strict stable parsing, dev-version comparison, numeric bounds, catalog ordering, deduplication, and capacity.
• python tools/test_generate_release_manifest.py
• python tools/test_pull_ota_contract.py
• python tools/test_release_workflow_contract.py
• python tools/test_ota_rollback_contract.py
• python tools/test_ota_public_key_header.py
• pio run -e esp32s3

Hardware recovery was tested with the stable public ODevStudio/openscale-ota-test v3.1.14 release. Its uploaded LittleFS image was intentionally corrupted while the signed manifest retained the correct hash. The device reported two target hash mismatches, rolled the application back, downloaded the signed v3.1.13/littlefs.bin rollback asset, patched LittleFS, and restarted successfully.

[tadelv]

PR #87 — Add production signed WiFi OTA updates

Solid security architecture: detached SHA-256 signature over the exact fetched manifest bytes, defense-in-depth asset-URL allowlisting, read-back partition SHA verification, staged firmware-then-LittleFS with NVS resume, and a rollback guard. The Python tooling has real unit tests. A few things to consider before merge, ranked.

Notable

1. ElegantOTA /update and the "no unsigned fallback" claim.
The PR body states "Signed manifests are mandatory; there is no unsigned fallback" — that's true for the pull path, but wifiOta() still runs ElegantOTA.begin(&server) from _wifi_init (src/hds.ino:368), keeping the unsigned browser-push /update endpoint live whenever WiFi is up. I understand ElegantOTA serves a different purpose (local/LAN push OTA), so it's not a regression introduced here. But the security framing in the PR body should be scoped to pull OTA so it isn't read as a total-OTA-security claim. Open question: is keeping ElegantOTA the long-term plan, or could the local-push use case eventually move onto our own signed channel so there's one OTA trust model instead of two?

2. Pending LittleFS resume blocks every boot for up to ~47–55 s when WiFi is down.
When a firmware-only OTA stages a LittleFS update, pullOtaHasPendingLittleFs() is true on every subsequent boot and pullOtaResumePendingLittleFs() runs unconditionally in setup() (end-of-setup). pullOtaEnsureWifi() waits up to HDS_OTA_WIFI_TIMEOUT_MS (45 s), plus NTP (8 s), plus the 2 s pullOtaFail delay. If the user's WiFi is temporarily down, the scale is unusable (no weighing) for ~47 s on every reboot until the network returns, with no escape. Consider: on failure, proceed to normal operation and let the next menu-triggered WiFi Update retry instead of blocking setup — or add an NVS skip-after-N-failures counter so boot-time resume gives up after a few tries and waits for an explicit user trigger.

3. Firmware-side OTA logic needs real unit tests, not grep contracts.
test_pull_ota_contract.py / test_ota_rollback_contract.py / test_release_workflow_contract.py assert that certain strings appear in the source files — they don't exercise behavior. The actual pure logic (pullOtaParseVersionTriplet, pullOtaCompareVersions, pullOtaBuildSelectableReleases, pullOtaManifestCompatible, pullOtaAddRelease ordering/dedup) is exactly the kind of pure, hardware-free code worth extracting and unit-testing with real cases. PlatformIO can run unit tests on the local machine with pio test -e native — no ESP32-S3 hardware needed — so this logic can be covered by real assertions, not source-string presence checks. The Python side (test_generate_release_manifest.py) already does this right (real unittest against real functions); the firmware side should match that bar.

Minor

4. A dev build should be able to pull a newer release.
pullOtaBuildSelectableReleases excludes releases where pullOtaCompareVersions(release, current) == 0, and pullOtaCompareVersions returns 0 (equal) when either side fails to parse. A dev FIRMWARE_VER like 3.1.13-dev fails pullOtaParseVersionTriplet (trailing -dev), compares as 0 against every release, and the picker excludes everything → "Newest stable" and exit. So a dev build can never pull OTA. Per discussion, the intent is that a dev build should be able to pull a later version. Suggest: parse a leading major.minor.patch prefix from the current version (ignore any -suffix) for comparison, so dev builds compare against their numeric base and can still select newer stable releases.

5. Latent u8g2 race: pull-ota task draws directly, bypassing otaDisplayMux.
The pull-ota task calls pullOtaDraw* (direct u8g2.firstPage/nextPage) on its own FreeRTOS task, while loop()'s b_ota branch calls processOtaDisplayUpdate(), which also draws to u8g2. In practice processOtaDisplayUpdate is a no-op unless an ElegantOTA push is in progress, so collision requires a browser-push OTA happening during a pull OTA — low probability. This is coupled to #1: if ElegantOTA stays, route pull-ota draws through the same otaDisplayMux/queue pattern; if local-push moves onto our own channel later, revisit then.

6. Local builds produce a WiFi Update menu item that always fails.
write_ota_public_key_header.py only runs in the release workflow; without HDS_OTA_MANIFEST_PUBLIC_KEY_PEM, pullOtaPublicKeyConfigured() is false and every attempt dies at "Signature failed" (fail-closed — correct). Fine by design, but a local dev hitting the menu sees a dead entry. Optional: gate the menu item, or emit a build-time warning when the key header is absent.

Info (acknowledged, no action needed)

• Rollback marks the app valid before the LittleFS write (README acknowledges). hdsOtaLocalChecksPass runs against the still-intact old FS, so it never validates the new FS. A persistent network outage after a firmware-only OTA leaves new firmware + corrupt/old FS with no rollback path; recovery depends on network returning, and #2's boot-block compounds this. Worth a sentence tying the two together in the README.
• Catalog chaining re-signs the whole catalog each release with the current key — TOFU on the compiled-in public key, no revocation. Standard for this scheme; just naming it.

Things that look right

• Signature is over the exact fetched body, then the same body is parsed — no parse/verify gap.
• Binary signature read via pullOtaReadHttpBytes (raw uint8_t), distinct from the text-body reader — binary-safe.
• pullOtaManifestCompatible checks chip, environment, partition schema, flash size, free sketch space, firmware size, FS label, actual FS partition size via esp_partition_find_first, and littlefs.size == fsPartitionSize.
• Read-back verification: pullOtaPartitionShaMatches re-hashes the partition via esp_partition_read after write.
• Release workflow verifies the previous catalog's signature before merging, builds the tagged commit (not main), creates as draft, confirms all 4 assets present, then un-drafts.
• pullOtaFail fail-closes (b_ota = false); Update.abort() on hash mismatch.

[ODevStudio]

Thanks for the detailed review. It was made against 19e9f327; the current head includes follow-up recovery, release-workflow, state-ownership, and native-test commits. Point by point:

1. Agreed. The PR description now scopes "no unsigned fallback" to WiFi pull OTA. ElegantOTA /update remains the separate LAN push/recovery path and HDS updater for USB. Unifying those trust models is outside this PR.

2. Addressed with a bounded transaction rather than allowing weighing during a partial installation. Pending recovery uses a 10-second WiFi attempt, target LittleFS is attempted at most twice, and the application is not marked valid until target LittleFS verifies. Failure rolls back the application. If a filesystem write started, the old application restores its signed matching LittleFS once; failed restore stops at UPDATE ERROR / Use HDS updater!.

   This path was exercised on hardware with ODevStudio/openscale-ota-test v3.1.14: the manifest expected target LittleFS SHA-256 fbbe429d..., while the intentionally corrupted uploaded asset was 8f4d8d87.... The device showed two hash mismatches, rolled back, then restored LittleFS from the signed v3.1.13 server asset (badbd98f...) and restarted successfully.

3. Partly addressed with executable native Unity tests over code used by the firmware. They cover strict/stuffixed version parsing, comparison, bounds, catalog insertion ordering, deduplication, and capacity. The ESP32-S3 build and existing integration contracts still cover wiring. Hardware-bound manifest compatibility and the reboot state machine were not forced into host mocks; the staged failure/rollback path was tested on the device as described above.

4. Fixed. A build such as 3.1.13-dev now compares using 3.1.13, can select 3.1.14+, and finds the matching stable 3.1.13 rollback LittleFS entry. Catalog entries themselves remain strict stable versions.

5. Not changed in this PR. The direct pull-OTA drawing path and ElegantOTA collision remains the low-probability race described. It should be handled together with the longer-term ElegantOTA decision rather than expanding this recovery change.

6. Intentionally unchanged. Local builds remain fail-closed when no public key is configured; gating the menu is optional follow-up UX work.

The README and PR description now reflect the current mark-valid timing, bounded attempts, application rollback, rollback LittleFS restore, and terminal updater error state.

[tadelv]

Follow-up review (adding to tadelv's earlier review)

The earlier review points are addressed in the current head. Adding a few new observations:

Suggestion: Consider pinning the OTA task to core 0

pullOtaUpdateTask is created with xTaskCreate (unpinned), so the FreeRTOS scheduler can place it on either core. During firmware download (pullOtaStreamAsset with U_FLASH), the web server is still running — stopWebServer() is only called for the LittleFS write (U_SPIFFS) via pullOtaPauseFilesystemServices(). If the OTA task lands on core 1 (same as CONFIG_ASYNC_TCP_RUNNING_CORE=1), the blocking HTTP download + streamed SHA-256 computation competes with AsyncTCP's event loop.

FreeRTOS preemption should prevent total starvation, but pinning to core 0 with xTaskCreatePinnedToCore(..., 0, ...) would eliminate the question entirely — core 0 is idle during OTA since loop() short-circuits on b_ota.

Note: File size

pull_ota.h is 1368 lines. Not a merge blocker (the existing codebase already has large files), but worth noting as a future split candidate — the pending-LittleFS NVS state machine (~300 lines of pullOtaStorePendingLittleFs / pullOtaLoadPendingLittleFs / pullOtaActivateRollbackLittleFs / etc.) is self-contained and could be its own header.

[ODevStudio]

Implemented.
Marginal gain.
AsyncTCP at priority 10 would already preempt an OTA task at priority 1 if both were runnable on the same core. ESP-IDF schedules the highest-priority compatible ready task on each core.
Pinning therefore improves workload placement and predictability, but it is not required to prevent AsyncTCP starvation.
Core 0 is also not idle during OTA. By default, ESP-IDF places the Wi-Fi task at priority 23, the timer task at priority 22, and the default event task at priority 20 on core 0; lwIP runs at priority 18 with configurable affinity. Because the OTA task remains at priority 1, these system tasks can preempt it, so the likely trade-off is slower OTA processing rather than impaired Wi-Fi correctness.