Skip to content

bin/seed_package_mirror.sh: dynamically seed all boards & toolchains - #2199

Open
tlaurion wants to merge 1 commit into
linuxboot:masterfrom
tlaurion:seed-package-mirror-dynamic
Open

bin/seed_package_mirror.sh: dynamically seed all boards & toolchains#2199
tlaurion wants to merge 1 commit into
linuxboot:masterfrom
tlaurion:seed-package-mirror-dynamic

Conversation

@tlaurion

@tlaurion tlaurion commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Fixes #2177

Problem

bin/seed_package_mirror.sh seeded the Purism package mirror using a
hardcoded list of five representative boards:

  • qemu-coreboot-fbwhiptail-tpm1-hotp (base x86)
  • UNTESTED_talos-2 (PPC)
  • librem_l1um_v2 (TPM2)
  • EOL_librem_l1um (coreboot 4.11)
  • EOL_x230-maximized (io386)

After #2174 moved the musl-cross-make component tarballs under
packages/, the mirror did not get all of them cached on demand,
so builds served those 7 tarballs from the Purism mirror (MIRROR_USED
in build/mirror_fallbacks.log) rather than from primary sources.

The hardcoded list is also fragile: boards are added/renamed and
coreboot forks change over time, so the mirror was always at risk of
silently missing newly added module tarballs.

Change

Rewrite bin/seed_package_mirror.sh to:

  • Discover all boards dynamically via boards/*/*.config instead of a
    hardcoded list, so new boards, architectures, and coreboot forks are
    picked up automatically with no maintenance.
  • Run make BOARD=<board> packages for every board. The define_module
    packages: target pre-downloads every versioned module tarball plus the
    coreboot crossgcc toolchain tarballs, which is what actually gets copied
    into the mirror.
  • Clean only the cached tarballs (packages/*/*) and .canary build
    stamps (build/**/.canary, modeled on the
    real.remove_canary_files-extract_patch_rebuild_what_changed Makefile
    helper)
    so module tarballs are re-downloaded + hash-verified and
    coreboot forks re-emit their crossgcc toolchain, without wiping already
    compiled artifacts.
  • Redirect make stdin from /dev/null to avoid an intermittent
    bash: syntax error near unexpected token 'do' that occurred under
    set -e + if ! make ... when make/wget consumed the script's stdin.
  • Tolerate per-board failures (e.g. a dead upstream URL such as the
    coreboot 4.11 acpica tarball) without aborting the whole seed,
    reporting a summary of failed boards at the end.
  • Copy tarballs arch-agnostically (packages/*/) instead of
    hardcoding x86/ppc64.

Verification

  • A full run over all boards completes with exit 0.
  • Mirror contains the complete tarball set (90 files), including all 7
    musl-cross-make component tarballs that were the core of Make sure Purism cache all tarballs from merged https://github.com/linuxboot/heads/pull/2174 #2177.
  • build/mirror_fallbacks.log shows the 7 musl-cross-make tarballs are now
    fetched from primary sources (PRIMARY) instead of MIRROR_USED.
  • Coreboot forks are not fully re-cloned: clearing their .canary
    re-runs the Makefile branch that re-fetches the pinned commit and
    regenerates the .canary (observed: .canary present after the run,
    fork dirs preserved), yielding the crossgcc toolchain tarballs without a
    full re-clone.

Copilot AI lite review requested due to automatic review settings September 2, 2026 19:12

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review any files in this pull request.


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@tlaurion
tlaurion force-pushed the seed-package-mirror-dynamic branch from 14ac392 to 1091d46 Compare September 2, 2026 19:18
@tlaurion

tlaurion commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

@JonathonHall-Purism should be future proof and make sure everything currently maintained is mirrored.

Rewrite bin/seed_package_mirror.sh to dynamically discover and seed every
board instead of a hardcoded list of five representative boards.

Fixes linuxboot#2177: the Purism mirror only cached tarballs for the
previously hardcoded boards, missing the musl-cross-make component tarballs
moved under packages/ by PR linuxboot#2174.

Changes:
- Discover all boards via boards/*/*.config instead of a hardcoded list,
  so new boards, architectures, and coreboot forks are picked up
  automatically.
- Run 'make BOARD=<board> packages' for each board. The define_module
  'packages:' target (Makefile) pre-downloads every versioned module
  tarball plus coreboot crossgcc toolchain tarballs.
- Clean only the cached tarballs (packages/*/*) and .canary build stamps
  (build/**/.canary, as in the real.remove_canary_files-* Makefile helper)
  so module tarballs are re-downloaded + hash-verified and coreboot forks
  re-emit their crossgcc toolchain, without wiping compiled artifacts.
- Redirect make stdin from /dev/null to avoid an intermittent 'syntax
  error near unexpected token do' under set -e + 'if ! make'.
- Tolerate per-board failures (dead upstream URLs) without aborting the
  whole seed, reporting a summary at the end.
- Copy tarballs arch-agnostically (packages/*/) instead of hardcoding
  x86/ppc64.

Signed-off-by: Thierry Laurion <insurgo@riseup.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make sure Purism cache all tarballs from merged https://github.com/linuxboot/heads/pull/2174

2 participants