From b924d0a97aa7cbddb0b45b6324ca70fcd300a8f9 Mon Sep 17 00:00:00 2001 From: Thierry Laurion Date: Wed, 2 Sep 2026 15:11:20 -0400 Subject: [PATCH] seed_package_mirror.sh: dynamically seed all boards & toolchains Rewrite bin/seed_package_mirror.sh to dynamically discover and seed every board instead of a hardcoded list of five representative boards. Fixes linuxboot/heads#2177: the Purism mirror only cached tarballs for the previously hardcoded boards, missing the musl-cross-make component tarballs moved under packages/ by PR #2174. Changes: - Discover all boards via boards/*/*.config instead of a hardcoded list, so new boards, architectures, and coreboot forks are picked up automatically. - Run 'make BOARD= packages' for each board. The define_module 'packages:' target (Makefile) pre-downloads every versioned module tarball plus coreboot crossgcc toolchain tarballs. - Clean only the cached tarballs (packages/*/*) and .canary build stamps (build/**/.canary, as in the real.remove_canary_files-* Makefile helper) so module tarballs are re-downloaded + hash-verified and coreboot forks re-emit their crossgcc toolchain, without wiping compiled artifacts. - Redirect make stdin from /dev/null to avoid an intermittent 'syntax error near unexpected token do' under set -e + 'if ! make'. - Tolerate per-board failures (dead upstream URLs) without aborting the whole seed, reporting a summary at the end. - Copy tarballs arch-agnostically (packages/*/) instead of hardcoding x86/ppc64. Signed-off-by: Thierry Laurion --- bin/seed_package_mirror.sh | 46 ++++++++++++++++++++++++++++++-------- 1 file changed, 37 insertions(+), 9 deletions(-) diff --git a/bin/seed_package_mirror.sh b/bin/seed_package_mirror.sh index 3e66a7b87..01637d096 100755 --- a/bin/seed_package_mirror.sh +++ b/bin/seed_package_mirror.sh @@ -48,17 +48,45 @@ cd "$(dirname "${BASH_SOURCE[0]}")/.." echo echo "Cleaning build to download all packages..." -# fetch packages for representative boards -rm -rf build/x86 build/ppc64 -rm -rf packages/x86 packages/ppc64 +# Clear cached tarballs and .canary build stamps (modeled on the Makefile +# real.remove_canary_files-extract_patch_rebuild_what_changed helper), so +# module tarballs are re-downloaded and hash-verified AND coreboot forks +# re-enter their build to produce the crossgcc toolchain tarballs that +# seed the mirror. Compiled artifacts are left intact — only the stamps +# and cached sources that gate re-fetch are removed. +rm -rf packages/*/* +find build -type f -name ".canary" -delete 2>/dev/null || true echo echo "Downloading packages..." -make packages BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp -make packages BOARD=UNTESTED_talos-2 # newt, PPC -make packages BOARD=librem_l1um_v2 # TPM2 -make packages BOARD=EOL_librem_l1um # coreboot 4.11 -make packages BOARD=EOL_x230-maximized # io386 +# Discover all boards dynamically and fetch every module tarball. +# 'make BOARD=X packages' triggers the 'packages:' target defined by +# define_module (Makefile:571) for every versioned module that board +# enables. Tarballs are cached by filename in packages//, so +# redundant downloads for boards sharing tarballs cost only Make +# setup overhead (~2s each). +# A board that fails (e.g. dead upstream URL) is reported but does not +# abort seeding the remaining boards. +failed=0 +boards=() +for cfg in boards/*/[!.]*.config; do + [ -f "$cfg" ] || continue + boards+=("$(basename "$(dirname "$cfg")")") +done +for board in "${boards[@]}"; do + echo " make BOARD=$board packages" + # Redirect stdin from /dev/null so make/wget cannot consume the + # script's stdin — under set -e + if ! that can cause bash to + # misparse the loop and produce an intermittent syntax error. + if ! make BOARD="$board" packages &2 + failed=$((failed+1)) + fi +done +[ "$failed" -eq 0 ] || echo "Warning: $failed board(s) failed to fetch all packages" >&2 echo echo "Copying to mirror directory..." mkdir -p "$ARG_MIRROR_DIR" -cp packages/x86/* packages/ppc64/* "$ARG_MIRROR_DIR/" +for arch_dir in packages/*/; do + [ -d "$arch_dir" ] || continue + cp "$arch_dir"* "$ARG_MIRROR_DIR/" 2>/dev/null || true +done