Skip to content

ci: bump packager base image from bullseye to bookworm - #1925

Merged
ShubhenduSinghF5 merged 3 commits into
nginx:dev-v2from
ShubhenduSinghF5:ss-fix-packager-bookworm
Sep 10, 2026
Merged

ShubhenduSinghF5 merged 3 commits into
nginx:dev-v2from
ShubhenduSinghF5:ss-fix-packager-bookworm

Conversation

@ShubhenduSinghF5

Copy link
Copy Markdown

Summary

Debian 11 (bullseye) reached its LTS end-of-life on 2026-08-31. The Debian LTS team has stopped re-signing the bullseye-security suite, so its InRelease file's Valid-Until: timestamp is now expired. apt-get install inside the CI packager image fails hard as a result:

E: Release file for http://deb.debian.org/debian-security/dists/bullseye-security/InRelease
   is expired (invalid since 1d 9h 52min 52s). Updates for this repository will not be applied.
...
The command '/bin/sh -c apt-get update && apt-get install -y make jq gnupg gnupg1 ...' returned a non-zero code: 100

This breaks the Build unsigned snapshot and Build signed snapshot jobs on every open PR against dev-v2 right now (not tied to any single PR — it's environmental drift).

Fix

Bump the packager base image one release forward:

-FROM docker.io/golang:1.24-bullseye AS base
+FROM docker.io/golang:1.24-bookworm AS base
  • bookworm = Debian 12, current stable, security suite refreshed regularly through 2028.
  • All apt packages the packager installs (gpgv1, monkeysphere, aptly, debsig-verify, createrepo-c, dnf, rpm, and the lib*-dev family) are available in bookworm.
  • Only the internal CI builder image changes — the .deb / .rpm / .apk artifacts we ship to customers are unchanged (they are produced by nfpm inside this container).

Scope

  • scripts/packages/packager/Dockerfile (1 line)

Test plan

CI on this PR will exercise the change end-to-end:

  • Build unsigned snapshot — must go green (this is the currently-red job).
  • All other jobs unchanged.

Locally I confirmed that bookworm is the only bullseye reference in an active Dockerfile (test/docker/performance/Dockerfile has a commented-out 1~bullseye PKG_RELEASE line that is inert; the 500+ matches under pkgs*.nginx.* are published customer package filenames, unrelated to the build image).

Risk

Very low. Base image bump on a builder-only container.

Related

Blocks every open PR that runs build-unsigned-snapshot — including #1837 (nginx master process bug fix).

Debian 11 (bullseye) reached LTS end-of-life on 2026-08-31. The bullseye-security suite is no longer being re-signed, so its InRelease file '"Valid-Until"' timestamp has expired. apt-get install now fails inside the CI packager image with:

  E: Release file for .../bullseye-security/InRelease is expired

This breaks the "Build unsigned snapshot" and "Build signed snapshot" jobs on every PR against dev-v2. Bump the base image to golang:1.24-bookworm (Debian 12, currently the stable release with a fresh security suite). All apt packages used by the packager image (gpgv1, monkeysphere, aptly, debsig-verify, createrepo-c, dnf, rpm, and the lib*-dev family) are still available in bookworm.

Signed-off-by: shusingh <shu.singh@f5.com>
@ShubhenduSinghF5
ShubhenduSinghF5 requested a review from a team as a code owner September 9, 2026 07:13
@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (dev-v2@ed76366). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff            @@
##             dev-v2    #1925   +/-   ##
=========================================
  Coverage          ?   52.16%           
=========================================
  Files             ?       51           
  Lines             ?     3518           
  Branches          ?        0           
=========================================
  Hits              ?     1835           
  Misses            ?     1538           
  Partials          ?      145           

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed76366...ff25966. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Debian bookworm dropped gnupg1, gpgv1 and monkeysphere (all GPG 1.x-era tooling). apt-get install fails on bookworm with:

  E: Package 'monkeysphere' has no installation candidate

None of these three packages are actually invoked by any script, Makefile, or packaging helper in this repo (grep confirms zero references outside this Dockerfile). They were vestigial residue from an era when debsig-verify needed PGP v3 tooling; modern gpg 2.x (installed as the gnupg package, still present) handles everything the packager and debsig-verify need.

Also drops a pre-existing duplicate "make" in the same apt-get line.

Signed-off-by: shusingh <shu.singh@f5.com>
Debian bullseye reached LTS end-of-life on 2026-08-31 and the
bullseye-security InRelease signature expired on 2026-09-08, causing
apt-get update to fail in every debian:bullseye-slim CI job since.

Debian bullseye is no longer a supportable target for nginx-agent v2.47+.
Remove it from the integration test matrix in ci.yml and update the
supported-OS table in Makefile accordingly. Customers still on bullseye
should upgrade to bookworm; the packages built by the bookworm packager
(bumped in the previous commit) remain compatible with bullseye runtimes
that pull from unfrozen third-party repos.

@devbgv devbgv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ShubhenduSinghF5
ShubhenduSinghF5 merged commit baea216 into nginx:dev-v2 Sep 10, 2026
30 checks passed
ShubhenduSinghF5 added a commit to ShubhenduSinghF5/agent that referenced this pull request Sep 16, 2026
Debian bullseye reached LTS end-of-life on 2026-08-31 and the
bullseye-security InRelease signature expired 2026-09-08, breaking
apt-get update against the default deb.debian.org sources. PR nginx#1925
worked around this by dropping bullseye from the integration test
matrix entirely.

NGINX OSS and NGINX Plus still support bullseye, so agent CI should
continue to validate against it. Rewrite the base image's apt
sources.list to archive.debian.org for bullseye-slim only, guarded by
a VERSION_CODENAME check in /etc/os-release so no other matrix row
is affected. archive.debian.org is a Debian-project-maintained mirror
that serves EOL releases with a non-expiring signing key already
present in the base image's debian-archive-keyring package.

Base image registry is unchanged: still docker.io/library/debian.
No new external hosts are contacted from the shipped Agent binary
or its packaged artifacts; this affects test/docker/ image builds
only.
ShubhenduSinghF5 added a commit that referenced this pull request Sep 17, 2026
…1937)

* ci: restore debian bullseye integration test via archive.debian.org

Debian bullseye reached LTS end-of-life on 2026-08-31 and the
bullseye-security InRelease signature expired 2026-09-08, breaking
apt-get update against the default deb.debian.org sources. PR #1925
worked around this by dropping bullseye from the integration test
matrix entirely.

NGINX OSS and NGINX Plus still support bullseye, so agent CI should
continue to validate against it. Rewrite the base image's apt
sources.list to archive.debian.org for bullseye-slim only, guarded by
a VERSION_CODENAME check in /etc/os-release so no other matrix row
is affected. archive.debian.org is a Debian-project-maintained mirror
that serves EOL releases with a non-expiring signing key already
present in the base image's debian-archive-keyring package.

Base image registry is unchanged: still docker.io/library/debian.
No new external hosts are contacted from the shipped Agent binary
or its packaged artifacts; this affects test/docker/ image builds
only.

* ci: switch bullseye-slim base image to debian/eol

Previous commit rewrote /etc/apt/sources.list to archive.debian.org
inside the Dockerfile. That fixed InRelease expiry, but exposed a
second failure mode: debian:bullseye-slim ships preinstalled packages
from bullseye-security (e.g. perl-base 5.32.1-4+deb11u5, gpgv
2.2.27-2+deb11u3) that don't exist in archive.debian.org, so apt's
solver reports unmet dependencies:

    gnupg : Depends: gpgv (< 2.2.27-2+deb11u2.1~)
            but 2.2.27-2+deb11u3 is to be installed
    perl  : Depends: perl-base (= 5.32.1-4+deb11u3)
            but 5.32.1-4+deb11u5 is to be installed

Switch to Debian's official EOL archive image
(docker.io/debian/eol:bullseye-slim), which is built directly from
archive.debian.org — its preinstalled packages and repo indices are
mutually consistent. Same publisher, same registry (docker.io),
signed with the same debian-archive-keyring key.

Scope the swap to OS_RELEASE=debian and OS_VERSION=bullseye-slim only
(Makefile conditional), so bookworm and every other matrix row are
byte-for-byte unchanged. OS_RELEASE stays 'debian' so the downstream
nginx-agent apt URL (packages.nginx.org/nginx-agent/debian/) is
unaffected.

Drop the now-redundant sed block from
test/docker/nginx-oss/deb/Dockerfile — debian/eol's sources.list
already targets archive.debian.org.

* ci: use debian/eol maintainer-provided snapshot URLs for bullseye apt

The debian/eol:bullseye-slim image was rebuilt after Debian 11 LTS EOL to
point 'bullseye' and 'bullseye-updates' at archive.debian.org, but left
'bullseye-security' on deb.debian.org where the security pool has been
pruned (InRelease still resolves but individual .deb files 404).

The image's own /etc/apt/sources.list ships snapshot.debian.org URLs
(commented) at the timestamp the image was built (20250809T133719Z),
providing a fully self-consistent frozen apt world where preinstalled
packages match repo indices. Enable those URLs plus the standard
Acquire::Check-Valid-Until=false for archived snapshots.

No version pins and no --allow-downgrades; the snapshot is the
maintainer's documented answer for this scenario.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants