Serve dated snapshots of Alpine APK indexes

README.md

@@ -734,6 +734,11 @@ signature verification keeps working. Indexes use the metadata cache
 (`metadata_ttl`, stale fallback); `.apk` packages are stored in the shared
 artifact cache and remain available when the upstream is unreachable.
 
+Set `upstream.apk_snapshots` (e.g. `"90d"`) to keep replaced indexes and serve
+them by date at `/apk/alpine@2026-10-01/v3.22/main`, so pinned package versions
+stay installable after the mirror moves on. See
+[Alpine APK snapshots](docs/configuration.md#alpine-apk-snapshots).
+
 To proxy other mirrors or private repositories, configure named upstreams
 under `upstream.apk` (this replaces the built-in default; re-add `alpine` if
 you still want it):

docs/configuration.md

@@ -323,6 +323,7 @@ Each upstream used by a built-in package route can be set in YAML or JSON under
 | `upstream.rpm` | `PROXY_UPSTREAM_RPM` | `https://dl.fedoraproject.org/pub/fedora/linux` |
 | `upstream.homebrew_api` | `PROXY_UPSTREAM_HOMEBREW_API` | `https://formulae.brew.sh/api` |
 | `upstream.homebrew_artifact` | `PROXY_UPSTREAM_HOMEBREW_ARTIFACT` | `https://ghcr.io` |
+| `upstream.apk_snapshots` | `PROXY_UPSTREAM_APK_SNAPSHOTS` | disabled; see [Alpine APK snapshots](#alpine-apk-snapshots) |
 
 Private, ULA, CGNAT, and loopback addresses are rejected by default. Add each private upstream hostname or IP address to `upstream.allow_private_hosts`. The matching environment variable accepts a comma-separated list. Loopback upstreams also require `upstream.allow_loopback: true`. That setting permits upstream requests and redirects to reach any loopback address.
 
@@ -576,6 +577,8 @@ with the metadata TTL and served byte-for-byte unchanged so apk signature
 verification keeps working; `.apk` packages use the shared artifact cache.
 When `upstream.apk` is empty, a single repository named `alpine` pointing at
 the official mirror is available; configuring any entry replaces that default.
+Earlier index versions can be kept and served by date; see
+[Alpine APK snapshots](#alpine-apk-snapshots).
 
 ```yaml
 upstream:
@@ -623,6 +626,53 @@ as signatures and keys, is passed through uncached. When `upstream.freebsd` is e
 repository named `freebsd` pointing at `https://pkg.freebsd.org` is available;
 configuring any entry replaces that default.
 
+### Alpine APK snapshots
+
+```yaml
+upstream:
+  # Keep replaced APK indexes for 90 days.
+  apk_snapshots: "90d"
+```
+
+Alpine mirrors only keep the current build of each package, so after an
+upgrade `apk add curl=8.5.0-r0` fails because the index no longer lists that
+build. With `upstream.apk_snapshots` set, the proxy keeps every version of an
+index (and of a detached signature) it stores, and serves them under a dated
+repository name:
+
+```
+http://localhost:8080/apk/alpine@2026-10-01/v3.22/main
+```
+
+`/apk/{name}@{YYYY-MM-DD}/…` returns the index that was current at the end of
+that day (UTC). The kept files are the signed upstream bytes, so apk verifies
+them as usual. A day before the first stored version of an index, or a day
+that has not started yet, is `404`. Today's snapshot can still change until the
+day ends. Packages under a snapshot path are the same as under the plain path:
+they come from the artifact cache, or from upstream if not cached; a build the
+upstream has dropped and the proxy never cached is `404`. A configured
+repository whose name contains `@` takes precedence over a snapshot path.
+
+Snapshots are recorded when an index is requested through the plain path, so
+they only exist for repositories clients use. A version counts from the first
+time the proxy served it, which can be up to `metadata_ttl` after upstream
+published it. A version is deleted once it was replaced more than the
+configured duration ago; the newest version of each index is always kept.
+Durations take a `d` suffix; empty or `"0"` disables snapshots. Snapshots need
+`cache_metadata`. Their files live under `_snapshots/` in storage and do not
+count towards `storage.max_size`.
+
+Turning snapshots off, or `cache_metadata` off, leaves the kept versions in
+place. To remove them, first set a short duration such as `"1h"` and let the
+hourly prune delete everything but the newest version of each index.
+
+Snapshots keep indexes, not packages. A package listed by an old snapshot can
+still be evicted by `storage.max_size` or [retention](#retention), and is then
+only available while upstream still has it. Retention counts from a package's
+last download, so a pinned build nobody installed for longer than the `alpine`
+retention is evicted even while a snapshot lists it. Use package rules with
+`"0"` for builds that must stay installable.
+
 ### Go checksum database
 
 ```yaml

internal/config/config.go

@@ -773,6 +773,12 @@ type UpstreamConfig struct {
 	// Default when empty: {"alpine": "https://dl-cdn.alpinelinux.org/alpine"}.
 	APK map[string]string `json:"apk" yaml:"apk"`
 
+	// APKSnapshots keeps earlier versions of Alpine APK indexes for this
+	// long after upstream replaced them, and serves them at
+	// /apk/{name}@{YYYY-MM-DD}/. Accepts a "d" suffix, e.g. "90d".
+	// Empty or "0" disables snapshots.
+	APKSnapshots string `json:"apk_snapshots" yaml:"apk_snapshots"`
+
 	// Arch maps repository names to Arch Linux pacman mirror base URLs,
 	// served at /arch/{name}/. The remaining request path mirrors the
 	// upstream layout, e.g. /arch/archlinux/core/os/x86_64/core.db.
@@ -881,6 +887,11 @@ func (u *UpstreamConfig) Validate() error {
 	if err := validateNamedUpstreams("upstream.apk", u.APK); err != nil {
 		return err
 	}
+	if u.APKSnapshots != "" {
+		if _, err := parseRetentionDuration("upstream.apk_snapshots", u.APKSnapshots); err != nil {
+			return err
+		}
+	}
 	if err := validateNamedUpstreams("upstream.arch", u.Arch); err != nil {
 		return err
 	}
@@ -1462,6 +1473,7 @@ func (c *Config) LoadFromEnv() {
 	setEnvString(&c.Upstream.Swift, "PROXY_UPSTREAM_SWIFT")
 	setEnvString(&c.Upstream.OCIDefault, "PROXY_UPSTREAM_OCI_DEFAULT")
 	setEnvString(&c.Upstream.Debian, "PROXY_UPSTREAM_DEBIAN")
+	setEnvString(&c.Upstream.APKSnapshots, "PROXY_UPSTREAM_APK_SNAPSHOTS")
 	setEnvString(&c.Upstream.RPM, "PROXY_UPSTREAM_RPM")
 	setEnvString(&c.Upstream.HomebrewAPI, "PROXY_UPSTREAM_HOMEBREW_API")
 	setEnvString(&c.Upstream.HomebrewArtifact, "PROXY_UPSTREAM_HOMEBREW_ARTIFACT")
@@ -1959,6 +1971,20 @@ func (c *Config) ParseGradleBuildCacheMaxUploadSize() int64 {
 	return size
 }
 
+// ParseAPKSnapshots returns how long earlier Alpine APK indexes are kept
+// after upstream replaced them. Returns 0 when snapshots are disabled or the
+// value is invalid.
+func (c *Config) ParseAPKSnapshots() time.Duration {
+	if c.Upstream.APKSnapshots == "" {
+		return 0
+	}
+	d, err := parseDayDuration(c.Upstream.APKSnapshots)
+	if err != nil || d < 0 {
+		return 0
+	}
+	return d
+}
+
 // ParseGradleBuildCacheMaxAge returns age-based eviction threshold.
 // Returns 0 when disabled or invalid.
 func (c *Config) ParseGradleBuildCacheMaxAge() time.Duration {

internal/config/config_test.go

@@ -2057,6 +2057,41 @@ upstream:
 	}
 }
 
+func TestAPKSnapshots(t *testing.T) {
+	cfg := Default()
+	if got := cfg.ParseAPKSnapshots(); got != 0 {
+		t.Errorf("default ParseAPKSnapshots = %v, want 0", got)
+	}
+
+	for value, want := range map[string]time.Duration{
+		"90d": 90 * 24 * time.Hour,
+		"12h": 12 * time.Hour,
+		"0":   0,
+	} {
+		cfg.Upstream.APKSnapshots = value
+		if err := cfg.Validate(); err != nil {
+			t.Errorf("Validate(%q): %v", value, err)
+		}
+		if got := cfg.ParseAPKSnapshots(); got != want {
+			t.Errorf("ParseAPKSnapshots(%q) = %v, want %v", value, got, want)
+		}
+	}
+
+	for _, value := range []string{"forever", "-1d"} {
+		cfg.Upstream.APKSnapshots = value
+		if err := cfg.Validate(); err == nil {
+			t.Errorf("Validate(%q) succeeded, want an error", value)
+		}
+	}
+
+	cfg = Default()
+	t.Setenv("PROXY_UPSTREAM_APK_SNAPSHOTS", "30d")
+	cfg.LoadFromEnv()
+	if cfg.Upstream.APKSnapshots != "30d" {
+		t.Errorf("APKSnapshots from env = %q, want 30d", cfg.Upstream.APKSnapshots)
+	}
+}
+
 func TestCooldownShortest(t *testing.T) {
 	for _, tc := range []struct {
 		name string

internal/database/database_test.go

@@ -613,7 +613,7 @@ func createTestPostgresDB(t *testing.T) *DB {
 	// Drop and recreate every table CreateSchema creates for clean test state;
 	// leftover migration records make the next CreateSchema fail on the
 	// migrations primary key.
-	tables := []string{"artifacts", "pending_deletes", "versions", "packages", "vulnerabilities", "metadata_cache", "metadata_dates", "job_runs", "fetch_leases", "upstream_holds", "migrations", "schema_info"}
+	tables := []string{"artifacts", "pending_deletes", "versions", "packages", "vulnerabilities", "metadata_cache", "metadata_dates", "job_runs", "fetch_leases", "upstream_holds", "metadata_snapshots", "migrations", "schema_info"}
 	for _, table := range tables {
 		_, _ = db.Exec("DROP TABLE IF EXISTS " + table + " CASCADE")
 	}

internal/database/metadata_snapshots.go

@@ -0,0 +1,156 @@
+package database
+
+import (
+	"database/sql"
+	"fmt"
+	"math"
+	"time"
+)
+
+// MetadataSnapshot is one kept version of a metadata document, so a request
+// for a past date can be answered with the document that was current then.
+// It is valid from ValidFromMS (Unix milliseconds, the time the proxy first
+// stored it) until the next snapshot of the same ecosystem and name.
+type MetadataSnapshot struct {
+	ID              int64          `db:"id"`
+	Ecosystem       string         `db:"ecosystem"`
+	Name            string         `db:"name"`
+	StoragePath     string         `db:"storage_path"`
+	ContentType     sql.NullString `db:"content_type"`
+	ContentEncoding sql.NullString `db:"content_encoding"`
+	ContentDigest   string         `db:"content_digest"`
+	Size            int64          `db:"size"`
+	ValidFromMS     int64          `db:"valid_from_ms"`
+}
+
+// ValidFrom returns the time the snapshot became current.
+func (s *MetadataSnapshot) ValidFrom() time.Time {
+	return time.UnixMilli(s.ValidFromMS)
+}
+
+const metadataSnapshotColumns = `id, ecosystem, name, storage_path, content_type, content_encoding,
+	content_digest, size, valid_from_ms`
+
+// MetadataSnapshotAt returns the snapshot of ecosystem and name that was
+// current at at: the newest one that became current before it. It returns nil
+// when there is none.
+func (db *DB) MetadataSnapshotAt(ecosystem, name string, at time.Time) (*MetadataSnapshot, error) {
+	return db.metadataSnapshotBefore(ecosystem, name, at.UnixMilli())
+}
+
+// LatestMetadataSnapshot returns the newest snapshot of ecosystem and name,
+// or nil when there is none.
+func (db *DB) LatestMetadataSnapshot(ecosystem, name string) (*MetadataSnapshot, error) {
+	return db.metadataSnapshotBefore(ecosystem, name, math.MaxInt64)
+}
+
+func (db *DB) metadataSnapshotBefore(ecosystem, name string, beforeMS int64) (*MetadataSnapshot, error) {
+	var snap MetadataSnapshot
+	err := db.Get(&snap, db.Rebind(`SELECT `+metadataSnapshotColumns+`
+		FROM metadata_snapshots
+		WHERE ecosystem = ? AND name = ? AND valid_from_ms < ?
+		ORDER BY valid_from_ms DESC, id DESC
+		LIMIT 1`), ecosystem, name, beforeMS)
+	if err == sql.ErrNoRows {
+		return nil, nil
+	}
+	if err != nil {
+		return nil, fmt.Errorf("reading metadata snapshot: %w", err)
+	}
+	return &snap, nil
+}
+
+// AddMetadataSnapshot records snap as the version of its document current
+// from snap.ValidFromMS, unless the snapshot current at that time already has
+// its digest, and reports whether it did. On Postgres it first takes an
+// advisory lock on the document, so two replicas can't both read the previous
+// digest and both insert. The casts let Postgres type the parameters of the
+// INSERT ... SELECT.
+func (db *DB) AddMetadataSnapshot(snap *MetadataSnapshot) (bool, error) {
+	tx, err := db.Beginx()
+	if err != nil {
+		return false, fmt.Errorf("recording metadata snapshot: %w", err)
+	}
+	defer func() { _ = tx.Rollback() }()
+	if db.dialect == DialectPostgres {
+		if _, err := tx.Exec("SELECT pg_advisory_xact_lock($1)", metadataSnapshotLockKey(snap.Ecosystem, snap.Name)); err != nil {
+			return false, fmt.Errorf("locking metadata snapshots: %w", err)
+		}
+	}
+	res, err := tx.Exec(db.Rebind(`
+		INSERT INTO metadata_snapshots (ecosystem, name, storage_path, content_type, content_encoding,
+		                                content_digest, size, valid_from_ms)
+		SELECT CAST(? AS TEXT), CAST(? AS TEXT), CAST(? AS TEXT), CAST(? AS TEXT), CAST(? AS TEXT),
+		       CAST(? AS TEXT), CAST(? AS BIGINT), CAST(? AS BIGINT)
+		WHERE COALESCE((
+			SELECT content_digest FROM metadata_snapshots
+			WHERE ecosystem = ? AND name = ? AND valid_from_ms <= CAST(? AS BIGINT)
+			ORDER BY valid_from_ms DESC, id DESC
+			LIMIT 1
+		), '') <> CAST(? AS TEXT)
+	`), snap.Ecosystem, snap.Name, snap.StoragePath, snap.ContentType, snap.ContentEncoding,
+		snap.ContentDigest, snap.Size, snap.ValidFromMS,
+		snap.Ecosystem, snap.Name, snap.ValidFromMS, snap.ContentDigest)
+	if err != nil {
+		return false, fmt.Errorf("recording metadata snapshot: %w", err)
+	}
+	n, err := res.RowsAffected()
+	if err != nil {
+		return false, err
+	}
+	if err := tx.Commit(); err != nil {
+		return false, fmt.Errorf("recording metadata snapshot: %w", err)
+	}
+	return n > 0, nil
+}
+
+// metadataSnapshotLockKey is the advisory lock AddMetadataSnapshot takes for
+// a document.
+func metadataSnapshotLockKey(ecosystem, name string) int64 {
+	return lockKey("metadata-snapshot:" + ecosystem + "\x00" + name)
+}
+
+// PruneMetadataSnapshots deletes the snapshots of ecosystem that stopped
+// being current before cutoff, that is those replaced by a newer snapshot
+// that became current before cutoff, and queues their blobs for deletion. The
+// newest snapshot of each document is always kept. It returns the number of
+// snapshots deleted.
+func (db *DB) PruneMetadataSnapshots(ecosystem string, cutoff time.Time) (int, error) {
+	tx, err := db.Beginx()
+	if err != nil {
+		return 0, err
+	}
+	defer func() { _ = tx.Rollback() }()
+
+	var expired []struct {
+		ID          int64  `db:"id"`
+		StoragePath string `db:"storage_path"`
+	}
+	err = tx.Select(&expired, db.Rebind(`
+		SELECT s.id, s.storage_path FROM metadata_snapshots s
+		WHERE s.ecosystem = ?
+		  AND EXISTS (
+			SELECT 1 FROM metadata_snapshots n
+			WHERE n.ecosystem = s.ecosystem AND n.name = s.name
+			  AND (n.valid_from_ms > s.valid_from_ms OR (n.valid_from_ms = s.valid_from_ms AND n.id > s.id))
+			  AND n.valid_from_ms < ?
+		  )
+	`), ecosystem, cutoff.UnixMilli())
+	if err != nil {
+		return 0, fmt.Errorf("listing expired metadata snapshots: %w", err)
+	}
+
+	queuedAt := time.Now().UTC()
+	for _, row := range expired {
+		if _, err := tx.Exec(db.Rebind(`DELETE FROM metadata_snapshots WHERE id = ?`), row.ID); err != nil {
+			return 0, fmt.Errorf("deleting metadata snapshot: %w", err)
+		}
+		if _, err := tx.Exec(db.Rebind(queuePendingDeleteQuery), row.StoragePath, queuedAt); err != nil {
+			return 0, fmt.Errorf("queueing metadata snapshot blob: %w", err)
+		}
+	}
+	if err := tx.Commit(); err != nil {
+		return 0, err
+	}
+	return len(expired), nil
+}