From 217c6cf33fa81d3672451fb81b7020e84d721ccf Mon Sep 17 00:00:00 2001 From: Charlie Le Date: Sat, 12 Sep 2026 12:28:12 -0700 Subject: [PATCH 1/5] Pull minio from quay.io instead of Docker Hub Every integration leg is failing at Preload Images: Error response from daemon: pull access denied for minio/minio, repository does not exist or may require 'docker login': denied: requested access to the resource is denied minio/minio is the first Docker Hub pull in the step, so no leg gets past it and all 24 fail in about 30 seconds. A CHANGELOG-only pull request reproduces it, so this is not specific to any change under test. master was last green at 4061a3dd6d. This is not a rate limit: the same pull fails right after a successful 'docker login' with the repository credentials. The docker.io/minio/minio repository is simply no longer accessible. MinIO still publishes the identical image to quay.io. quay.io/minio/minio :RELEASE.2024-05-28T17-19-04Z is public and is a manifest list with 8 children, so it covers both the amd64 and arm64 runners. Point the integration tests, the CI preload list, and the three development docker-compose stacks at quay.io. The tag is unchanged, so no behaviour changes. Signed-off-by: Charlie Le --- .github/workflows/test-build-deploy.yml | 2 +- development/tsdb-blocks-storage-s3-gossip/docker-compose.yml | 2 +- .../tsdb-blocks-storage-s3-single-binary/docker-compose.yml | 2 +- development/tsdb-blocks-storage-s3/docker-compose.yml | 2 +- integration/e2e/images/images.go | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/test-build-deploy.yml b/.github/workflows/test-build-deploy.yml index ee2696660ab..8cd17e5cb7e 100644 --- a/.github/workflows/test-build-deploy.yml +++ b/.github/workflows/test-build-deploy.yml @@ -317,7 +317,7 @@ jobs: done } - retry docker pull minio/minio:RELEASE.2024-05-28T17-19-04Z + retry docker pull quay.io/minio/minio:RELEASE.2024-05-28T17-19-04Z retry docker pull consul:1.8.4 retry docker pull quay.io/coreos/etcd:v3.5.29 if [ "$TEST_TAGS" = "integration_backward_compatibility" ]; then diff --git a/development/tsdb-blocks-storage-s3-gossip/docker-compose.yml b/development/tsdb-blocks-storage-s3-gossip/docker-compose.yml index 455a3e1f7ae..0bfbc0f6c1d 100644 --- a/development/tsdb-blocks-storage-s3-gossip/docker-compose.yml +++ b/development/tsdb-blocks-storage-s3-gossip/docker-compose.yml @@ -8,7 +8,7 @@ services: - 8500:8500 minio: - image: minio/minio + image: quay.io/minio/minio command: [ "server", "/data" ] environment: - MINIO_ACCESS_KEY=cortex diff --git a/development/tsdb-blocks-storage-s3-single-binary/docker-compose.yml b/development/tsdb-blocks-storage-s3-single-binary/docker-compose.yml index 4e6b396d194..ea247084cfa 100644 --- a/development/tsdb-blocks-storage-s3-single-binary/docker-compose.yml +++ b/development/tsdb-blocks-storage-s3-single-binary/docker-compose.yml @@ -8,7 +8,7 @@ services: - 8500:8500 minio: - image: minio/minio + image: quay.io/minio/minio command: [ "server", "/data" ] environment: - MINIO_ACCESS_KEY=cortex diff --git a/development/tsdb-blocks-storage-s3/docker-compose.yml b/development/tsdb-blocks-storage-s3/docker-compose.yml index 5bb3a9d3708..54d819d745d 100644 --- a/development/tsdb-blocks-storage-s3/docker-compose.yml +++ b/development/tsdb-blocks-storage-s3/docker-compose.yml @@ -8,7 +8,7 @@ services: - 8500:8500 minio: - image: minio/minio + image: quay.io/minio/minio command: [ "server", "/data" ] environment: - MINIO_ACCESS_KEY=cortex diff --git a/integration/e2e/images/images.go b/integration/e2e/images/images.go index aeb5858974e..c2714676b14 100644 --- a/integration/e2e/images/images.go +++ b/integration/e2e/images/images.go @@ -8,7 +8,7 @@ package images var ( Memcached = "memcached:1.6.1" Redis = "docker.io/redis:7.0.4-alpine" - Minio = "minio/minio:RELEASE.2024-05-28T17-19-04Z" + Minio = "quay.io/minio/minio:RELEASE.2024-05-28T17-19-04Z" Consul = "consul:1.8.4" ETCD = "quay.io/coreos/etcd:v3.5.29" Prometheus = "quay.io/prometheus/prometheus:v3.9.1" From b4d7c101d451dace53d49d3a1b468c7d331bba2f Mon Sep 17 00:00:00 2001 From: Charlie Le Date: Sat, 12 Sep 2026 12:28:30 -0700 Subject: [PATCH 2/5] Authenticate the integration job's remaining Docker Hub pulls Preload Images still pulls consul, memcached, redis and postgres from Docker Hub. #7464 removed the Install Docker Client step from this job, and that script is where 'docker login' runs, so those pulls have been anonymous since and are subject to the anonymous rate limit. Log in explicitly, matching what the build job does. Pull requests from forks have no secrets, so skip the login there and leave those pulls anonymous instead of failing the step. This is hardening, not the fix for the current breakage: minio failed even when authenticated. Signed-off-by: Charlie Le --- .github/workflows/test-build-deploy.yml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/.github/workflows/test-build-deploy.yml b/.github/workflows/test-build-deploy.yml index 8cd17e5cb7e..d8465618e55 100644 --- a/.github/workflows/test-build-deploy.yml +++ b/.github/workflows/test-build-deploy.yml @@ -294,6 +294,23 @@ jobs: name: integration-tests-${{ matrix.arch }} - name: Extract Integration Tests Archive run: tar -xzvf integration-tests-${{ matrix.arch }}.tar.gz + - name: Login to Docker Hub + # Preload Images still pulls consul, memcached, redis and postgres from Docker Hub. #7464 + # dropped the Install Docker Client step from this job, and with it the `docker login` that + # script performs, so those pulls have been anonymous ever since and are exposed to the + # anonymous rate limit. Authenticate here, as the build job already does. + # + # The secret is empty on pull requests from forks, so skip the login there and let the pulls + # stay anonymous rather than failing the step outright. + env: + DOCKER_REGISTRY_USER: ${{ secrets.DOCKER_REGISTRY_USER }} + DOCKER_REGISTRY_PASSWORD: ${{ secrets.DOCKER_REGISTRY_PASSWORD }} + run: | + if [ -z "${DOCKER_REGISTRY_PASSWORD:-}" ]; then + echo "No Docker Hub credentials available (fork pull request); pulling anonymously." + exit 0 + fi + docker login -u "$DOCKER_REGISTRY_USER" -p "$DOCKER_REGISTRY_PASSWORD" - name: Preload Images # We download docker images used by integration tests so that all images are available # locally and the download time doesn't account in the test execution time, which is subject From 66a0d295a89d5e3446546f8585bef823a1189a43 Mon Sep 17 00:00:00 2001 From: Charlie Le Date: Sat, 12 Sep 2026 13:55:56 -0700 Subject: [PATCH 3/5] Resolve the latest release image to a published version Backport of #7786 onto release-1.22. integration/util.go derived the query fuzz comparison image straight from VERSION. The moment VERSION becomes 1.22.0-rc.0 on this branch, integration_query_fuzz tries to pull quay.io/cortexproject/cortex:v1.22.0-rc.0, which does not exist: that image is pushed by the tag build's deploy job, and deploy is gated on integration passing first. Ask quay.io which GA tags are actually published instead, and take the highest one at or below VERSION. The CI step mirrors the same resolution and exports CORTEX_LATEST_RELEASE_IMAGE for the preload step. Set the CORTEX_LATEST_RELEASE_IMAGE repository variable to bypass the lookup. Signed-off-by: Charlie Le --- .github/workflows/test-build-deploy.yml | 77 ++++++++++++++++++- integration/util.go | 91 ++++++++++++++++++++++- integration/util_test.go | 98 +++++++++++++++++++++++++ 3 files changed, 261 insertions(+), 5 deletions(-) create mode 100644 integration/util_test.go diff --git a/.github/workflows/test-build-deploy.yml b/.github/workflows/test-build-deploy.yml index d8465618e55..a0a97e3a16e 100644 --- a/.github/workflows/test-build-deploy.yml +++ b/.github/workflows/test-build-deploy.yml @@ -311,6 +311,81 @@ jobs: exit 0 fi docker login -u "$DOCKER_REGISTRY_USER" -p "$DOCKER_REGISTRY_PASSWORD" + - name: Resolve Latest Release Image + # The query fuzz tests compare the build under test against the latest *published* release. + # VERSION cannot answer "what is published" on its own: on a release branch it is bumped to + # the version being prepared (e.g. 1.22.0-rc.0) long before anything pushes that tag, and + # even on the GA tag push the v1.22.0 image is only pushed by `deploy`, which needs this job + # to pass first. The registry is the only source of truth, so ask it which GA tags exist and + # take the highest one that does not exceed VERSION. + # + # The <= bound (rather than simply "the highest published GA tag") only changes the result + # when a newer release already exists on quay than the branch being tested, e.g. preparing + # 1.21.2 on release-1.21 after v1.22.0 has shipped. + # + # Set the CORTEX_LATEST_RELEASE_IMAGE repository variable to bypass the lookup entirely. + if: matrix.tags == 'integration_query_fuzz' + env: + CORTEX_LATEST_RELEASE_IMAGE: ${{ vars.CORTEX_LATEST_RELEASE_IMAGE }} + run: | + if [ -n "${CORTEX_LATEST_RELEASE_IMAGE:-}" ]; then + echo "Using the CORTEX_LATEST_RELEASE_IMAGE override: ${CORTEX_LATEST_RELEASE_IMAGE}" + echo "CORTEX_LATEST_RELEASE_IMAGE=${CORTEX_LATEST_RELEASE_IMAGE}" >> "$GITHUB_ENV" + exit 0 + fi + + version=$(tr -d '[:space:]' < testdata/VERSION) + base=${version%%-*} + if ! printf '%s' "$base" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+$'; then + echo "ERROR: VERSION '${version}' does not begin with a major.minor.patch version." >&2 + exit 1 + fi + + # List the GA tags published to quay.io. filter_tag_name keeps the release tags and drops + # the per-commit master-* ones; the API pages at 100 tags, so follow has_additional. + tags_file=$(mktemp) + page=1 + while [ "$page" -le 20 ]; do + body="" + for attempt in 1 2 3; do + if body=$(curl -sSf --max-time 30 \ + "https://quay.io/api/v1/repository/cortexproject/cortex/tag/?onlyActiveTags=true&limit=100&page=${page}&filter_tag_name=like:v"); then + break + fi + echo "WARNING: listing quay.io tags page ${page} failed (attempt ${attempt}/3); retrying..." >&2 + body="" + sleep $((attempt * 5)) + done + if [ -z "$body" ]; then + echo "ERROR: unable to list the published tags from quay.io." >&2 + exit 1 + fi + printf '%s' "$body" | jq -r '.tags[].name' >> "$tags_file" + [ "$(printf '%s' "$body" | jq -r '.has_additional')" = "true" ] || break + page=$((page + 1)) + done + + published=$(grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' "$tags_file" | sed 's/^v//' | sort -u -V) + if [ -z "$published" ]; then + echo "ERROR: quay.io reported no published GA release tags." >&2 + exit 1 + fi + + if printf '%s\n' "$published" | grep -qxF "$base"; then + # VERSION itself names a published release, which is the steady state on master. + resolved="$base" + else + # Splice the (unpublished) base into the sorted list and take the entry just below it. + resolved=$(printf '%s\n%s\n' "$published" "$base" | sort -V | + awk -v base="$base" '$0 == base { exit } { previous = $0 } END { print previous }') + fi + if [ -z "$resolved" ]; then + echo "ERROR: quay.io has no published GA release at or below ${base}." >&2 + exit 1 + fi + + echo "VERSION is ${version}; the latest release published at or below ${base} is v${resolved}." + echo "CORTEX_LATEST_RELEASE_IMAGE=quay.io/cortexproject/cortex:v${resolved}" >> "$GITHUB_ENV" - name: Preload Images # We download docker images used by integration tests so that all images are available # locally and the download time doesn't account in the test execution time, which is subject @@ -346,7 +421,7 @@ jobs: retry docker pull quay.io/cortexproject/cortex:v1.21.0 retry docker pull quay.io/cortexproject/cortex:v1.21.1 elif [ "$TEST_TAGS" = "integration_query_fuzz" ]; then - retry docker pull quay.io/cortexproject/cortex:v$(cat testdata/VERSION) + retry docker pull "$CORTEX_LATEST_RELEASE_IMAGE" retry docker pull quay.io/prometheus/prometheus:v3.9.1 elif [ "$TEST_TAGS" = "integration_configs_db" ]; then retry docker pull postgres:9.6.16 diff --git a/integration/util.go b/integration/util.go index 0ec7721838c..89e4e248f10 100644 --- a/integration/util.go +++ b/integration/util.go @@ -8,6 +8,7 @@ import ( "os" "os/exec" "path/filepath" + "strconv" "strings" "github.com/pkg/errors" @@ -36,20 +37,102 @@ func getCortexProjectDir() string { return os.Getenv("GOPATH") + "/src/github.com/cortexproject/cortex" } -// getLatestReleaseImage returns the Cortex image reference for the latest release, -// derived from the VERSION file at the project root. +// getLatestReleaseImage returns the Cortex image reference for the latest published +// release. +// +// CORTEX_LATEST_RELEASE_IMAGE short-circuits the resolution. CI always sets it: the +// integration workflow asks quay.io which GA tags actually exist and picks the highest one +// that does not exceed VERSION, because the registry is the only source of truth for what +// is published (see .github/workflows/test-build-deploy.yml). +// +// Without it — a local run — fall back to deriving the version from the VERSION file at the +// project root, which needs no network but cannot see what the registry holds. func getLatestReleaseImage() (string, error) { + if image := os.Getenv("CORTEX_LATEST_RELEASE_IMAGE"); image != "" { + return image, nil + } + content, err := os.ReadFile(filepath.Join(getCortexProjectDir(), "VERSION")) if err != nil { return "", errors.Wrap(err, "unable to read VERSION file") } - version := strings.TrimSpace(string(content)) + version, err := latestReleaseVersion(strings.TrimSpace(string(content))) + if err != nil { + return "", err + } + + return fmt.Sprintf("quay.io/cortexproject/cortex:v%s", version), nil +} + +// latestReleaseVersion maps the contents of the VERSION file to a version that has very +// likely been published to the container registries. It is the offline fallback for +// getLatestReleaseImage; CI resolves against the registry instead. +// +// VERSION does not always name a published release. On a release branch it is bumped to +// the version being prepared (e.g. "1.22.0-rc.0") long before the deploy job publishes +// that tag, and the integration job runs before deploy. So a pre-release version resolves +// to the release preceding it, which is always already published by then: +// +// 1.21.1 -> 1.21.1 (VERSION on master is the last GA, whose image exists) +// 1.22.0-rc.0 -> 1.21.0 (the previous minor always shipped a .0) +// 1.22.2-rc.1 -> 1.22.1 (the preceding patch of the same minor) +// +// A GA VERSION is assumed published, which holds everywhere except the GA tag build itself +// — there v1.22.0 is only pushed by deploy, after this runs. That case is why CI consults +// the registry rather than relying on this. +func latestReleaseVersion(version string) (string, error) { if version == "" { return "", errors.New("VERSION file is empty") } - return fmt.Sprintf("quay.io/cortexproject/cortex:v%s", version), nil + // Anything after the first "-" is a pre-release identifier (e.g. "-rc.0"). + base, preRelease, isPreRelease := strings.Cut(version, "-") + if !isPreRelease { + return version, nil + } + + major, minor, patch, err := parseVersion(base) + if err != nil { + return "", errors.Wrapf(err, "unable to resolve the release preceding pre-release version %q", version) + } + + switch { + case patch > 0: + // A patch pre-release: the preceding patch of the same minor is published. + patch-- + case minor > 0: + // A minor pre-release: the previous minor's initial release is published. Using + // .0 rather than its latest patch keeps this derivable from VERSION alone. + minor-- + patch = 0 + default: + // A major pre-release (e.g. "2.0.0-rc.0"). The last release of the previous major + // is not derivable from VERSION, so the maintainer has to say which one it is. + return "", errors.Errorf("cannot resolve the release preceding major pre-release version %q (base %q, pre-release %q):"+ + " set CORTEX_LATEST_RELEASE_IMAGE to the latest published release image", version, base, preRelease) + } + + return fmt.Sprintf("%d.%d.%d", major, minor, patch), nil +} + +func parseVersion(version string) (major, minor, patch int, err error) { + parts := strings.Split(version, ".") + if len(parts) != 3 { + return 0, 0, 0, errors.Errorf("expected a major.minor.patch version, got %q", version) + } + + out := make([]int, len(parts)) + for i, part := range parts { + if out[i], err = strconv.Atoi(part); err != nil { + return 0, 0, 0, errors.Wrapf(err, "invalid version %q", version) + } + if out[i] < 0 { + return 0, 0, 0, errors.Errorf("invalid version %q", version) + } + } + + return out[0], out[1], out[2], nil } func writeFileToSharedDir(s *e2e.Scenario, dst string, content []byte) error { diff --git a/integration/util_test.go b/integration/util_test.go new file mode 100644 index 00000000000..7cefd46a1da --- /dev/null +++ b/integration/util_test.go @@ -0,0 +1,98 @@ +//go:build integration_query_fuzz + +package integration + +import ( + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestLatestReleaseVersion(t *testing.T) { + tests := map[string]struct { + version string + expected string + expectedErr bool + }{ + "a GA version is already published": { + version: "1.21.1", + expected: "1.21.1", + }, + "a GA version with a zero patch is already published": { + version: "1.21.0", + expected: "1.21.0", + }, + "a minor release candidate falls back to the previous minor": { + version: "1.22.0-rc.0", + expected: "1.21.0", + }, + "a later minor release candidate falls back to the same previous minor": { + version: "1.22.0-rc.3", + expected: "1.21.0", + }, + "a patch release candidate falls back to the preceding patch": { + version: "1.22.1-rc.0", + expected: "1.22.0", + }, + "a later patch release candidate falls back to the preceding patch": { + version: "1.22.3-rc.1", + expected: "1.22.2", + }, + "a major release candidate cannot be resolved": { + version: "2.0.0-rc.0", + expectedErr: true, + }, + "an empty VERSION is rejected": { + version: "", + expectedErr: true, + }, + "a malformed pre-release base is rejected": { + version: "1.22-rc.0", + expectedErr: true, + }, + "a non-numeric pre-release base is rejected": { + version: "1.x.0-rc.0", + expectedErr: true, + }, + } + + for name, testData := range tests { + t.Run(name, func(t *testing.T) { + actual, err := latestReleaseVersion(testData.version) + if testData.expectedErr { + require.Error(t, err) + return + } + + require.NoError(t, err) + assert.Equal(t, testData.expected, actual) + }) + } +} + +func TestGetLatestReleaseImage(t *testing.T) { + // CI exports CORTEX_LATEST_RELEASE_IMAGE for this build tag, which would short-circuit the + // resolution we are trying to exercise. Clear it so the fallback path is what runs. + t.Setenv("CORTEX_LATEST_RELEASE_IMAGE", "") + + // Point getCortexProjectDir() at a scratch checkout so we can exercise the VERSION file + // contents a release branch would actually have. + dir := t.TempDir() + t.Setenv("CORTEX_CHECKOUT_DIR", dir) + require.NoError(t, os.WriteFile(filepath.Join(dir, "VERSION"), []byte("1.22.0-rc.0\n"), 0o600)) + + image, err := getLatestReleaseImage() + require.NoError(t, err) + assert.Equal(t, "quay.io/cortexproject/cortex:v1.21.0", image) +} + +func TestGetLatestReleaseImage_HonorsOverride(t *testing.T) { + t.Setenv("CORTEX_LATEST_RELEASE_IMAGE", "quay.io/cortexproject/cortex:v1.20.1") + + image, err := getLatestReleaseImage() + require.NoError(t, err) + assert.Equal(t, "quay.io/cortexproject/cortex:v1.20.1", image) +} From 343bf3ff88c7d4ef36d4b69b7c4006474d0e1523 Mon Sep 17 00:00:00 2001 From: Charlie Le Date: Sat, 12 Sep 2026 12:03:52 -0700 Subject: [PATCH 4/5] Mark release 1.22.0 in progress Add a '## 1.22.0 in progress' section below an empty '## master / unreleased', move the existing unreleased entries into it, and order them [CHANGE] -> [FEATURE] -> [ENHANCEMENT] -> [BUGFIX] per RELEASE.md. Also fill the gaps reported by ./tools/release/check-changelog.sh v1.21.1...master: - new entries for #7513, #7514 and #7559 - fold #7323, #7434, #7458, #7463, #7487, #7505, #7687, #7691, #7716, #7726, #7775 and #7807 into the entries they belong to Signed-off-by: Charlie Le --- CHANGELOG.md | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a2032c41cbd..a46face8419 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,8 +1,8 @@ # Changelog ## master / unreleased -* [ENHANCEMENT] Query Frontend: Log `X-Grafana-User` header in query stats, slow query, and query request logs when Grafana's `send_user_header` is enabled. #7799 -* [FEATURE] Engine: Add `-querier.selector-batch-size` and `-ruler.selector-batch-size` flags to configure series batching in the Thanos promQL engine. 0 disables batching. #7763 + +## 1.22.0 in progress * [CHANGE] Ruler: Remove the deprecated `-ruler.evaluation-delay-duration` flag and its `ruler_evaluation_delay_duration` per-tenant limit. Use `-ruler.query-offset` / `ruler_query_offset`, which no longer takes the higher of the two values. Cortex decodes the runtime config strictly, so a leftover `ruler_evaluation_delay_duration` override makes the runtime config fail to load: Cortex **exits at startup** (`module failed`, `module=runtime-config`), and on an already-running process every reload fails, pinning the last good overrides and dropping `cortex_runtime_config_last_reload_successful` to 0. Run `grep -r ruler_evaluation_delay_duration` over your runtime configs before upgrading. #7792 * [CHANGE] Remove the deprecated `-.fifocache.size` flag and its `size` YAML field (deprecated in 1.1.0). Use `-.fifocache.max-size-items` or `-.fifocache.max-size-bytes`; a cache configured only via `size` now starts with no capacity. #7791 * [CHANGE] Querier: Remove the deprecated `-querier.ingester-metadata-streaming` flag and its `ingester_metadata_streaming` YAML field (deprecated in 1.18.0, default `true`). Streaming RPCs are now always used for the metadata APIs. Also removes the dead hidden `ingester_streaming` YAML field left over from `-querier.ingester-streaming`. #7791 @@ -17,7 +17,8 @@ - `-ingester.max-series-per-query` (a chunks-storage limit, ignored since blocks storage; use `-querier.max-fetched-series-per-query`) * [CHANGE] Ingester: Formally deprecate `-blocks-storage.tsdb.max-exemplars`, scheduled for removal in v1.24.0. Use the per-tenant `max_exemplars` limit instead. The flag still works as the global fallback when `max_exemplars` is 0, but setting it now logs a warning and increments `deprecated_flags_inuse_total`. #7793 * [CHANGE] Ingester: Graduate native histogram ingestion (`-blocks-storage.tsdb.enable-native-histograms`) from experimental. #7789 -* [CHANGE] Querier: Make query time range configurations per-tenant: `query_ingesters_within`, `query_store_after`, and `shuffle_sharding_ingesters_lookback_period`. Uses `model.Duration` instead of `time.Duration` to support serialization but has minimum unit of 1ms (nanoseconds/microseconds not supported). #7160 +* [CHANGE] Querier: Make query time range configurations per-tenant: `query_ingesters_within`, `query_store_after`, and `shuffle_sharding_ingesters_lookback_period`. Uses `model.Duration` instead of `time.Duration` to support serialization but has minimum unit of 1ms (nanoseconds/microseconds not supported). #7160 #7323 +* [CHANGE] Alertmanager: Remove the obsolete startup migration of local state files into per-tenant directories (scheduled for removal in 1.11.0). Upgrading from a release older than 1.9.0 with a persisted local state directory now requires upgrading to an intermediate release first, so the migration can run. #7513 * [CHANGE] Cache: Setting `-blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl` to 0 will disable the bucket-index cache. #7446 * [CHANGE] HA Tracker: Move `-distributor.ha-tracker.failover-timeout` from a global config to a per-tenant runtime config. The flag name and default value (30s) remain the same. #7481 * [FEATURE] Parquet: Support sharded parquet file conversion and querying. #7610 @@ -25,7 +26,7 @@ * [FEATURE] Distributor: Add experimental `-distributor.num-query-workers` flag to use a goroutine worker pool for query fan-out calls to ingesters. Reuses pre-grown goroutine stacks to eliminate the `runtime.copystack` overhead (~8% CPU) observed on rulers with wide ingester fan-out. Falls back to spawning a new goroutine when no worker is available. #7623 * [FEATURE] Ingester: Add experimental active series tracker that counts active series by configurable label matchers (including regex) per tenant and exposes `cortex_ingester_active_series_per_tracker` metric. Configured via `active_series_trackers` in runtime config overrides. #7476 * [FEATURE] Ingester: Add experimental head-only queried series metric. `cortex_ingester_queried_head_series` tracks unique series queried from head via HLL. Enabled via `-ingester.head-queried-series-metrics-enabled`. #7500 -* [FEATURE] Ruler: Add per-tenant `ruler_alert_generator_url_template` runtime config option to customize alert generator URLs using Go templates. Includes a `jsonEscape` template function for safely embedding expressions in JSON-encoded URL parameters (e.g., Grafana Explore panes). Supports Grafana Explore, Perses, and other UIs. #7302 +* [FEATURE] Ruler: Add per-tenant `ruler_alert_generator_url_template` runtime config option to customize alert generator URLs using Go templates. Includes a `jsonEscape` template function for safely embedding expressions in JSON-encoded URL parameters (e.g., Grafana Explore panes). Supports Grafana Explore, Perses, and other UIs. #7302 #7458 * [FEATURE] Distributor: Add experimental `-distributor.enable-start-timestamp` flag for Prometheus Remote Write 2.0. When enabled, `StartTimestamp (ST)` is ingested. #7371 * [FEATURE] Memberlist: Add `-memberlist.cluster-label` and `-memberlist.cluster-label-verification-disabled` to prevent accidental cross-cluster gossip joins and support rolling label rollout. #7385 * [FEATURE] Querier: Add timeout classification to classify query timeouts as 4XX (user error) or 5XX (system error) based on phase timing. When enabled, queries that spend most of their time in PromQL evaluation return `422 Unprocessable Entity` instead of `503 Service Unavailable`. #7374 @@ -33,6 +34,7 @@ * [FEATURE] Querier: Add resource-based query eviction that automatically cancels the heaviest running query when CPU or heap utilization exceeds configured thresholds. #7488 * [FEATURE] Storage: Add support for Oracle Cloud Infrastructure (OCI) Object Storage as a backend for blocks, ruler, and alertmanager storage. Configured via `-.oci.*` flags with `backend: oci`. #7718 * [FEATURE] StoreGateway: Add experimental optional limit `blocks-storage.bucket-store.max-concurrent-data-bytes` on the data bytes (postings, series and chunks) fetched via the Series() API call and processed concurrently across all queries per store gateway to protect from oomkill. This returns an error that is retryable at querier level. #7271 +* [FEATURE] Engine: Add `-querier.selector-batch-size` and `-ruler.selector-batch-size` flags to configure series batching in the Thanos promQL engine. 0 disables batching. #7763 * [ENHANCEMENT] Upgrade prometheus alertmanager version to v0.32.1. #7462 * [ENHANCEMENT] Tenant Federation: Avoid purging the regex resolver LRU cache on user-sync ticks when the set of known users has not changed. #7489 * [ENHANCEMENT] Parquet Converter: Add `parquet-converter.max-block-label-names` limit to skip conversion of TSDB blocks with too many label names. #7625 @@ -43,13 +45,13 @@ * [ENHANCEMENT] Ingester: Add WAL record metrics to help evaluate the effectiveness of WAL compression type (e.g. snappy, zstd): `cortex_ingester_tsdb_wal_record_part_writes_total`, `cortex_ingester_tsdb_wal_record_parts_bytes_written_total`, and `cortex_ingester_tsdb_wal_record_bytes_saved_total`. #7420 * [ENHANCEMENT] Distributor: Introduce dynamic `Symbols` slice capacity pooling. #7398 #7401 * [ENHANCEMENT] Metrics Helper: Add native histogram support for aggregating and merging, including dual-format histogram handling that exposes both native and classic bucket formats. #7359 -* [ENHANCEMENT] Cache: Add per-tenant TTL configuration for query results cache to control cache expiration on a per-tenant basis with separate TTLs for regular and out-of-order data. #7357 -* [ENHANCEMENT] Update build image and Go version to 1.26. #7437 +* [ENHANCEMENT] Cache: Add per-tenant TTL configuration for query results cache to control cache expiration on a per-tenant basis with separate TTLs for regular and out-of-order data. `-frontend.out-of-order-results-cache-ttl` falls back to `-frontend.results-cache-ttl` when unset, and then to the global cache backend TTL. #7357 #7775 +* [ENHANCEMENT] Update build image and Go version to 1.26. #7434 #7437 #7716 #7726 * [ENHANCEMENT] Upgraded container base images from `alpine:3.23` to `gcr.io/distroless/static-debian12`, reducing image size and attack surface. #7637 * [ENHANCEMENT] Query Scheduler: Add `cortex_query_scheduler_tracked_requests` metric to track the current number of requests held by the scheduler. #7355 * [ENHANCEMENT] Compactor: Prevent partition compaction to compact any blocks marked for deletion. #7391 * [ENHANCEMENT] Distributor: Optimize memory allocations by reusing the existing capacity of these pooled slices in the Prometheus Remote Write 2.0 path. #7392 -* [ENHANCEMENT] Upgrade gRPC from v1.71.2 to v1.79.3 to address CVE-2026-33186. #7460 +* [ENHANCEMENT] Upgrade gRPC from v1.71.2 to v1.79.3 to address CVE-2026-33186. #7460 #7463 * [ENHANCEMENT] Query Frontend: Add `query_too_expensive` reason to QFE and `reason` field to query stats. #7479 * [ENHANCEMENT] Instrument Ingester CPU profile with source for read APIs. #7494 * [ENHANCEMENT] Ingester: Convert expanded postings cache from FIFO to LRU eviction to retain frequently-queried entries under memory pressure. #7510 @@ -68,18 +70,20 @@ * [ENHANCEMENT] Ingester: Add `cortex_ingester_tsdb_head_max_timestamp` metric that re-exports the TSDB head max timestamp (`prometheus_tsdb_head_max_time`) per user, to help investigate ingestion issues like out-of-bounds (too old sample) errors. #7694 * [ENHANCEMENT] Ingester: Include the TSDB head max time in the `out of bounds` and `too old sample` error messages, so that users can see how far behind the accepted time range a rejected sample is. #7695 * [ENHANCEMENT] Compactor: Reduce object storage GET calls when updating the bucket index by skipping re-reading parquet converter markers for blocks that already have a valid-version parquet entry in the previous index. #7669 -* [ENHANCEMENT] Upgrade Thanos and promql-engine to latest. #7740 #7788 +* [ENHANCEMENT] Upgrade Thanos and promql-engine to latest. #7505 #7691 #7740 #7788 * [ENHANCEMENT] Ruler: Adjust ruler frontend decoder to not wrap query error messages with execution prefix, this makes error responses consistent between internal and external ruler paths. #7741 * [ENHANCEMENT] Distributor: Deduplicate metric metadata when converting PRW 2.0 requests. PRW 2.0 attaches metadata to every series, so a metric family was previously expanded into one `MetricMetadata` per series. #7760 +* [ENHANCEMENT] Ingester: Add `cortex_ingester_head_metric_names` gauge exposing the number of unique metric names in the TSDB head per tenant. Registered when `-ingester.active-series-metrics-enabled` is true. #7514 +* [ENHANCEMENT] Query Frontend: Log `X-Grafana-User` header in query stats, slow query, and query request logs when Grafana's `send_user_header` is enabled. #7799 * [ENHANCEMENT] Querier: Use non-pointer HistogramBucket slice in response codec. #7809 -* [ENHANCEMENT] Update build image and Go version to 1.27.0. #7814 +* [ENHANCEMENT] Update build image and Go version to 1.27.0. #7807 #7814 * [ENHANCEMENT] Querier: Reduce merge iterator `BatchSize` from 12 to 8. #7823 * [BUGFIX] Querier: Fix queryWithRetry and labelsWithRetry returning (nil, nil) on cancelled context by propagating ctx.Err(). #7370 * [BUGFIX] Metrics Helper: Fix non-deterministic bucket order in merged histograms by sorting buckets after map iteration, matching Prometheus client library behavior. #7380 * [BUGFIX] Distributor: Return HTTP 401 Unauthorized when tenant ID resolution fails in the Prometheus Remote Write 2.0 path. #7389 * [BUGFIX] Packaging: Fix RPM and deb packages to install the binary to `/usr/bin`, install the systemd unit to the correct system path (`/usr/lib/systemd/system` for RPM, `/lib/systemd/system` for deb), and mark the sysconfig/default env file as a config file so it is not overwritten on upgrade. #7445 * [BUGFIX] Compactor: Handle not-found and access-denied errors from `Attributes()` in bucket index updater, preventing a stale cached `Get()` from causing the entire cleanup cycle to fail when `meta.json` has been deleted from object storage. #7454 -* [BUGFIX] Compactor: Fix stale `cortex_bucket_index_last_successful_update_timestamp_seconds` metric not being cleaned up when tenant ownership changes due to ring rebalancing. This caused false alarms on bucket index update rate when a tenant moved between compactors. #7485 +* [BUGFIX] Compactor: Fix stale `cortex_bucket_index_last_successful_update_timestamp_seconds` metric not being cleaned up when tenant ownership changes due to ring rebalancing. This caused false alarms on bucket index update rate when a tenant moved between compactors. #7485 #7487 * [BUGFIX] Compactor: Fix flake in `TestCompactor_DeleteLocalSyncFiles` and `TestPartitionCompactor_DeleteLocalSyncFiles` by polling on user ownership rather than just the `CompactionRunsCompleted` counter, which increments even when the second compactor sees zero owned users due to a transient ring-view skew at startup. #7565 * [BUGFIX] Ingester: Close TSDB when compaction fails during `createTSDB`, preventing resource leaks (file descriptors, mmap handles) that could lead to ingester instability. #7560 * [BUGFIX] Tenant Federation: Fix result cache returning stale data after a new tenant is added when `-tenant-federation.regex-matcher-enabled=true`. The resolved tenant set is now hashed and included in the cache key so that any change to the matched tenant list automatically invalidates cached entries. Non-regex users are unaffected. #7562 @@ -88,6 +92,7 @@ * [BUGFIX] Ingester: Release the TSDB appender on every early-return path in `Push` (e.g. out-of-order label set) by deferring `Rollback`. Previously such requests leaked TSDB head series references, mmap'd chunks and pending state per request, causing the `cortex_ingester_tsdb_head_active_appenders` gauge to grow unbounded. #7528 * [BUGFIX] Ingester: Fix `panic: send on closed channel` in `ActiveQueriedSeriesService` on shutdown by removing the redundant channel close in `stopping()` and relying on `ctx.Done()` to signal worker exit. #7533 * [BUGFIX] Ring: Fix ring token conflict resolution only applied to updated instance and make constantly token conflict check during instance observe period. #7554 +* [BUGFIX] Ring: Fix `DoBatch` never running its cleanup callback when a per-instance callback panics. `wg.Done()` is now deferred, so `wg.Wait()` no longer blocks forever and the context timers and request buffers owned by the cleanup function are released. #7559 * [BUGFIX] Query Frontend: Fix native histogram responses not being handled correctly in `minTime()` sort ordering for split_by_interval merge. #7555 * [BUGFIX] Compactor: Ensure visit marker heartbeat goroutine completes before blocks cleaner returns. #7386 * [BUGFIX] Querier: Fix unbounded resource leak in the bucket-scan blocks finder (used when the bucket index is disabled). Per-tenant metadata fetchers, their Prometheus registries, and on-disk meta caches are now evicted once a tenant is no longer active, instead of being retained for the lifetime of the process. #7573 @@ -101,7 +106,7 @@ * [BUGFIX] Querier: Fix panic due to request tracker truncating multi-byte UTF-8 character #7640 * [BUGFIX] Ingester: Fix panic (`HistogramProtoToHistogram called with a float histogram`) when ingesting a float native histogram with a zero count (e.g. a staleness marker or empty histogram). The decoder is now selected by histogram type via `IsFloatHistogram()` instead of by count value. #7645 * [BUGFIX] Querier: Fix parquet queryable fallback returning a nil error instead of the actual query error in `LabelValues` and `LabelNames`. #7638 -* [BUGFIX] Storage: Default the Azure `endpoint_suffix` to `blob.core.windows.net` instead of empty. Cortex builds the Thanos Azure config directly and bypasses Thanos' default, so an unset suffix produced an invalid FQDN (`.`) and components hung on startup with DNS errors. #5449 +* [BUGFIX] Storage: Default the Azure `endpoint_suffix` to `blob.core.windows.net` instead of empty. Cortex builds the Thanos Azure config directly and bypasses Thanos' default, so an unset suffix produced an invalid FQDN (`.`) and components hung on startup with DNS errors. #5449 #7687 * [BUGFIX] Store Gateway: Fix misleading "no index cache backend addresses" validation error being reported for chunks-cache, metadata-cache, and parquet caches when their memcached or redis backend is configured without addresses. The message is now the cache-type-agnostic "no cache backend addresses". #7675 * [BUGFIX] Querier/Query Frontend: Fix DNS watcher dropping all query-frontend/scheduler worker connections on a transient DNS lookup failure. #7698 * [BUGFIX] Ring: Fix DynamoDB KV CAS not retrying on transactional conditional check failures. `TransactWriteItems` reports condition failures as `TransactionCanceledException` with a `ConditionalCheckFailed` cancellation reason, which was not recognized as retryable, so any concurrent ring update conflict (e.g. many ingesters joining during a rolling update) failed immediately instead of re-reading and retrying. `TransactionConflict` cancellation reasons are also treated as retryable. #7706 From 371653326124d4becc193f2c7eaed9455b06ab35 Mon Sep 17 00:00:00 2001 From: Charlie Le Date: Sat, 12 Sep 2026 13:56:21 -0700 Subject: [PATCH 5/5] Update version to 1.22.0-rc.0 Signed-off-by: Charlie Le --- VERSION | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/VERSION b/VERSION index 2844977405c..6668549e16a 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.21.1 +1.22.0-rc.0