Skip to content

antalya-26.6: test_auth_token_profile_events fails in all integration builds (broken by #2222) #2323

Description

@CarlosFelipeOR

Describe the situation

test_database_iceberg_lakekeeper_catalog/test.py::test_auth_token_profile_events fails on every integration build of antalya-26.6. It has never passed on the branch: it was introduced already broken by PR #2222 and failed on the first MasterCI run that contained it.

The failure is deterministic — assert 0 >= 1 — and reproduces in all four integration build families, including on retry.

This issue:


How to reproduce the behavior

Environment

  • Version: 26.6.2.20001.altinityantalya
  • Branch: antalya-26.6 @ b3c747bb0358bd959c0f7b53431e2238e2f55f26
  • Build type: reproduces on asan+ubsan, msan, tsan and plain arm binary

Steps

  1. Check out the branch and run the test:
python3 -m ci.praktika run "Integration tests (amd_tsan, 1/6)" \
  --test test_database_iceberg_lakekeeper_catalog/test.py::test_auth_token_profile_events

Reproduces on the first attempt, on any of the integration build configurations.


Expected behavior

The test asserts that the DataLake REST catalog fetches an OAuth token once and then reuses it from cache, by reading the corresponding ProfileEvents from system.query_log.


Actual behavior

test_database_iceberg_lakekeeper_catalog/test.py:470: in test_auth_token_profile_events
    assert refreshed >= 1
E   assert 0 >= 1

Root cause analysis

Produced by an AI assistant from the source tree and CI logs — please double-check the reasoning.

Defect 1 — the test reads ProfileEvents that no longer exist

The last commit of PR #2222, da005ba98b7 "Rename several events" (which is also the PR's head commit), renamed the events in src/Common/ProfileEvents.cpp and src/Databases/DataLake/RestCatalog.cpp but did not update the test that the same PR had just added:

Name queried by the test Actual name in the binary
DataLakeRestCatalogAuthTokenRefreshed DataLakeRestCatalogAuthTokenRetrieve
DataLakeRestCatalogAuthTokenCacheHits DataLakeRestCatalogAuthTokenCachedValid
DataLakeRestCatalogAuthTokenRefreshedOnUnauthorized DataLakeRestCatalogUnauthorized

tests/integration/test_database_iceberg_lakekeeper_catalog/test.py:421 and :425 still query the old names. A Map lookup on a missing key returns 0, so assert refreshed >= 1 fails 100% of the time by construction.

Defect 2 — the test measures the wrong query

The test carries this comment:

The catalog client is initialized lazily on the first database access, not during CREATE DATABASE.

That is not true, and never has been. registerDatabaseDataLake passes lazy_init = args.create_query.attach || args.internal (src/Databases/DataLake/DatabaseDataLake.cpp:1214), so a plain CREATE DATABASE builds the catalog eagerly in the constructor. RestCatalog's constructor then calls loadConfig(), which issues GET /v1/config and therefore fetches the OAuth token — during the CREATE DATABASE query, before any SHOW TABLES runs.

Confirmed in the server log of the failing CI job — the config request is attributed to the CREATE DATABASE query id, and the SHOW TABLES the test measures makes no such request:

00:58:06.409 {f494286a-…} executeQuery: CREATE DATABASE test_auth_token_profile_events_…
00:58:06.410 {f494286a-…} RestCatalog(demo): Requesting: …/v1/config?warehouse=demo   ← token fetched here
00:58:07.653 {…-show-1-87a0964b-…} executeQuery: SHOW TABLES FROM …                   ← only namespace/table listing

Consequently the event rename alone does not account for the failure. This was checked in an actual local run: with only the queried names swapped for the ones the binary defines, the test still fails with the same assert 0 >= 1.

Eager construction on CREATE predates the test. Before ClickHouse#108674 the constructor called initialize() unconditionally; ClickHouse#108674 made only ATTACH lazy; ClickHouse#109724 added || args.internal. CREATE has been eager throughout, and RestCatalog's constructor has called loadConfig() since 2025-02.


Additional context

CI failure

Failing jobs in that run — all four, all retry_failed:

Job Result
Integration tests (amd_asan_ubsan, db disk, old analyzer, 7/8) FAIL
Integration tests (amd_msan, 7/10) FAIL
Integration tests (amd_tsan, 1/6) FAIL
Integration tests (arm_binary, distributed plan, 3/4) FAIL

Related PR

Why PR #2222's CI did not catch it

Every integration job on PR #2222 is marked SKIPPED. The PR description checks ci_exclude_asan, ci_exclude_ubsan, ci_exclude_tsan, ci_exclude_msan and ci_exclude_aarch64, and no other integration job ran either. The test was therefore never executed before merge.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    antalyacicdImprovements and fixes to the CICD process

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions