CAS: concurrent ATTACH PARTITION / INSERT loses the 90s ref_catalog/_ckpt policy (Code 210); common-pool num_tries then 300s-backs off REPLACE_RANGE

[alsugiliazova]

✅ I checked the Altinity Stable Builds lifecycle table, and the Altinity Stable Build version I'm using is still supported.

Type of problem

Bug report - something's broken

Describe the situation

Under the concurrent /alter/attach partition combinatorics, CAS cannot finish ordinary writes. A 10-row INSERT and ALTER TABLE … ATTACH PARTITION … FROM fail with Code: 210 NETWORK_ERROR after the CAS write policy gives up on the pool-global cas/ref_catalog object or a per-namespace _ckpt. That is a user-visible failure on MergeTree as well as ReplicatedMergeTree — no replica handshake is required.

On replicated destinations the same load also ages REPLACE_RANGE queue entries that never ran: selectQueueEntry increments num_tries before trySchedule, a saturated background_common_pool (default 8, max_tasks == max_threads, no wait queue) refuses without rolling the counter back, and the first real CAS error then hits the 300 s max_postpone_time_for_failed_replicated_tasks_ms cap. Replicas miss rows; SYSTEM SYNC REPLICA loses to that budget.

This is not #2310. #2310 is the later relink-confirm Unknown → NO_REPLICA_HAS_PART path (part 1 replica divergence). This issue is the control-plane write that dies first on part 3, including on non-replicated tables. They compose: catalog timeouts leave the ledger recovering, which feeds #2310.

Discovered on PR #2300 cas_s3_cache_alter_attach_3 (job), SHA f377ba3a499beacf5d032f3d34a69185765b8699, package 26.6.4.20001.altinityantalya. Same class on every CAS attach part-3 shard of that run (x86/arm × --cas / --cas-s3-cache, attempts 1 and 2 = 8/8). Isolated /alter/attach partition/part 1/* and a 63-scenario subset pass; the full part-3 Pool() is what saturates the catalog.

This issue:

• Fails user INSERT / ATTACH with NETWORK_ERROR after a 90 s CAS policy, not a network fault
• Is CAS-only and load-sensitive. Local-disk attach part 3 is not in this PR’s job set; isolated CAS runs pass
• Is not unique to CAS improvements #2300 — same attach-load class already failed on the CAS baseline (R7). CAS improvements #2300 still owns making antalya-26.6 CAS attach reliable
• Must not be xfailed. The assertion is correct

---

How to reproduce the behavior

Environment

• Version: 26.6.4.20001.altinityantalya (PR CAS improvements #2300 head f377ba3)
• Build type: release (Altinity Antalya)
• Arch: x86_64 and aarch64 (both fail)
• Flags: --use-keeper --with-analyzer
• Storage: --cas or --cas-s3-cache (storage_policy = 'cas_policy', shared pool)

CI combinatorics (reliably hits it)

From altinity/clickhouse-regression/alter. Part 3 runs seven features in one Pool(), and simple attach is @Repeat(100) with an inner Pool(6):

./regression.py --local --with-analyzer \
  --clickhouse-binary-path docker://<pr-2300-package> \
  --cas \
  --only '/alter/attach partition/part 3/*'

--cas-s3-cache fails the same way. --only of a single failed leaf does not reproduce — the catalog needs the concurrent CREATE/INSERT/ATTACH storm.

On this SHA, /alter/attach partition/part 1/* and part 2/* passed on all four CAS attach jobs. Part 3 is the reliable trigger.

Local measurements (CAS author, part 1 full set, ~4 min)

These are the amplifier, not the part-3 INSERT failure:

• REPLACE_RANGE / DROP_RANGE run in the common background pool
• On CAS, one REPLACE_RANGE is ~1.4 s p50 (mean 1.73 s)
• Suite keeps ~120 replicated tables in flight; pool ~95 % busy
• Queue rows wait minutes (20+ s typical, up to 214 rows/node)
• 116 / 1006 queue rows had num_tries ≥ 19, empty last_exception, num_postponed = 0 (max 58) — never executed, only refused admission
• First real error then gets the 300 s backoff immediately

---

Expected behavior

1. A 10-row INSERT into a CAS MergeTree table succeeds. ATTACH PARTITION FROM on the same disk succeeds.
2. CAS control-plane contention (cas/ref_catalog token-CAS, _ckpt RMW) is queued or retried inside the write, or surfaced as a retryable error the client/test can wait out — not a 90 s hard fail mapped to NETWORK_ERROR.
3. A replication-queue entry that was never scheduled must not increment num_tries. The first real exception on that entry must start backoff from try 1, not from a counter already at 19–58.
4. After attach, every replica has the source partition’s rows (RQ.SRS-048.CAS.Relink.AttachPartitionFrom).

---

Actual behavior

On release builds (26.6.4.20001.altinityantalya, #2300 f377ba3)

This job (cas_s3_cache_alter_attach_3), first leaf:

/alter/attach partition/part 3/check simple attach partition 2/simple attach partition/run #9/partition key _intDiv_a_2__intDiv_b_2__ tables partitioned_MergeTree empty_partitioned_MergeTree

Code: 210. DB::Exception: Received from localhost:9000. DB::Exception: CAS write could not be committed (read of 'data/cas/ref_catalog': gave up at the policy deadline after 13 attempt(s)); retrying later. (NETWORK_ERROR)
(query: INSERT INTO source_41bbc7e0_acde_11f1_92d3_9200088b5af9 (p, i) SELECT 3, rand64() FROM numbers(10))

Same burst (~06:11–06:12 UTC) also failed rbac/pattern #159 and two partition key ReplicatedMergeTree combinations (TestFlows only prints the first traceback).

Sibling part-3 job (aarch64 cas_s3_cache_alter_attach_3):

Code: 210. DB::Exception: CAS write could not be committed (CAS _ckpt for namespace 'alter-cas-clickhouse1/store/adf/adf9421d-…@cas@': persistent CAS contention, the checkpoint contribution was not published); retrying later. (NETWORK_ERROR)
(query: ALTER TABLE destination_14b532f0_… ATTACH PARTITION 3 FROM source_14b5327a_…)

No crash, no sanitizer, no server death. Move/replace CAS jobs on the same SHA are green.

Consistency on this SHA

Job	part 1	part 2	part 3	move	replace
CAS x86 / arm	OK	OK	Fail ×2	OK	OK
CAS S3 cache x86 / arm	OK	OK	Fail ×2	OK	OK

8/8 part-3 CAS jobs failed. Different leaves each time (run #0 vs #9, different keys/engines).

---

Root cause analysis

Two defects compose. Line numbers are #2300 f377ba3 (altinity/feature/antalya-26.6/CAS-improvements), read from a local clone after fetching that branch. Do not use the older #2310 source map (684161d).

Defect 1 — CAS control-plane write loses a 90 s policy (this job)

cas/ref_catalog is one object for the whole pool (spec INV-3):

/// The whole-pool namespace catalog (spec INV-3): one object, key `cas/ref_catalog`
/// (`Layout::refCatalogKey`), read on every fold round and every recovery, mutated by one token-CAS
/// write per lifecycle transition.

Key: prefix + "/cas/ref_catalog" (CasLayout.h refCatalogKey()). Every namespace CREATE/recovery/fold re-GETs it. Concurrent attach combinatorics (many tables, many namespaces) turns that object into a hotspot.

Default write policy is 90 s (CasRetry.h):

    /// `within(90'000)` -- the default write policy.
    static Retry standard() { return within(90'000); }

When the read/RMW loop hits that bound it throws NETWORK_ERROR:

void throwCasWriteRetryLater(const String & why)
{
    logCasWriteRetryLater(why);
    throw Exception(ErrorCodes::NETWORK_ERROR, "CAS write could not be committed ({}); retrying later", why);
}

void CasOperation::giveUpReadDeadline(...)
{
    last_read_stop = ReadStop::PolicyExhausted;
    throwCasWriteRetryLater(fmt::format("{} of '{}': gave up at the {} deadline after {} attempt(s)",
                                        verb, subject, bound.lease_bound ? "lease" : "policy", attempts_made));
}

That is the exact job-log string: read of 'data/cas/ref_catalog': gave up at the policy deadline after 13 attempt(s).

The sibling ATTACH failure is the same class on the per-namespace checkpoint. After readModifyWrite returns GaveUp (not FenceLost):

        throwCasWriteRetryLater("CAS _ckpt for namespace '" + life.ns.string()
            + "': persistent CAS contention, the checkpoint contribution was not published");

The comment on throwCasWriteRetryLater says the condition is “expected to self-heal (the caller retries)”. The caller here is a user INSERT / ATTACH with no retry. The test (and a client) sees a hard error.

Defect 2 — num_tries increments on an entry that was never scheduled (amplifier)

ATTACH PARTITION FROM on ReplicatedMergeTree is a dummy-range REPLACE_RANGE (StorageReplicatedMergeTree.cpp ~457–460). That type is not fetch and not merge/mutate, so it goes to the common executor:

    assignee.scheduleCommonTask(
        std::make_shared<ExecutableLambdaAdapter>(
            [this, selected_entry]() mutable { return processQueueEntry(selected_entry); }, ...),
        /* need_trigger */ true);
    return true;   // return value of scheduleCommonTask is ignored

Common pool: max_threads == max_tasks == background_common_pool_size (default 8). Unlike merge/mutate, there is no extra task slot.

    shared->common_executor = std::make_shared<OrdinaryBackgroundExecutor>(
        ThreadName::MERGETREE_COMMON,
        background_common_pool_size,
        background_common_pool_size,   // max_tasks == max_threads
        ...
    );

trySchedule refuses immediately when value >= max_tasks_count (MergeTreeBackgroundExecutor.cpp ~161–171). No queue.

Order that ages a never-run entry:

1. selectQueueEntry() → selectEntryToProcess() constructs CurrentlyExecuting, which does ++entry->num_tries (ReplicatedMergeTreeQueue.cpp:2019) and sets currently_executing.
2. scheduleCommonTask → trySchedule. If the pool is full this returns false. The ExecutableLambdaAdapter is destroyed; ~CurrentlyExecuting clears currently_executing and future-part tags but does not decrement num_tries (ReplicatedMergeTreeQueue.cpp:2077–2111).
3. scheduleDataProcessingJob still return true.
4. Backoff (getPostponeTimeMsForEntry) is 1 << num_tries ms, capped at max_postpone_time_for_failed_replicated_tasks_ms = 300 s, but only after last_exception_time_ms is set. Schedule refusals leave last_exception empty, so the entry is immediately eligible again and the counter keeps climbing (author: 116/1006 rows, num_tries ≥ 19, empty exception, num_postponed = 0).
5. The first real failure (Defect 1, or CAS relink confirm returns Unknown after ATTACH PARTITION FROM; replicas never receive attached parts #2310 confirm Unknown) therefore starts at try 19–58 and is postponed for the full 300 s. SYNC REPLICA in the suite is also 300 s.

How this relates to #2310

#2310 is still correct and still open: confirm Unknown is collapsed with No, taxonomy row 3 throws NO_REPLICA_HAS_PART and forbids byte-fetch (DataPartsExchange.cpp:1632–1636, ContentAddressedExchange.h:19–22). That is the first real error once the queue finally runs.

This issue is why the ledger is not quiescent (catalog/_ckpt timeouts → recovery) and why that first error is immediately unrecoverable inside the 300 s test budget (Defect 2). Fixing only #2310 leaves part-3 INSERT/ATTACH red. Fixing only the pool size leaves confirm Unknown and the num_tries accounting bug.

---

Suggested fix

Do all three. Raising background_common_pool_size to 16–32, or cutting suite parallelism, is a CI workaround only.

A. Catalog / _ckpt must not fail a user write in 90 s (this issue)

• Serialize writers of cas/ref_catalog (single in-process mutex / admission), or give the catalog its own longer/unbounded policy distinct from blob PUTs
• On _ckpt GaveUp (contention, not fence): retry under the caller’s lease, or return a value the attach/insert path can wait on — do not throw NETWORK_ERROR to the client
• Do not map “policy exhausted on a shared control object” to NETWORK_ERROR for a user query; that code is the replication-queue retry class and is the wrong signal here

B. Do not increment num_tries until the task is admitted (this issue, also worth upstream)

• Increment num_tries in processQueueEntry / executeLogEntry after trySchedule succeeded, or decrement in ~CurrentlyExecuting when func never ran
• Honour scheduleCommonTask’s false in scheduleDataProcessingJob (today it is ignored)
• Optionally: give REPLACE_RANGE the fetch pool, or give the common executor max_tasks > max_threads like merge/mutate

C. Confirm Unknown ≠ “source in doubt” (#2310)

Already specified there: split No / Unknown on the wire, or byte-fetch when gate 0 still has the part.

Do not

• Xfail /alter/attach partition/…
• Only lengthen the test timeout
• Call Defect 1 “the test is too aggressive”. Isolated runs pass; a 10-row INSERT must survive concurrent attaches on the same pool

Tests that must go green

1. cas_alter_attach_3 and cas_s3_cache_alter_attach_3 (x86 + aarch64) on this package
2. The first leaf of this job: MergeTree → MergeTree simple attach tuple-key INSERT
3. Existing cas/tests/replicated.py replicated_attach_partition_from
4. CAS relink confirm returns Unknown after ATTACH PARTITION FROM; replicas never receive attached parts #2310’s confirm-Unknown follower-convergence test, once filed there

---

Additional context

CI failure

• PR: #2300 CAS improvements
• Job: RegressionTestsRelease / CASS3CacheAlter (attach, 3) / cas_s3_cache_alter_attach_3
• Run: 34388936376 attempt 2 (attempt 1 failed the same four part-3 jobs)
• Job log: https://github.com/Altinity/ClickHouse/actions/runs/34388936376/job/102757852231?pr=2300
• Siblings (same SHA, same class):
  • cas_alter_attach_3 x86
  • cas_s3_cache_alter_attach_3 arm (_ckpt contention on ATTACH)
  • cas_alter_attach_3 arm
• Commit: f377ba3a499beacf5d032f3d34a69185765b8699
• Package: 26.6.4.20001.altinityantalya
• CI report: https://s3.amazonaws.com/altinity-build-artifacts/PRs/2300/f377ba3a499beacf5d032f3d34a69185765b8699/34388936376/ci_run_report.html
• Base: antalya-26.6

Related issues

• #2310 — relink confirm Unknown after ATTACH (part 1 replica divergence). Keep that issue. This one is the write-deadline + num_tries pair that part 3 hits first.
• #2173 — cross-disk tmp_replace_from_* unique-ref (different path)
• #2233 — relink NETWORK_ERROR storm on soak (related load)
• #2031 — static audit; CAS-112 is the uncached ref_catalog GET on every positive ref-log append

Requirements

RQ.SRS-048.CAS.MergeTree.InsertSelect / Transparency
  A 10-row INSERT on cas_policy must succeed.

RQ.SRS-048.CAS.Relink.AttachPartitionFrom
  ATTACH PARTITION FROM SHALL preserve correct logical partition contents
  on all replicas.

RQ.SRS-048.CAS.Relink.CrossPool.ByteFallback
  SHALL fall back to byte fetch when confirmation is unavailable.

Scope (do not file one issue per leaf)

All unexpected Fail leaves on the four part-3 CAS jobs of run 34388936376 are this class: simple attach (tuple keys, @Repeat(100)), partition key / partition key datetime, rbac. Ancestor Fail rows are not separate bugs.

Workarounds (CI only)

• Raise background_common_pool_size 2–4× (8 → 16 or 32)
• Cut intra-part TestFlows Pool() concurrency / Repeat(100)
• Neither fixes Defect 1 on a 10-row INSERT once the catalog is hot

[filimonov]

Part 1 is build on assumptions, and is (most probably) incorrect.
Part 2 is real defect of the upstream code.

Both are not release blockers, let's try background_common_pool_size=32 and/or reduce test concurrency