CAS: concurrent SELECT FINAL drops the mount lease with no object-store outage (cas_selects)

[alsugiliazova]

Describe the situation

On a quiet local MinIO, with no injected fault, concurrent SELECT … FINAL (joins / LIMIT / LIMIT BY / PREWHERE) drives the CAS pool from Live to TransientNotLive. Every subsequent read on cas_disk is refused with NETWORK_ERROR (210) until the disk self-remounts.

Found in RegressionTestsRelease / CAS (selects) / cas_selects on PR #2300 (CAS improvements), commit d876cda683f06972f54470910c0366a93002671d.

This is not the #2243 port-exhaustion signature. That issue was closed after <http_keep_alive_timeout>60</http_keep_alive_timeout> was added to the --cas disk config. This run's clickhouse-server.log has zero EADDRNOTAVAIL / Cannot assign requested address / CasMountLeaseKeeper / background renewal failed lines. The client still gets the lease-not-held refusal; the server never logs why the lease was dropped.

This issue:

• Takes cas_disk offline for reads for ~35 s in the middle of /selects/final/force/concurrent (02:20:05–02:20:39 UTC)
• Recovers: later scenarios in the same feature (PREWHERE with DML, WHERE) pass, so the pool did return to Live
• Is CAS-specific: non-CAS selects on the same run is green; cas_s3_cache_selects (cache in front of CAS) is green
• Is not deterministic across arch: AArch64 cas_selects on the same SHA passed
• Does not crash the server (no <Fatal>, no sanitizer, no core)

Related: #2243 (closed) — same suite, same error string, but that report was driven by ephemeral-port exhaustion (errno 99) and CasMountLeaseKeeper give-up. Comment on that issue (2026-09-09) already notes cas_selects still failing 15 of 47 executions after the keep-alive workaround, most of them with mount lease not held and 0 EADDRNOTAVAIL. This run is another of those.

---

How to reproduce the behavior

Environment

• Version: 26.6.2.20001.altinityantalya
• Build type: release, amd (build_amd_binary)
• Package: PR #2300 d876cda683f06972f54470910c0366a93002671d
  • https://altinity-build-artifacts.s3.amazonaws.com/PRs/2300/d876cda683f06972f54470910c0366a93002671d/build_amd_binary/clickhouse
• Store: MinIO (http://minio:9001/cas/data/), single container
• Disk: cas_disk overriding the default policy (selects/cas_mode.py), no filesystem cache in front
• Analyzer: --with-analyzer

Steps

1. Run the concurrent FINAL feature against a CAS default policy (the isolated failing scenario may not be enough — CAS: object-store connections exhaust the container's ephemeral ports, fencing out the mount lease #2243 needed the whole /selects/final/* subtree to accumulate parts):

python3 -u selects/regression.py \
  --clickhouse https://altinity-build-artifacts.s3.amazonaws.com/PRs/2300/d876cda683f06972f54470910c0366a93002671d/build_amd_binary/clickhouse \
  --cas --local --with-analyzer --test-to-end \
  --only '/selects/final/*' \
  -l test.log

2. The first leaf that fails in this run:

SELECT count()
FROM ReplacingMergeTree_table_<uid>_core FINAL
INNER JOIN MergeTree_table_<uid>_core
    ON ReplacingMergeTree_table_<uid>_core.id = MergeTree_table_<uid>_core.id
FORMAT TabSeparated
SETTINGS final = 1

3. While it runs, poll system.cas_mounts (lifecycle / lease holder) on clickhouse1. The expected transition is Live → TransientNotLive → Live with no object-store fault.

---

Expected behavior

Concurrent SELECT … FINAL on a healthy CAS disk must keep the mount lease. Reads should return rows. A lease blip is the designed response to backing unavailability — it must not happen when MinIO is up and no fault was injected.

---

Actual behavior

On this release build (amd)

The test harness sees a server exception (not a result mismatch):

Received exception from server (version 26.6.2):
Code: 210. DB::Exception: Received from localhost:9000. DB::Exception:
content-addressed disk 'cas_disk' -- mount lease not held; backing may be
temporarily unreachable; the operation is admitted again once the disk
recovers to Live; TRANSIENT unavailability, not damage:
(while reading from part data/store/34c/34c354f8-e80c-4280-9bb9-0ecf0d6dda6e/1_123_123_1/
 in table default.ReplacingMergeTree_table_2fb28bf9_abf3_11f1_990f_92000882c33e_core
 (34c354f8-e80c-4280-9bb9-0ecf0d6dda6e) located on disk cas_disk of type s3,
 from mark 0 with max_rows_to_read = 1, offset = 0):
While reading part 1_123_123_1:
While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder).
(NETWORK_ERROR)

(query: SELECT count() FROM ReplacingMergeTree_table_2fb28bf9_abf3_11f1_990f_92000882c33e_core  FINAL INNER JOIN MergeTree_table_2f30564a_abf3_11f1_89c5_92000882c33e_core on ReplacingMergeTree_table_2fb28bf9_abf3_11f1_990f_92000882c33e_core.id = MergeTree_table_2f30564a_abf3_11f1_89c5_92000882c33e_core.id FORMAT TabSeparated
)

Call site in the suite: selects/tests/steps/join.py:67 → helpers/cluster.py:1289 (assert False on Exception: in the client output).

Failing tests (leaf + parents)

From the job log, 02:20:05–02:21:25 UTC (container clock 04:20–04:21 CEST):

Time (UTC)	Result	Path	Duration
02:20:05	Fail	/selects/final/force/concurrent/joins/ReplacingMergeTree_table_core	26s
02:20:05	Fail	/selects/final/force/concurrent/joins	41s
02:20:08	Fail	/selects/final/force/concurrent/SELECT LIMIT BY parallel/MergeTree_table_…_core	3s
02:20:12	Fail	/selects/final/force/concurrent/SELECT LIMIT BY parallel/ReplacingMergeTree_table_…_core	3s
02:20:12	Fail	/selects/final/force/concurrent/SELECT LIMIT BY parallel	7s
02:20:17	Fail	/selects/final/force/concurrent/SELECT LIMIT BY parallel inserts, deletes, updates	5s
02:20:20	Fail	/selects/final/force/concurrent/SELECT LIMIT parallel/MergeTree_table_…_core	3s
02:20:24	Fail	/selects/final/force/concurrent/SELECT LIMIT parallel/ReplacingMergeTree_table_…_core	3s
02:20:24	Fail	/selects/final/force/concurrent/SELECT LIMIT parallel	6s
02:20:31	Fail	/selects/final/force/concurrent/SELECT LIMIT parallel inserts, deletes, updates	6s
02:20:34	Fail	/selects/final/force/concurrent/SELECT PREWHERE parallel/MergeTree_table_…_core	3s
02:20:39	Fail	/selects/final/force/concurrent/SELECT PREWHERE parallel/ReplacingMergeTree_table_…_core	4s
02:20:39	Fail	/selects/final/force/concurrent/SELECT PREWHERE parallel	7s
02:21:25	Fail	/selects/final/force/concurrent	4m 31s

Parents /selects/final/force, /selects/final, /selects fail only because of the above.

Suite totals from the same job:

1 module (1 failed)
9 features (5 ok, 3 failed, 1 xfail)
88 scenarios (81 ok, 6 failed, 1 xfail)
3020 examples (3009 ok, 11 failed)
62395 steps (60340 ok, 868 failed, 368 skipped, 16 errored, 803 xfail)
Total time 22m 38s

The 3 failed features / 1 unsatisfied requirement are this concurrent subtree plus its ancestors. The other /selects/final work (general, alias, user rights) passed.

What the server log does not contain

clickhouse1 clickhouse-server.err.log (607 lines) is CollapsingMergeTree sign-balance warnings and the usual startup cert/cas_ prefix warnings. No lease, fence, or remount line.

clickhouse-server.log (~220 MB), streamed for:

• mount lease not held
• TRANSIENT unavailability
• Code: 210
• CasMountLeaseKeeper
• background renewal failed
• Cannot assign requested address / e.code() = 99 / EADDRNOTAVAIL
• self-remount / PartWriteTxn / superseded mount

0 hits on all of those (except a startup stack frame through claimMountAwaitingExpiry at 04:10:11, disk bring-up). The only <Error> lines after startup are expected ACCESS_DENIED from user_rights at 04:22:18, after the concurrent feature had already recovered.

So: the client was refused because the pool was TransientNotLive, and the server wrote no Error/Warning that names the lease loss.

Same-run controls

Job	Arch	Result
RegressionTestsRelease / CAS (selects) / cas_selects	amd	failed (this report)
RegressionTestsAarch64 / CAS (selects) / cas_selects	arm	passed
RegressionTestsRelease / CASS3Cache (selects) / cas_s3_cache_selects	amd	passed
RegressionTestsRelease / Common (selects) / selects	amd	passed

---

Root cause analysis

The refusal itself is the designed TransientNotLive gate, not a bad exception class.

ContentAddressedMetadataStorage::checkOpAdmitted (ContentAddressedMetadataStorage.cpp:1138-1154):

if (lc == Cas::PoolLifecycle::Live)
    return CasOpAdmission::Proceed;

if (lc == Cas::PoolLifecycle::TransientNotLive)
    Cas::throwCasTransientUnavailable(
        fmt::format("content-addressed disk '{}'", disk_name),
        "mount lease not held; backing may be temporarily unreachable; the operation is admitted "
        "again once the disk recovers to Live");

throwCasTransientUnavailable (CasRequestControl.cpp:203-214) mints ErrorCodes::NETWORK_ERROR so ReplicatedMergeTreePartCheckThread will not detach the part.

Live → TransientNotLive is CasMountRuntime::noteLeaseLost() (called from the mount-fence trip). Recovery is TransientNotLive → Live after self-remount. The ~35 s outage length matches the constructed remount wait described on #2243 (~1.2× mount_lease_ttl_ms).

What is not explained by this run: why noteLeaseLost() fired. The #2243 path (lease renewal failing on EADDRNOTAVAIL) is absent from these logs. Candidates that still fit a silent drop:

1. Lease renewal failed as a transient the keeper did not log at Error/Warning.
2. The known renewOnce / onRenewCommitted exception-safety hole (audit bc3-7): a successful renew can still call onRenewFailed() and fence the mount.
3. scheduleRemount latching remount_running before the thread starts (audit bc3-8), stretching the TransientNotLive window.
4. Load-induced renewal starvation under concurrent FINAL prefetch with no cache (same pressure as CAS: object-store connections exhaust the container's ephemeral ports, fencing out the mount lease #2243, different failure mode now that keep-alive is 60 s).

The soak harness already classifies this exact message as mount_fenced and retries it (cas/soak/tests/test_mount_fence_retry.py, cas/soak/soak/cluster.py). That is correct for chaos. It is not correct as the only answer for a green-path cas_selects run: the selects suite must not xfail or retry this, or the availability hole disappears from CI.

---

Additional context

CI failure

• Job: RegressionTestsRelease / CAS (selects) / cas_selects
• PR: #2300 CAS improvements (feature/antalya-26.6/CAS-improvements → antalya-26.6)
• Commit: d876cda683f06972f54470910c0366a93002671d
• Run: 34294710256 attempt 1, 2026-09-09 02:02–02:28 UTC
• Binary: https://altinity-build-artifacts.s3.amazonaws.com/PRs/2300/d876cda683f06972f54470910c0366a93002671d/build_amd_binary/clickhouse
• Suite report: report.html
• Artifacts index: S3 index
• GitHub artifact: cas_selects-artifacts-release_zookeeper (id 10085730770)
• Direct logs:
  • raw.log
  • fails.log.txt
  • clickhouse1 err.log
  • clickhouse1 server.log

Command line from the job

python3 -u selects/regression.py \
  --clickhouse https://altinity-build-artifacts.s3.amazonaws.com/PRs/2300/d876cda683f06972f54470910c0366a93002671d/build_amd_binary/clickhouse \
  --cas --test-to-end --no-colors --local --collect-service-logs --output new-fails \
  --cicd --log raw.log --with-analyzer

Version attr recorded by the job: 26.6.2.20001.altinityantalya.

Startup (clickhouse1, not causal)

2026.09.09 04:10:10 <Warning> ContentAddressedSettings: content_addressed disk
  `storage_configuration.disks.cas_disk`: server_root_id use the superseded
  unprefixed spelling and are applied for now; write them with the `cas_` prefix.
2026.09.09 04:10:10 <Information> DiskObjectStorage(cas_disk): Starting up disk cas_disk
2026.09.09 04:10:11 <Information> DiskObjectStorage(cas_disk): Disk cas_disk started up

S3 client for cas_disk initialized with s3_retry_attempts: 500 (then a second client with 0 — the CAS metadata path).

Related

• #2243 — port exhaustion fencing the lease; closed; keep-alive workaround in helpers/cas_storage.py (CAS_S3_KEEP_ALIVE_TIMEOUT = 60, CAS_S3_KEEP_ALIVE_MAX_REQUESTS = 1000)
• #2233 — different mechanism (relink NETWORK_ERROR storm on soak)
• Do not xfail /selects/final/force/concurrent — that hides the availability hole. cas_s3_cache_selects staying green is a control, not a fix.

Suggested next facts

• Rerun amd cas_selects on this SHA (n=1 here; arm already passed).
• On a repro, dump system.cas_mounts and system.events (DiskConnectionsCreated/Reused/Expired/Reset) for the 02:20 window, and raise the CAS mount-lease logger so a silent noteLeaseLost is visible.

[filimonov]

Re-checked this against the job's own artifact (the S3 selects/ path was overwritten by the cas_s3_cache_selects job at the time, fixed since by 0fa0a34) and reproduced it locally. Several statements in the report need correcting, and the root cause is not in CAS.

Corrections to the report

• The server does log why the lease was dropped. In the GitHub artifact cas_selects-artifacts-release_zookeeper (run 34294710256), clickhouse1 has at 04:19:54.132:
  <Warning> CasMountLeaseRenewer: CAS mount renewal 'selects-cas-clickhouse1' fenced after 66 physical attempts in 4010 ms (classification=external_lease_deadline, …)
  followed by CasPool: CAS whole-chain remount attempt 1 failed at step 'pool_identity_probe', attempt 2, the ~36.5 s token-stability wait, and attempt 3 succeeded at step 'publish_live' at 04:20:37. The grep list in the report used names from before the renewal rework: the CasMountLeaseKeeper logger and its background renewal failed message were replaced in this PR (37c9bd4 / c3f7b20, 2026-09-05, migration onto CasRequests) by the CasMountLeaseRenewer logger, which reports one line per outcome: CAS mount renewal '<root>' recovered after N physical attempts in M ms (...) at Information, or ... fenced after N physical attempts in M ms (classification=...) at Warning, plus physical retry attempt K lines at Debug. Neither of the old strings exists in the tree any more, so grepping for them returns nothing on any run.
• It is the CAS: object-store connections exhaust the container's ephemeral ports, fencing out the mount lease #2243 port-exhaustion signature. The same log has 2814 lines of Poco::Exception. Code: 1000, e.code() = 99, Net Exception: Cannot assign requested address: 172.19.0.4:9001 between 04:19:43 and 04:19:56 (peak 342 per second), on manifest PUTs, blob GETs and all 66 renewal attempts. Only clickhouse1 is affected; nodes 2 and 3 have zero.
• The keep-alive workaround was already in effect. Regression release got 69be3aa720 (keep-alive 60 s / 1000 requests) at 01:41 UTC, the job checked out release at 02:02 UTC.
• AArch64 is not immune. Run 9 (b13776d) aarch64 cas_selects failed with the identical signature: 1244 errno 99 lines at 19:57:33–36, fenced after 63 physical attempts in 3975 ms, Live again 48 s later.
• Hypotheses 2–4 in the report (renewOnce/onRenewCommitted, remount_running, silent loss) do not apply to these runs; the renewer behaved as designed and was one of the victims.

Local reproduction

selects/regression.py --cas --local --with-analyzer --only '/selects/final/*' (clickhouse-regression release, release build of the PR branch, the suite's object store is RustFS rustfs/rustfs:1.0.0-beta.12 behind the minio:9001 service name, keep-alive 60/1000), sampling /proc/net/tcp states and system.events in clickhouse1 every 2 s:

Run	Disk	Peak TIME_WAIT	Connections created / reset	errno 99 lines	Lease
1	CAS (metadata_type=cas)	29360	204804 / 206263	4667	survived, 87/88 scenarios ok
2	plain S3 (metadata_type=local, same suite)	29790	142696 / 146981	13438	survived

The container's ephemeral range is 32768–60999 (28k ports) and tcp_tw_reuse=2 (loopback only), so a closed connection holds a port for 60 s. Both disks exhaust the range during /selects/final/force/concurrent; whether the renewal happens to fall inside the exhausted window decides between "green" and this issue.

Mechanism

• 100 % of DiskConnectionsReset are attributed to queries in system.query_log; CAS background operations (renewal, GC, ledger) reset nothing.
• DiskConnectionsReset == DiskConnectionsCreated: every connection serves exactly one request and is then closed instead of returned to the pool. ReadBufferSeekCancelConnection is 0, so seeks are not involved.
• A/B on a suite table with 229 parts: SELECT sum(id) = 229 GETs, 229 resets, with the prefetched pool on or off, remote_filesystem_read_prefetch=0, or max_threads=1. SELECT * on the same table = 0 resets.
• The suite's tables consist of hundreds of one-granule compact parts (~800 B, 4 columns). Reading a column subset consumes only that column's block of data.bin, while read_until_position (MergeTreeReaderStream::adjustRightMark) spans the whole granule. The HTTP body is left unread, PooledConnection marks the response incomplete and resets the socket. Under the concurrent scenarios that is several hundred sockets per second.
• Why it looks CAS-specific: cas_s3_cache_selects is green because the cache absorbs the GETs, and the non-CAS selects job runs on a local disk, so it is not a control.

Fix direction

Generic MergeTree-over-object-storage, upstream-worthy:

1. ReadBufferFromS3 (or the pooled connection) should drain a small unread remainder on release instead of resetting the connection; here the remainder is a few hundred bytes.
2. Alternatively, per-column byte ranges for compact-part reads so the requested range matches what is consumed.

The lease renewer needs no change for this.

Workaround for the regression stands

sysctls: - net.ipv4.tcp_tw_reuse=1 (optionally a wider net.ipv4.ip_local_port_range) on the clickhouse containers in docker-compose/clickhouse-service.yml. It is namespaced, so it applies per container without touching the host. This removes the port exhaustion and therefore this failure; it does not change the connection churn itself.

Artifacts and sampling data are in the PR watch notes; happy to attach the per-second tables.