Skip to content

Honor materialized_postgresql_schema in MaterializedPostgreSQL table engine#107425

Merged
alexey-milovidov merged 24 commits into
masterfrom
fix-matpg-table-engine-schema
Jul 15, 2026
Merged

Honor materialized_postgresql_schema in MaterializedPostgreSQL table engine#107425
alexey-milovidov merged 24 commits into
masterfrom
fix-matpg-table-engine-schema

Conversation

@alexey-milovidov

@alexey-milovidov alexey-milovidov commented Jun 13, 2026

Copy link
Copy Markdown
Member

The standalone MaterializedPostgreSQL table engine did not work for a relation that lives in a non-default PostgreSQL schema (selected via the materialized_postgresql_schema setting).

Update after merging master. The original failure of this PR — CREATE PUBLICATION ... FOR TABLE ONLY <table> issued without the schema, so pqxx::undefined_table: ERROR: relation "<table>" does not exist — is now fixed on master by #107423 (commit 190a65c1556). That change quotes the single-table publication's tables list via doubleQuoteWithSchema, which also schema-qualifies it. The redundant publication tables-list change originally made here has therefore been reverted, leaving createPublicationIfNeeded identical to master.

What this PR fixes now: the replication identity. getPublicationName and the default getReplicationSlotName derived their names from <postgres_database>_<table> only, ignoring the schema. Two standalone MaterializedPostgreSQL tables that replicate a table with the same name from two different schemas of the same PostgreSQL database therefore shared one publication and one default replication slot. The second CREATE dropped and recreated the shared publication, and because the publication carried only the bare relation name (schema_as_a_part_of_table_name stays false in this mode), the two consumers cross-talked — one replica could stop receiving its schema's changes or ingest the other schema's rows.

Fix. For the single-table engine, derive both the publication name and the default replication slot name from a bounded, collision-resistant hash of the full (database, schema, table) identity when materialized_postgresql_schema is set to a non-default schema, so the two configurations get distinct PostgreSQL objects and stay isolated. A plain database_schema_table concatenation is not injective (schema = a_b, table = c and schema = a, table = b_c both fold to database_a_b_c), and the replication-slot name is additionally lower-cased with - mapped to _, so even PostgreSQL-distinct schemas such as Foo/foo or a-b/a_b would otherwise collide. Hashing a length-prefixed (hence unambiguous) serialization of the identity keeps the generated name injective in practice and fixed-length, and the human-readable database prefix kept in front of the hash (purely for recognizability in pg_replication_slots/pg_publication) is itself capped, so the slot stays within PostgreSQL's identifier limit regardless of the database, schema and table length. The default schema is left out of the identity: an empty materialized_postgresql_schema, or the explicit default materialized_postgresql_schema = 'public', keeps the legacy schema-unaware names, so existing single-table deployments keep their current publication and slot names and ATTACH still finds them.

Database engine. The same schema-blind identity affected the MaterializedPostgreSQL database engine (empty remote table name): getPublicationName and the default getReplicationSlotName derived the name from <postgres_database> alone, so two CREATE DATABASE ... ENGINE = MaterializedPostgreSQL statements over the same PostgreSQL database but different non-default materialized_postgresql_schema shared one publication (and, without a unique replication consumer identifier, one default slot) and cross-talked in the same way. The fix reuses the same collision-resistant, length-bounded identity for the database engine when it replicates a single common non-default schema — the remote table name is empty, so the identity is over (database, schema, ""), which stays distinct from any single-table identity. The default-schema database-engine path keeps the legacy <postgres_database> names, and existing non-default-schema deployments are covered by the legacy-identity adoption described below. The materialized_postgresql_schema_list / materialized_postgresql_tables_list filters are intentionally left unchanged: keying the identity on them would rename the publication and slot of every existing filtered database-engine deployment and break its ATTACH (a re-sync would reload a snapshot into the already-existing nested tables, duplicating data), so hardening those filters needs a separate compatibility/migration story.

Backward compatibility: legacy-identity adoption on ATTACH. Deployments created before the identity became schema-aware already exist for both modes: the database engine's single-schema mode is years old (restart-covered by test_2.py::test_database_with_single_non_default_schema), and the single-table engine works with a non-default schema since 26.6 (#107423 is in v26.6.1.1193-stable). They own the legacy, schema-blind slot and publication, so switching the generated names alone would break them on upgrade: restart / ATTACH would look for the schema-aware slot, miss the existing one, run an initial sync that reloads a snapshot into the already-populated nested tables, and leave the legacy slot orphaned on the PostgreSQL side, retaining WAL forever. A new adoptLegacyReplicationIdentityIfNeeded therefore runs on attach (in startSynchronization and, for the database engine, in fetchRequiredTables, which consults the publication first): when the schema-aware objects do not exist but the legacy ones do, the handler adopts the legacy names and keeps using them, including for DROP cleanup. The slot carries the evidence (its loss is what triggers the destructive re-sync); for slot names that do not depend on the schema (a user-managed slot or a unique replication consumer identifier), the publication carries it instead. Adoption is fail-closed: the legacy names are schema-blind and can belong to a coexisting engine over the default schema of the same PostgreSQL database, so the legacy publication is adopted only when every table it publishes belongs to this engine's schema; otherwise the schema-aware identity is kept and the attach proceeds as a fresh setup instead of hijacking another engine's slot and publication.

Tests. Adds test_two_schemas_same_table_name_single_storage, which replicates the same table name from two schemas into two standalone engines and asserts that both the initial snapshot and ongoing replication stay isolated. Adds test_two_schemas_same_table_name_database_engine, the database-engine counterpart: it replicates the same table name from two schemas of one PostgreSQL database into two standalone database engines (no unique consumer identifier, so isolation comes solely from the schema-aware naming) and asserts that the initial snapshot and ongoing replication stay isolated and that the two engines own distinct publications and replication slots within the 63-character limit. Adds test_schema_aware_identity_publication_separator_collision (a_b.c versus a.b_c) and test_schema_aware_identity_slot_hyphen_distinct (a-b.t versus a_b.t), which assert that the formerly-colliding identities now own distinct publications and replication slots and replicate in isolation. Keeps test_single_table_engine_with_non_default_schema (single non-default schema). Adds test_default_schema_preserves_legacy_identity, which creates the single-table engine with the explicit default materialized_postgresql_schema = 'public' and asserts (via pg_replication_slots and pg_publication) that the legacy schema-unaware publication and slot names are used, so ATTACH of tables created before the identity became schema-aware keeps working. Adds test_schema_aware_identity_long_database_name, which replicates a non-default-schema table under a long PostgreSQL database name (whose unbounded slot name would have exceeded the identifier limit) and asserts that the initial snapshot and ongoing replication succeed and that the generated publication and slot names stay within PostgreSQL's 63-character limit. Adds test_legacy_identity_adopted_on_attach_table_engine and test_legacy_identity_adopted_on_restart_database_engine, which turn a freshly created deployment into a legacy one on the PostgreSQL side (while the table is detached / the server is stopped) and assert that after ATTACH/restart the legacy identity is adopted: rows written meanwhile arrive through the legacy slot, the schema-aware objects are not recreated, and DROP cleans up the adopted objects. Adds test_legacy_identity_not_adopted_for_foreign_publication: a schema-scoped database that lost its schema-aware slot must not adopt the legacy slot/publication of a coexisting default-schema database — it re-syncs under its own identity and both databases stay isolated.

Closes: #59950
Related: #49045
Related: #107423

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Fix the MaterializedPostgreSQL table and database engines so that a single table or database can be replicated from a non-default PostgreSQL schema (materialized_postgresql_schema), including the case where tables with the same name exist in several schemas of the same database (previously they would share a publication and replication slot and cross-talk).

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

🤖 Generated with Claude Code

Version info

  • Merged into: 26.7.1.971 (included in 26.7 and later)

…engine

The standalone `MaterializedPostgreSQL` table engine ignored the
`materialized_postgresql_schema` setting. Unlike the database engine, which
schema-qualifies and quotes the tables list in `fetchRequiredTables`, the table
engine puts the bare remote table name into `tables_list` (in the storage
constructor) and never processes it. As a result `createPublicationIfNeeded`
issued `CREATE PUBLICATION ... FOR TABLE ONLY <table>` without the schema,
failing with `relation "<table>" does not exist` whenever the table lived in a
non-default schema.

Build the publication's tables list from the storages (which applies the schema
via `doubleQuoteWithSchema`) for the table engine, mirroring the database engine
behaviour. The rest of the flow (structure fetch, snapshot COPY, consumer
mapping) already used the schema correctly.

Closes: #59950
Related: #49045

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@clickhouse-gh

clickhouse-gh Bot commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Workflow [PR], commit [1595e91]

Summary:


AI Review

Summary

This PR makes MaterializedPostgreSQL use schema-aware publication and default replication-slot identities for non-default PostgreSQL schemas, extends that isolation to the database engine, and adopts legacy identities on ATTACH so existing deployments do not silently re-snapshot when only the old PostgreSQL objects exist. I rechecked the current head against the full PR diff, the existing review threads, and the current green CI report; the previously reported correctness gaps are fixed in the code now and I did not find a remaining contract violation in the current change.

Final Verdict
  • Status: ✅ Approve

@clickhouse-gh clickhouse-gh Bot added the pr-bugfix Pull request with bugfix, not backported by default label Jun 13, 2026
Comment thread src/Storages/PostgreSQL/PostgreSQLReplicationHandler.cpp Outdated
The default replication slot name is
`<postgres_database>_<table>_ch_replication_slot`, which is checked
against PostgreSQL's 63-character identifier limit in
`checkReplicationSlot` at handler construction. With the previous names
the slot name `postgres_database_test_table_engine_schema_table_ch_replication_slot`
was 68 characters, so the test aborted with
`Too big replication slot size` (a `LOGICAL_ERROR`) before the
`materialized_postgresql_schema` fix was ever exercised. Under sanitizer
builds this aborted the server, cascading into `Connection refused`
failures across the other integration test shards.

Shorten the table name so the slot name stays within the limit and the
test actually validates the schema handling end-to-end.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alexey-milovidov

Copy link
Copy Markdown
Member Author

CI triage

Integration tests (6 shards: amd_asan_ubsan flaky, amd_asan_ubsan db disk old analyzer, amd_tsan, amd_msan, arm_binary distributed plan, amd_llvm_coverage) — all the same new test, test_postgresql_replica_database_engine/test_3.py::test_single_table_engine_with_non_default_schema. Root cause was in the test itself, not the fix: the default replication slot name <postgres_database>_<table>_ch_replication_slot was 68 characters (postgres_database_test_table_engine_schema_table_ch_replication_slot), over the 64-char limit checked in checkReplicationSlot, so the table creation threw Too big replication slot size (a LOGICAL_ERROR) at handler construction — before the materialized_postgresql_schema path was ever exercised. Under sanitizer builds the LOGICAL_ERROR aborted the server, which cascaded into the Connection refused failures seen in the other shards. Fixed in 07f248a by shortening the table name; verified locally — the test now passes end-to-end (STATUS: OK, including the snapshot and ongoing-replication assertions that CI never reached).

The two remaining reds are unrelated to this change (a one-line PostgreSQL publication change):

  • Stateless tests (arm_asan_ubsan, azure, sequential)Server died / segfault during MetricLog flush (SystemLog<MetricLogElement>::flushImplPushingPipelineExecutor::finish). This is the known crash tracked in [CI crash] Segfault in ContextData destructor during MetricLog flush #99882; the same Server died/segfault hit ~44–60 distinct PRs/day in this check on 2026-06-10/06-11.
  • Stress test (arm_msan) — a recursive MSan report in HandledSignals::reset__sanitizer::Die() (src/Common/SignalHandlers.cpp:685) on shutdown; recurring on multiple unrelated PRs over the last 10 days. I could not find an existing tracking issue.

@groeneai, could you take a look at the Stress test (arm_msan) HandledSignals::reset MSan failure and provide a fix in a separate PR (or link an existing one if it's already being handled)? The stateless segfault is already covered by #99882.

@groeneai

Copy link
Copy Markdown
Contributor

@alexey-milovidov The Stress test (arm_msan) HandledSignals::reset failure has no existing tracking issue or fix PR. I opened one: #107442.

Root cause is reentrancy of the sanitizer death callback. At process exit the static HandledSignals singleton is destroyed and MSan poisons its members (the report ends with "Member fields were destroyed", the use-after-dtor origin). A later MSan warning calls __sanitizer::Die(), which runs sanitizerDeathCallback(). That callback is DISABLE_SANITIZER_INSTRUMENTATION, but the attribute does not propagate to callees, and it calls HandledSignals::reset(false). reset() iterates the poisoned handled_signals while still instrumented, raising another MSan warning that calls Die() again, recursing until the original report is destroyed and the job errors out.

The fix marks HandledSignals::reset with DISABLE_SANITIZER_INSTRUMENTATION so the only callee reachable from the death callback is no longer instrumented. No behavior change, and it is a no-op in non-sanitizer builds. This mirrors your earlier fix for the same callback in #78178. Verified at codegen level with the CI MSan flags: the reset body emits 4 __msan_warning calls without the attribute and 0 with it.

alexey-milovidov and others added 3 commits June 16, 2026 09:49
…ine-schema

# Conflicts:
#	tests/integration/test_postgresql_replica_database_engine/test_3.py
…aware

After merging master, the original publication tables-list change from this PR
became redundant: master's #107423 ("Fix MaterializedPostgreSQL replication for
upper-case database/table names", commit `190a65c1556`) added an
`else if (!is_materialized_postgresql_database)` branch in
`createPublicationIfNeeded` that quotes the single-table `tables_list` via
`doubleQuoteWithSchema`, which already applies the `materialized_postgresql_schema`
setting. So the `CREATE PUBLICATION ... FOR TABLE ONLY` query is schema-qualified
on master already. Revert that part here so master's branch is no longer dead code.

What was still missing — and what this commit fixes — is the replication *identity*.
`getPublicationName` and the default `getReplicationSlotName` derived the name from
`<postgres_database>_<table>` only, ignoring the schema. Two standalone
`MaterializedPostgreSQL` tables replicating a table with the same name from two
different PostgreSQL schemas of the same database therefore shared one publication
and one default replication slot. The second `CREATE` dropped and recreated the
shared publication, and because the publication carries only the bare relation name
(`schema_as_a_part_of_table_name` stays false in this mode) the consumers
cross-talked: one replica could stop receiving its schema's changes or ingest the
other schema's rows.

Include the schema in the publication name and the default replication slot name for
the single-table engine when `materialized_postgresql_schema` is set, so the two
configurations get distinct PostgreSQL objects and stay isolated. The database
engine path (empty remote table name) and the default-schema table-engine path are
unchanged, so existing deployments keep their current publication and slot names.

Add an integration test `test_two_schemas_same_table_name_single_storage` that
replicates the same table name from two schemas and asserts both the initial
snapshot and ongoing replication stay isolated. Shorten the schema and table names
in `test_single_table_engine_with_non_default_schema` so the now schema-aware
default slot name stays within PostgreSQL's identifier length limit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alexey-milovidov

Copy link
Copy Markdown
Member Author

/continue-pr session summary

Merged master (the branch was 1553 commits behind and had a red check) and resolved the AI Review ❌ Block finding.

Key finding from the merge. master had independently fixed the original failure of this PR. #107423 (commit 190a65c1556, "Quote the remote table name for single-table MaterializedPostgreSQL publications") added an else if (!is_materialized_postgresql_database) branch in createPublicationIfNeeded that quotes the single-table tables list via doubleQuoteWithSchema — which also schema-qualifies it. So CREATE PUBLICATION ... FOR TABLE ONLY "schema"."table" is already correct on master, and the original relation "..." does not exist is fixed there. The publication tables-list change originally made in this PR became redundant (and would have left master's branch as dead code), so it was reverted; createPublicationIfNeeded is now identical to master.

Fix for the AI Block (the genuinely-remaining issue). getPublicationName and the default getReplicationSlotName were schema-blind (<database>_<table>), so two standalone MaterializedPostgreSQL tables replicating a same-named table from two different schemas of the same PostgreSQL database shared one publication and one default slot and cross-talked. Commit 246e3e503ae makes both the publication name and the default replication slot name schema-aware for the single-table engine when materialized_postgresql_schema is set. The database-engine path and the default-schema table-engine path are unchanged, so existing publication/slot names are preserved.

Tests.

  • Added test_two_schemas_same_table_name_single_storage — replicates the same table name from two schemas into two standalone engines and asserts the initial snapshot and ongoing replication stay isolated (schema 2's values, offset by 1000, never leak into replica 1, and vice versa).
  • Shortened the schema/table names in test_single_table_engine_with_non_default_schema so the now schema-aware default slot name stays within PostgreSQL's identifier length limit.

The unresolved AI thread on PostgreSQLReplicationHandler.cpp:627 has been answered and resolved.

Verification. -fsyntax-only on the merged PostgreSQLReplicationHandler.cpp passes; the slot-name lengths for both tests were checked against the 64-char limit (58 and 44 chars). A full build after a 1553-commit merge is impractical, so the two-schema isolation test relies on CI's integration-test run (it exercises test_postgresql_replica_database_engine/test_3.py directly).

CI triage. The only red on the previous head was Stress test (arm_debug) → "Hung check failed, possible deadlock found", a widespread, tracked flake (#107941): it hit 8–50 distinct master-based PRs/day over the last two weeks (49 on 2026-06-16, this PR's CI date), and is unrelated to a PostgreSQL publication/slot naming change. The new commit re-triggers full CI.

For the author: the basic relation "..." does not exist fix that the changelog entry/category originally described now lives in #107423; this PR's remaining contribution is the multi-schema isolation hardening + tests. The description and changelog entry were updated accordingly; please confirm the "Bug Fix" category is still how you want it given the split with #107423.

Comment thread src/Storages/PostgreSQL/PostgreSQLReplicationHandler.cpp Outdated
…chema

Making the standalone `MaterializedPostgreSQL` table-engine replication
identity schema-aware (in the previous commit) changed the generated default
publication and replication-slot names for *every* non-empty
`materialized_postgresql_schema`, including the common explicit-default case
`materialized_postgresql_schema = 'public'`.

That breaks `ATTACH` of an existing standalone `MaterializedPostgreSQL` table
created before the identity became schema-aware: the default slot was
`postgres_database_<table>_ch_replication_slot`, but the new code looks for
`postgres_database_<schema>_<table>_ch_replication_slot`. When that slot is
absent, `startSynchronization` runs the initial sync, creates a new slot, and
`loadFromSnapshot` inserts a fresh snapshot into the already-existing nested
table (`createNestedIfNeeded` is a no-op when the nested table exists),
duplicating data and leaving the old slot and publication behind.

Treat PostgreSQL's default schema (`public`, or an empty setting) the same as
no schema in `getPublicationName` and `getReplicationSlotName`: only a
genuinely non-default schema is included in the generated names. This preserves
the legacy identity for the default schema (so `ATTACH` keeps finding the
existing objects) while still keeping the collision fix intact, because
`public.t` and `schema1.t` still get distinct names
(`postgres_database_t` vs `postgres_database_schema1_t`).

Adds `test_default_schema_preserves_legacy_identity`, which creates the
single-table engine with `materialized_postgresql_schema = 'public'` and
asserts that the publication and replication slot use the legacy schema-unaware
names, not the schema-aware ones.

Addresses the AI review on #107425

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread src/Storages/PostgreSQL/PostgreSQLReplicationHandler.cpp Outdated
…ine-schema

Master's "Enable more Ruff checks (F401/F403/F405/F541/F841) and clean up
tests" (`d6c076decf7`) rewrote the `from helpers.postgres_utility import (...)`
block of `test_3.py`, dropping every name that master's own copy of the file
did not use. The auto-merge took master's reduced import block, so the three
helpers this PR's new tests call as bare functions — `create_postgres_schema`,
`create_postgres_table`, and `create_postgres_table_with_schema` — were no
longer imported, and `ruff` flagged them as `F821` undefined-name on the CI
merge ref (the Style check failure on `f353f743778`). Re-add the three imports
(they still exist in `postgres_utility.py`); `ruff check` now passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread src/Storages/PostgreSQL/PostgreSQLReplicationHandler.cpp Outdated
alexey-milovidov and others added 4 commits June 25, 2026 20:41
…ine-schema

# Conflicts:
#	tests/integration/test_postgresql_replica_database_engine/test_3.py
…tant

The single-table MaterializedPostgreSQL engine derived its publication and
default replication-slot names from a plain `postgres_database_postgres_schema_postgres_table`
concatenation when `materialized_postgresql_schema` was set to a non-default
schema. That is not injective: `schema = a_b`, `table = c` and `schema = a`,
`table = b_c` both yield `postgres_database_a_b_c_*`, and the replication slot
name is additionally folded by `normalizeReplicationSlot` (lower-cased, `-`
mapped to `_`), so PostgreSQL-distinct schemas such as `Foo`/`foo` or
`a-b`/`a_b` map to one slot. Two standalone tables built from such identities
would then share a publication or a replication slot and their consumers would
cross-talk — the very failure the schema-aware identity is meant to remove.

Derive the schema-aware single-table identity from a bounded, collision-resistant
hash of a length-prefixed (hence unambiguous) serialization of the full
(database, schema, table) triple. The generated name is now injective in
practice, fixed-length (so it stays within PostgreSQL's identifier limit
regardless of the schema/table length) and within the replication-slot character
set. The database-engine path and the default-schema (empty or `public`)
table-engine path keep their legacy names unchanged.

Adds focused integration coverage for both collision classes: `a_b.c` versus
`a.b_c` (publication/slot separator collision) and `a-b.t` versus `a_b.t`
(slot hyphen-folding collision), asserting that the two engines own distinct
PostgreSQL objects and stay isolated.

Addresses the AI Review "Request changes" findings on
#107425

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alexey-milovidov

Copy link
Copy Markdown
Member Author

/continue-pr session summary

No code changes were needed — the PR is approved and has no unresolved review threads.

  • AI Review verdict: ✅ Approve (no required changes); 0 unresolved inline threads.
  • Master merge. The branch was 250 commits behind master and red, so I merged origin/master (commit 1a91a4f87d5). The merge was clean — no conflicts — and the net PR diff is unchanged (PostgreSQLReplicationHandler.cpp +75/-7 and the new test_3.py cases, +428/-7 total).
  • Verification. -fsyntax-only on the merged PostgreSQLReplicationHandler.cpp passes (the new <Common/SipHash.h> include resolves and getSchemaAwareIdentityHash compiles). Member-init order is correct: postgres_schema is declared before replication_slot/publication_name, so it is constructed first in the initializer list. The integration test test_3.py is valid Python and contains all five described cases.
  • CI triage. The only reds were Stress test (amd_tsan) and Stress test (arm_tsan) → "Hung check failed, possible deadlock found", the chronic, master-wide flake already linked by the CI bot to Hung check failed, possible deadlock found #107941 (the AST-fuzzer log tail shows __topKFilter/text-index queries, unrelated to PostgreSQL replication-identity string generation). The new commit re-triggers full CI.

Comment thread src/Storages/PostgreSQL/PostgreSQLReplicationHandler.cpp Outdated
…ase names

The schema-aware single-table publication and default replication-slot
identities folded the schema and table into a fixed-length hash but kept the
PostgreSQL database name as a full prefix, so the generated names were not
actually bounded. With a moderately long database name (>= 28 characters) the
default slot `<database>_<16 hex hash>_ch_replication_slot` exceeded
PostgreSQL's 63-character identifier limit, and `checkReplicationSlot` threw
`Too big replication slot size` in the constructor before the table could be
created — a regression for non-default-schema tables that worked with a short,
schema-blind slot name. The same unbounded prefix was used by
`getPublicationName`.

Cap the cosmetic, human-readable database prefix at 16 bytes in the new
`getSchemaAwareIdentityName` helper, which both `getPublicationName` and the
default `getReplicationSlotName` now use. The full `(database, schema, table)`
identity is still carried by the hash, so capping the prefix cannot reintroduce
a collision, and the generated names (slot, its `_tmp` variant, and the
publication) stay within the limit regardless of the database name length.

Add `test_schema_aware_identity_long_database_name`, which replicates a
non-default-schema table under a 39-character PostgreSQL database name and
asserts the initial snapshot and ongoing replication succeed and that the
generated publication/slot names stay within the 63-character limit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alexey-milovidov

Copy link
Copy Markdown
Member Author

/continue-pr session summary

Addressed the AI Review ⚠️ Request changes verdict (the slot/publication length-bound finding on PostgreSQLReplicationHandler.cpp:217). Pushed 1a91a4f87d5..881c9f748a3.

Fix. The schema-aware single-table identity folded the schema and table into a fixed-length hash but kept the full PostgreSQL database name as a prefix, so the generated names were not actually bounded. With a database name of ≥ 28 characters the default slot <database>_<16 hex hash>_ch_replication_slot exceeded PostgreSQL's 63-character identifier limit and checkReplicationSlot threw Too big replication slot size in the constructor — a regression for non-default-schema tables (the old schema-blind slot for the same short table was only 50 bytes). The same unbounded prefix was used by getPublicationName.

A new getSchemaAwareIdentityName helper caps the cosmetic, human-readable database prefix at 16 bytes, and both getPublicationName and the default getReplicationSlotName use it. The full (database, schema, table) identity is still carried by the hash, so capping the prefix cannot reintroduce a collision. The generated names are now bounded regardless of database-name length: slot ≤ 53 bytes (its _tmp variant ≤ 57), publication ≤ 48 bytes. The legacy default-schema path (<database>_<table>_*) is unchanged, so ATTACH of pre-existing slots/publications keeps working.

Test. Added test_schema_aware_identity_long_database_name, which replicates a non-default-schema table under a 39-character PostgreSQL database name (the unbounded slot would have been 76 bytes) and asserts the initial snapshot and ongoing replication succeed and that the generated publication/slot names stay within the 63-character limit.

Verification. -fsyntax-only on the edited PostgreSQLReplicationHandler.cpp passes under the project's full -Werror flags (no -Wdocumentation issues). The new test compiles (py_compile) and passes ruff check (0.15.13, the CI version). The unresolved AI thread has been answered and resolved.

CI triage (previous head, all reds unrelated to a PostgreSQL identity-string change):

  • Unit tests (msan, function_prop_fuzzer)FunctionsStress.stress / AllTests: the function property fuzzer hit a reinterpret/Decimal-scale mismatch in src/Functions (this PR touches no Functions code). This is a chronic, master-wide fuzzer flake — 48 distinct PRs failed it on 2026-06-30 alone — tracked by the auto-generated FunctionsStress #107242 (FunctionsStress, labels testing/fuzz).
  • Stress test (amd_tsan / arm_asan_ubsan / arm_tsan) → "Hung check failed, possible deadlock found": the chronic, master-wide flake already linked by the CI bot to Hung check failed, possible deadlock found #107941.

The new commit re-triggers full CI.

@alexey-milovidov

Copy link
Copy Markdown
Member Author

/continue-pr session summary

No code changes were needed — the PR is approved with no unresolved review threads.

  • AI Review verdict: ✅ Approve (no required changes); 0 unresolved inline threads. Reviewed the diff holistically once more: the schema-aware identity logic is correct, the constructor member-init order is sound (postgres_schema is declared and initialized before replication_slot/tmp_replication_slot/publication_name, so it is constructed first), and the generated names are bounded (16-byte database prefix + _ + 16-byte hash + _ch_replication_slot = 53 bytes, +_tmp = 57, inside the 63/64-byte limit).
  • No master merge. The branch already merged origin/master earlier today (merge-base 1a91a4f87d5, < 1 day old). It is "behind" only by one day of master churn, and all three reds are master-wide flakes that a re-merge cannot fix (re-merging would just re-trigger full CI and re-hit the same flakes), so no re-merge was done.
  • CI triage (head 881c9f748a3, all three reds unrelated to a PostgreSQL replication-identity string change — this PR touches only PostgreSQLReplicationHandler.cpp and test_3.py):
  • Reran the failed jobs of run 28457029613 to try to clear the flakes.
  • PR body was updated to mention the database-prefix capping in the "Fix." paragraph and to list the test_schema_aware_identity_long_database_name test added in 881c9f748a3 (the body's test list was stale).

@alexey-milovidov

Copy link
Copy Markdown
Member Author

/continue-pr session summary

No code changes — the PR is correct, AI Review verdict is ✅ Approve, there are 0 unresolved review threads, and it is MERGEABLE/BLOCKED (waiting on a human approval + merge). CI for head 881c9f748a3 is still running (24 jobs in progress).

No re-merge. The branch merged origin/master earlier today (merge-base 1a91a4f87d5, < 1 day old). All current reds are master-wide flakes that a re-merge cannot fix, so re-merging would only re-trigger full CI and re-hit the same flakes.

CI triage (this PR touches only PostgreSQLReplicationHandler.cpp and test_3.py, so none of the reds can be caused by it):

  • Fast test (arm_darwin)03135_keeper_client_find_commands — a macOS-only keeper-client stderr-noise flake (the --- log tail --- shows the test ran all commands to completion; the failure type is "having stderror"). It is master-wide: in the last 30 days it failed 17 different PRs on Fast test (arm_darwin) and nowhere else (107826, 108072, 108833, 108427, 107425, 101783, 106718, 108148, 108543, 100651, 105131, 104965, 108418, 108432, 107480, 108765, 107475). A Keeper-client test on macOS is unrelated to a PostgreSQL replication-identity string change. No tracking issue exists yet.
  • Stress test (amd_tsan) / Stress test (arm_tsan) → "Hung check failed, possible deadlock found" — the chronic, master-wide Hung-check flake already linked by the CI bot to Hung check failed, possible deadlock found #107941 (the AST-fuzzer/deadlock-detector flake), which touches none of this PR's files.

The failed jobs cannot be re-run while the "PR" workflow run is still in progress; they will be re-run once it finishes if still needed. Otherwise the PR is ready to merge.

Comment thread src/Storages/PostgreSQL/PostgreSQLReplicationHandler.cpp
…ase engine

Follow-up to the schema-aware single-table identity: extend it to the
`MaterializedPostgreSQL` database engine when it replicates a single common
non-default schema (`materialized_postgresql_schema`).

Previously `getPublicationName` and the default `getReplicationSlotName`
derived the database-engine identity from `<postgres_database>` alone, ignoring
`materialized_postgresql_schema`. Two `CREATE DATABASE ... ENGINE =
MaterializedPostgreSQL` statements over the same PostgreSQL database but
different non-default schemas therefore shared one publication
(`<postgres_database>_ch_publication`) and, without a unique replication
consumer identifier, one default slot (`<postgres_database>`). The second
`CREATE` dropped and recreated the shared publication for its own schema's
tables, so the first consumer stopped receiving its schema's changes; and
because single-schema mode keeps `schema_as_a_part_of_table_name` false, a
same-named table in the other schema could be ingested by the wrong consumer.

The fix reuses the existing collision-resistant, length-bounded
`getSchemaAwareIdentityName` (the remote table name is empty for the database
engine, so the identity is over `(database, schema, "")`, which stays distinct
from any single-table identity). The default-schema path — an empty
`materialized_postgresql_schema` or the explicit `public` — keeps the legacy
schema-unaware `<postgres_database>` publication/slot names, so existing
database-engine deployments keep their current PostgreSQL objects and `ATTACH`
still finds them. The `materialized_postgresql_schema_list` /
`materialized_postgresql_tables_list` filters are intentionally left
unchanged: keying the identity on them would rename every existing filtered
database-engine deployment's publication and slot and break its `ATTACH`.

Adds `test_two_schemas_same_table_name_database_engine`, which replicates the
same table name from two schemas of one PostgreSQL database into two standalone
database engines (no unique consumer identifier) and asserts that the initial
snapshot and ongoing replication stay isolated and that the two engines own
distinct publications and replication slots within PostgreSQL's identifier
limit.

Addresses the AI Review finding on
#107425

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread src/Storages/PostgreSQL/PostgreSQLReplicationHandler.cpp
…eployments

The schema-aware publication and default replication-slot names broke
backward compatibility for `MaterializedPostgreSQL` deployments created
before the identity became schema-aware: on restart / `ATTACH` the
handler looked for the schema-aware slot, missed the existing legacy
one, ran an initial sync that reloaded a snapshot into the
already-populated nested tables, and left the legacy slot orphaned on
the PostgreSQL side, retaining WAL forever. This affects the database
engine with a non-default `materialized_postgresql_schema` (supported
for years, restart-covered by
`test_2.py::test_database_with_single_non_default_schema`) and equally
the single-table engine with a non-default schema, which works since
26.6 (#107423 schema-qualified the single-table publication, and that
commit is in `v26.6.1.1193-stable`), so single-table deployments with
legacy names exist in the wild too.

A new `adoptLegacyReplicationIdentityIfNeeded` runs on attach, before
the publication is consulted (in `startSynchronization` and, for the
database engine, in `fetchRequiredTables`, which reads the publication
first): when the schema-aware objects do not exist but the legacy
schema-blind ones do, the handler switches to the legacy names and
keeps using them, including for `DROP` cleanup in `shutdownFinal`. The
slot carries the evidence (its loss is what triggers the destructive
re-sync); when the slot name does not depend on the schema (a
user-managed slot or a unique replication consumer identifier), the
publication carries it instead.

Adoption is fail-closed: the legacy names are schema-blind and can
belong to a coexisting engine over the default schema of the same
PostgreSQL database, so the legacy publication is adopted only when
every table it publishes belongs to this engine's schema; otherwise the
schema-aware identity is kept and the attach proceeds as a fresh setup
instead of hijacking another engine's slot and publication (two
consumers on one slot/publication would cross-talk, the very failure
the schema-aware identity removes).

Adds `test_legacy_identity_adopted_on_attach_table_engine` and
`test_legacy_identity_adopted_on_restart_database_engine`, which turn a
freshly created deployment into a legacy one on the PostgreSQL side
(while the table is detached / the server is stopped) and assert that
after `ATTACH`/restart the legacy identity is adopted: rows written
meanwhile arrive through the legacy slot, the schema-aware objects are
not recreated, and `DROP` cleans up the adopted objects. Adds
`test_legacy_identity_not_adopted_for_foreign_publication`: a
schema-scoped database that lost its schema-aware slot must not adopt
the legacy slot/publication of a coexisting default-schema database —
it re-syncs under its own identity and both databases stay isolated.

Addresses the AI review blocker on the database-engine identity change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alexey-milovidov

Copy link
Copy Markdown
Member Author

/continue-pr session summary

Addressed the AI Review ❌ Request changes blocker on PostgreSQLReplicationHandler.cpp:230 (the schema-aware database-engine identity broke ATTACH of existing non-default-schema deployments). Pushed c16f80c6e93..3b565c2c33f.

Fix: legacy-identity adoption on attach. A new adoptLegacyReplicationIdentityIfNeeded runs on attach — in startSynchronization and, for the database engine, in fetchRequiredTables (which consults the publication before startSynchronization runs). When the schema-aware slot/publication do not exist but the legacy schema-blind ones do, the handler adopts the legacy names and keeps using them, including for DROP cleanup in shutdownFinal: an upgraded deployment reuses its existing slot instead of re-snapshotting into the already-populated nested tables and orphaning the legacy slot (which would retain WAL forever). The slot carries the evidence; when the slot name does not depend on the schema (user-managed slot or unique replication consumer identifier), the publication carries it instead.

Scope: the table engine needs the same story. The blocker cited only the database engine, but #107423 (which made the single-table engine work with a non-default schema under the legacy names) is in v26.6.1.1193-stable, so legacy-named single-table deployments exist in the wild too. The adoption covers both engines uniformly.

Fail-closed against hijacking. The legacy names are schema-blind and can belong to a coexisting engine over the default schema of the same PostgreSQL database (<postgres_database> / <postgres_database>_ch_publication). The legacy publication is adopted only when every table it publishes (per pg_publication_tables) belongs to this engine's schema; otherwise the schema-aware identity is kept and the attach proceeds as a fresh setup rather than putting two consumers on one slot/publication.

Tests (all pass locally, release build, together with the four existing test_3.py schema-identity tests):

  • test_legacy_identity_adopted_on_attach_table_engine — detach, rewrite the PostgreSQL side into the legacy identity, attach: rows written meanwhile arrive through the adopted legacy slot, the schema-aware objects are not recreated, DROP cleans up the adopted objects.
  • test_legacy_identity_adopted_on_restart_database_engine — the same for the database engine across a server restart (the upgrade scenario), with catch-up through the legacy slot.
  • test_legacy_identity_not_adopted_for_foreign_publication — a schema-scoped database that lost its schema-aware slot does not adopt the coexisting default-schema database's legacy slot/publication; it re-syncs under its own identity and both databases stay isolated.

Also updated the PR description with the compatibility story. The previously green CI was for c16f80c6e93; a new run is now in flight for 3b565c2c33f.

Comment thread src/Storages/PostgreSQL/PostgreSQLReplicationHandler.cpp Outdated
Address the AI Review Block finding on the attach-time legacy-identity
adoption (#107425).

`adoptLegacyReplicationIdentityIfNeeded` refuses to adopt a legacy
publication that publishes a table from a different schema, because it
belongs to another engine and adopting it would make the two consumers
cross-talk. But it only logged and returned, and since the schema-aware
slot is gone at that point, `startSynchronization` then fell into
`initial_sync` and reloaded a snapshot into the already-existing nested
tables (`createNestedIfNeeded` is a no-op once they exist). That silently
re-snapshots a populated replica on restart, duplicating on-disk data and
turning the attach into a full reload. The `MaterializedPostgreSQL`
wrapper reads with `FINAL`, so the duplicate rows are hidden at read time,
but the extra parts are still written.

Per the fail-close principle, surface the error instead of performing a
consequential action on the fallback path: throw a
`POSTGRESQL_REPLICATION_INTERNAL_ERROR` so the initial sync (and any
re-snapshot) never runs and an operator can resolve the
slot/publication conflict. On both the table-engine attach path
(`checkConnectionAndStart`) and the database-engine startup path
(`tryStartSynchronization`) the throw is caught and logged, so the server
keeps running; replication simply does not start.

Rework `test_legacy_identity_not_adopted_for_foreign_publication`
accordingly: it now asserts the attach fails closed (the fail-closed
message is logged and the schema-aware slot is never recreated), which
proves the initial sync did not run - the earlier logical row-count check
went through the `FINAL` wrapper and could not have detected a duplicate
snapshot.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread src/Storages/PostgreSQL/PostgreSQLReplicationHandler.cpp Outdated
Follow-up to the fail-close on attach in `adoptLegacyReplicationIdentityIfNeeded`.

The legacy replication slot and publication names are schema-blind, so the mere
existence of the legacy slot does not prove the legacy objects belong to this
engine: a same-database deployment over the default schema (or another schema
targeting the same bare table) owns identically-named objects. When the
schema-aware slot is gone but the schema-blind legacy slot survives (`Branch A`),
the previous code only ran the ownership check inside
`if (publication_exists(legacy_publication_name))`. If the legacy publication was
missing or empty, the check was skipped, the engine adopted the ambiguous legacy
slot, and `createPublicationIfNeeded` recreated the legacy publication for this
engine's schema, hijacking another engine's replication identity instead of
failing closed.

Now the legacy identity is adopted only when the legacy publication exists and
every table it publishes belongs to this engine's schema. If it is missing, empty,
or publishes a table from another schema, ownership cannot be proven: the attach
fails closed with an exception and the legacy slot is left untouched.

Adds a regression test for the "foreign legacy slot present, publication missing"
case (`test_legacy_identity_not_adopted_when_publication_missing`).

Addresses the `clickhouse-gh` AI Review blocker at
`PostgreSQLReplicationHandler.cpp:457`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread src/Storages/PostgreSQL/PostgreSQLReplicationHandler.cpp
alexey-milovidov and others added 4 commits July 13, 2026 11:16
…able engine

The AI review of #107425 flagged that the new attach-time fail-close in
`adoptLegacyReplicationIdentityIfNeeded` (`POSTGRESQL_REPLICATION_INTERNAL_ERROR`) is
recoverable once an operator resolves the PostgreSQL-side replication-slot/publication
conflict, but the standalone table-engine attach path never retried it:
`checkConnectionAndStart` only rescheduled on `pqxx::broken_connection`, and the generic
`catch (...)` logged the exception once and returned. So a legacy table hitting this
branch on server start or `ATTACH TABLE` stayed permanently unsynchronized until another
restart or re-`ATTACH`, while the database engine kept retrying via
`DatabaseMaterializedPostgreSQL::tryStartSynchronization`.

Reschedule the standalone table-engine startup task for this recoverable error too, so
replication starts on its own after the operator resolves the conflict, without a restart
or a manual re-attach. The conflict is thrown before anything destructive runs, and each
retry re-checks ownership and refuses again while the conflict persists, so no re-snapshot
can happen in the meantime. The exception is only ever raised on attach, so the diagnostic
message is updated to state that startup keeps retrying and replication starts
automatically once the conflict is resolved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…il-close

Regression test for the retry gap flagged in review of #107425. A standalone
MaterializedPostgreSQL table with a non-default schema attaches into a recoverable
ownership conflict (schema-aware slot gone, an orphan schema-blind legacy slot present with
no legacy publication), fails closed, and the retrying startup task resumes replication on
its own once the operator restores the schema-aware slot — without a server restart or a
manual re-`ATTACH`. The restored slot is reused (no re-snapshot), and new rows written
after recovery reach the replica through it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ing created

`isReplicationSlotExist` read `restart_lsn` and `confirmed_flush_lsn` from
`pg_replication_slots` with `as<std::string>()` unconditionally. Both fields
are NULL while the slot is still being created (PostgreSQL registers the row
before assigning it a consistent snapshot point) and for a physical slot of
the same name, so the read threw `pqxx::conversion_error` - a
`std::logic_error` - and logging it from `checkConnectionAndStart` aborts
sanitizer builds. The attach-time retry loop made this easy to hit:
`test_table_engine_retries_recoverable_attach_conflict` probes the slot every
few hundred milliseconds and occasionally lands inside
`pg_create_logical_replication_slot`, which took the server down in 2 of 5
runs of the test in CI (Integration tests (amd_asan_ubsan, flaky) and
Integration tests (amd_asan_ubsan, db disk, old analyzer, 1/6)).

Throw a recoverable `POSTGRESQL_REPLICATION_INTERNAL_ERROR` instead: on the
attach path the startup task retries and a slot caught mid-creation is ready
by the next attempt, while the drop-only call sites already handle exceptions
and log that the slot must be dropped manually.

CI: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=107425&sha=5045eabee905afaccab15f35fb588d93c8472bba&name_0=PR&name_1=Integration%20tests%20%28amd_asan_ubsan%2C%20flaky%29
PR: #107425

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alexey-milovidov

Copy link
Copy Markdown
Member Author

/continue-pr session summary

The new test_table_engine_retries_recoverable_attach_conflict was flaky in CI (2 of 5 runs), and the artifacts show why: it exposed a real bug. isReplicationSlotExist reads restart_lsn and confirmed_flush_lsn from pg_replication_slots with as<std::string>() unconditionally, but both fields are NULL while a slot is still being created (PostgreSQL registers the row before assigning it a consistent snapshot point). The attach-time retry loop probes the slot every few hundred milliseconds, and a probe that landed inside the test's pg_create_logical_replication_slot threw pqxx::conversion_error — a std::logic_error — which aborts sanitizer builds when logged from checkConnectionAndStart, taking the server down mid-test (Attempt to convert SQL null to std::string, thread BgSchPool, isReplicationSlotExiststartSynchronization). The NULL read predates this PR; the retry loop just made it easy to hit.

Pushed 5045eabee90..6d1c0592c00:

  • Merged master (also pulls in Fix flaky 02346_text_index_bug108519_qcc_skip_index under randomized statistics #110260, which fixes the unrelated 02346_text_index_bug108519_qcc_skip_index flake behind most of the stateless reds).
  • isReplicationSlotExist now throws a recoverable POSTGRESQL_REPLICATION_INTERNAL_ERROR when the slot exists but its LSN fields are NULL: on the attach path the startup task retries and a slot caught mid-creation is ready by the next attempt; the drop-only call sites already handle exceptions.

Remaining reds on the previous head were: the 02346 flake (fixed by the merge), Scraping system tables minio timeouts (infra), and the Upgrade check flagging a DatabasePostgreSQL::getTablesIterator connection-timeout error message from the database that 04210_show_remote_databases_in_system_tables leaves behind — an allowlist gap in upgrade_runner.sh with a fix already open in #109350. CH Inc sync was red only because the private-side run hit the same two test failures fixed here.

alexey-milovidov and others added 2 commits July 13, 2026 21:54
`checkConnectionAndStart` only rescheduled the standalone table-engine
startup task on the attach path for `pqxx::broken_connection` and for an
`Exception` whose code is `POSTGRESQL_REPLICATION_INTERNAL_ERROR`. Any
other failure - in particular a `pqxx::sql_error`, which is not a
ClickHouse `Exception` and so lands in the generic `catch (...)` - was
logged and then silently dropped without rescheduling, so the retry loop
died and the attached table stayed permanently unsynchronized until a
server restart or a manual re-attach.

`test_table_engine_retries_recoverable_attach_conflict` hit exactly this:
after the operator restores the lost schema-aware replication slot, the
retrying startup task adopts it and constructs the consumer, but
`advanceLSN` occasionally throws `pqxx::sql_error` "replication slot ...
is active for PID N" because a just-released connection still holds the
slot for a moment. That threw out of `catch (...)` without rescheduling,
the retry loop stopped, and the rows written after the slot was restored
never arrived (`SELECT count()` stayed at 50 instead of reaching 100).

Reschedule on any error on the attach path, matching the database-engine
path, which retries on any exception via
`DatabaseMaterializedPostgreSQL::tryStartSynchronization`. Each retry
re-checks ownership and refuses again while a genuine conflict persists,
so no re-snapshot can happen in the meantime; a transient holder such as
"slot is active for PID N" is gone by the next attempt.

CI: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=107425&sha=5eb324caf32d543b93e2cd8b736558f514bebd3a&name_0=PR&name_1=Integration%20tests%20%28amd_asan_ubsan%2C%20flaky%29
PR: #107425

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@clickhouse-gh

clickhouse-gh Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

LLVM Coverage Report

Metric Baseline Current Δ
Lines 85.90% 85.80% -0.10%
Functions 92.70% 92.70% +0.00%
Branches 78.10% 78.00% -0.10%

Changed lines: Changed C/C++ lines covered: 97/128 (75.78%) · Uncovered code

Full report · Diff report

@alexey-milovidov alexey-milovidov self-assigned this Jul 15, 2026

@alexey-milovidov alexey-milovidov left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alexey-milovidov
alexey-milovidov merged commit c59099f into master Jul 15, 2026
176 of 177 checks passed
@alexey-milovidov
alexey-milovidov deleted the fix-matpg-table-engine-schema branch July 15, 2026 01:17
@robot-ch-test-poll3 robot-ch-test-poll3 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-bugfix Pull request with bugfix, not backported by default pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MaterializedPostgreSQL table engine does not support schema specification

3 participants