Honor materialized_postgresql_schema in MaterializedPostgreSQL table engine#107425
Conversation
…engine The standalone `MaterializedPostgreSQL` table engine ignored the `materialized_postgresql_schema` setting. Unlike the database engine, which schema-qualifies and quotes the tables list in `fetchRequiredTables`, the table engine puts the bare remote table name into `tables_list` (in the storage constructor) and never processes it. As a result `createPublicationIfNeeded` issued `CREATE PUBLICATION ... FOR TABLE ONLY <table>` without the schema, failing with `relation "<table>" does not exist` whenever the table lived in a non-default schema. Build the publication's tables list from the storages (which applies the schema via `doubleQuoteWithSchema`) for the table engine, mirroring the database engine behaviour. The rest of the flow (structure fetch, snapshot COPY, consumer mapping) already used the schema correctly. Closes: #59950 Related: #49045 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Workflow [PR], commit [1595e91] Summary: ✅
AI ReviewSummaryThis PR makes Final Verdict
|
The default replication slot name is `<postgres_database>_<table>_ch_replication_slot`, which is checked against PostgreSQL's 63-character identifier limit in `checkReplicationSlot` at handler construction. With the previous names the slot name `postgres_database_test_table_engine_schema_table_ch_replication_slot` was 68 characters, so the test aborted with `Too big replication slot size` (a `LOGICAL_ERROR`) before the `materialized_postgresql_schema` fix was ever exercised. Under sanitizer builds this aborted the server, cascading into `Connection refused` failures across the other integration test shards. Shorten the table name so the slot name stays within the limit and the test actually validates the schema handling end-to-end. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI triageIntegration tests (6 shards: The two remaining reds are unrelated to this change (a one-line PostgreSQL publication change):
@groeneai, could you take a look at the |
|
@alexey-milovidov The Root cause is reentrancy of the sanitizer death callback. At process exit the static The fix marks |
…ine-schema # Conflicts: # tests/integration/test_postgresql_replica_database_engine/test_3.py
…aware After merging master, the original publication tables-list change from this PR became redundant: master's #107423 ("Fix MaterializedPostgreSQL replication for upper-case database/table names", commit `190a65c1556`) added an `else if (!is_materialized_postgresql_database)` branch in `createPublicationIfNeeded` that quotes the single-table `tables_list` via `doubleQuoteWithSchema`, which already applies the `materialized_postgresql_schema` setting. So the `CREATE PUBLICATION ... FOR TABLE ONLY` query is schema-qualified on master already. Revert that part here so master's branch is no longer dead code. What was still missing — and what this commit fixes — is the replication *identity*. `getPublicationName` and the default `getReplicationSlotName` derived the name from `<postgres_database>_<table>` only, ignoring the schema. Two standalone `MaterializedPostgreSQL` tables replicating a table with the same name from two different PostgreSQL schemas of the same database therefore shared one publication and one default replication slot. The second `CREATE` dropped and recreated the shared publication, and because the publication carries only the bare relation name (`schema_as_a_part_of_table_name` stays false in this mode) the consumers cross-talked: one replica could stop receiving its schema's changes or ingest the other schema's rows. Include the schema in the publication name and the default replication slot name for the single-table engine when `materialized_postgresql_schema` is set, so the two configurations get distinct PostgreSQL objects and stay isolated. The database engine path (empty remote table name) and the default-schema table-engine path are unchanged, so existing deployments keep their current publication and slot names. Add an integration test `test_two_schemas_same_table_name_single_storage` that replicates the same table name from two schemas and asserts both the initial snapshot and ongoing replication stay isolated. Shorten the schema and table names in `test_single_table_engine_with_non_default_schema` so the now schema-aware default slot name stays within PostgreSQL's identifier length limit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
/continue-pr session summaryMerged Key finding from the merge. Fix for the AI Block (the genuinely-remaining issue). Tests.
The unresolved AI thread on Verification. CI triage. The only red on the previous head was For the author: the basic |
…chema Making the standalone `MaterializedPostgreSQL` table-engine replication identity schema-aware (in the previous commit) changed the generated default publication and replication-slot names for *every* non-empty `materialized_postgresql_schema`, including the common explicit-default case `materialized_postgresql_schema = 'public'`. That breaks `ATTACH` of an existing standalone `MaterializedPostgreSQL` table created before the identity became schema-aware: the default slot was `postgres_database_<table>_ch_replication_slot`, but the new code looks for `postgres_database_<schema>_<table>_ch_replication_slot`. When that slot is absent, `startSynchronization` runs the initial sync, creates a new slot, and `loadFromSnapshot` inserts a fresh snapshot into the already-existing nested table (`createNestedIfNeeded` is a no-op when the nested table exists), duplicating data and leaving the old slot and publication behind. Treat PostgreSQL's default schema (`public`, or an empty setting) the same as no schema in `getPublicationName` and `getReplicationSlotName`: only a genuinely non-default schema is included in the generated names. This preserves the legacy identity for the default schema (so `ATTACH` keeps finding the existing objects) while still keeping the collision fix intact, because `public.t` and `schema1.t` still get distinct names (`postgres_database_t` vs `postgres_database_schema1_t`). Adds `test_default_schema_preserves_legacy_identity`, which creates the single-table engine with `materialized_postgresql_schema = 'public'` and asserts that the publication and replication slot use the legacy schema-unaware names, not the schema-aware ones. Addresses the AI review on #107425 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ine-schema Master's "Enable more Ruff checks (F401/F403/F405/F541/F841) and clean up tests" (`d6c076decf7`) rewrote the `from helpers.postgres_utility import (...)` block of `test_3.py`, dropping every name that master's own copy of the file did not use. The auto-merge took master's reduced import block, so the three helpers this PR's new tests call as bare functions — `create_postgres_schema`, `create_postgres_table`, and `create_postgres_table_with_schema` — were no longer imported, and `ruff` flagged them as `F821` undefined-name on the CI merge ref (the Style check failure on `f353f743778`). Re-add the three imports (they still exist in `postgres_utility.py`); `ruff check` now passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ine-schema # Conflicts: # tests/integration/test_postgresql_replica_database_engine/test_3.py
…tant The single-table MaterializedPostgreSQL engine derived its publication and default replication-slot names from a plain `postgres_database_postgres_schema_postgres_table` concatenation when `materialized_postgresql_schema` was set to a non-default schema. That is not injective: `schema = a_b`, `table = c` and `schema = a`, `table = b_c` both yield `postgres_database_a_b_c_*`, and the replication slot name is additionally folded by `normalizeReplicationSlot` (lower-cased, `-` mapped to `_`), so PostgreSQL-distinct schemas such as `Foo`/`foo` or `a-b`/`a_b` map to one slot. Two standalone tables built from such identities would then share a publication or a replication slot and their consumers would cross-talk — the very failure the schema-aware identity is meant to remove. Derive the schema-aware single-table identity from a bounded, collision-resistant hash of a length-prefixed (hence unambiguous) serialization of the full (database, schema, table) triple. The generated name is now injective in practice, fixed-length (so it stays within PostgreSQL's identifier limit regardless of the schema/table length) and within the replication-slot character set. The database-engine path and the default-schema (empty or `public`) table-engine path keep their legacy names unchanged. Adds focused integration coverage for both collision classes: `a_b.c` versus `a.b_c` (publication/slot separator collision) and `a-b.t` versus `a_b.t` (slot hyphen-folding collision), asserting that the two engines own distinct PostgreSQL objects and stay isolated. Addresses the AI Review "Request changes" findings on #107425 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
/continue-pr session summaryNo code changes were needed — the PR is approved and has no unresolved review threads.
|
…ase names The schema-aware single-table publication and default replication-slot identities folded the schema and table into a fixed-length hash but kept the PostgreSQL database name as a full prefix, so the generated names were not actually bounded. With a moderately long database name (>= 28 characters) the default slot `<database>_<16 hex hash>_ch_replication_slot` exceeded PostgreSQL's 63-character identifier limit, and `checkReplicationSlot` threw `Too big replication slot size` in the constructor before the table could be created — a regression for non-default-schema tables that worked with a short, schema-blind slot name. The same unbounded prefix was used by `getPublicationName`. Cap the cosmetic, human-readable database prefix at 16 bytes in the new `getSchemaAwareIdentityName` helper, which both `getPublicationName` and the default `getReplicationSlotName` now use. The full `(database, schema, table)` identity is still carried by the hash, so capping the prefix cannot reintroduce a collision, and the generated names (slot, its `_tmp` variant, and the publication) stay within the limit regardless of the database name length. Add `test_schema_aware_identity_long_database_name`, which replicates a non-default-schema table under a 39-character PostgreSQL database name and asserts the initial snapshot and ongoing replication succeed and that the generated publication/slot names stay within the 63-character limit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
/continue-pr session summaryAddressed the AI Review Fix. The schema-aware single-table identity folded the schema and table into a fixed-length hash but kept the full PostgreSQL database name as a prefix, so the generated names were not actually bounded. With a database name of ≥ 28 characters the default slot A new Test. Added Verification. CI triage (previous head, all reds unrelated to a PostgreSQL identity-string change):
The new commit re-triggers full CI. |
/continue-pr session summaryNo code changes were needed — the PR is approved with no unresolved review threads.
|
/continue-pr session summaryNo code changes — the PR is correct, AI Review verdict is No re-merge. The branch merged CI triage (this PR touches only
The failed jobs cannot be re-run while the "PR" workflow run is still in progress; they will be re-run once it finishes if still needed. Otherwise the PR is ready to merge. |
…ase engine Follow-up to the schema-aware single-table identity: extend it to the `MaterializedPostgreSQL` database engine when it replicates a single common non-default schema (`materialized_postgresql_schema`). Previously `getPublicationName` and the default `getReplicationSlotName` derived the database-engine identity from `<postgres_database>` alone, ignoring `materialized_postgresql_schema`. Two `CREATE DATABASE ... ENGINE = MaterializedPostgreSQL` statements over the same PostgreSQL database but different non-default schemas therefore shared one publication (`<postgres_database>_ch_publication`) and, without a unique replication consumer identifier, one default slot (`<postgres_database>`). The second `CREATE` dropped and recreated the shared publication for its own schema's tables, so the first consumer stopped receiving its schema's changes; and because single-schema mode keeps `schema_as_a_part_of_table_name` false, a same-named table in the other schema could be ingested by the wrong consumer. The fix reuses the existing collision-resistant, length-bounded `getSchemaAwareIdentityName` (the remote table name is empty for the database engine, so the identity is over `(database, schema, "")`, which stays distinct from any single-table identity). The default-schema path — an empty `materialized_postgresql_schema` or the explicit `public` — keeps the legacy schema-unaware `<postgres_database>` publication/slot names, so existing database-engine deployments keep their current PostgreSQL objects and `ATTACH` still finds them. The `materialized_postgresql_schema_list` / `materialized_postgresql_tables_list` filters are intentionally left unchanged: keying the identity on them would rename every existing filtered database-engine deployment's publication and slot and break its `ATTACH`. Adds `test_two_schemas_same_table_name_database_engine`, which replicates the same table name from two schemas of one PostgreSQL database into two standalone database engines (no unique consumer identifier) and asserts that the initial snapshot and ongoing replication stay isolated and that the two engines own distinct publications and replication slots within PostgreSQL's identifier limit. Addresses the AI Review finding on #107425 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eployments The schema-aware publication and default replication-slot names broke backward compatibility for `MaterializedPostgreSQL` deployments created before the identity became schema-aware: on restart / `ATTACH` the handler looked for the schema-aware slot, missed the existing legacy one, ran an initial sync that reloaded a snapshot into the already-populated nested tables, and left the legacy slot orphaned on the PostgreSQL side, retaining WAL forever. This affects the database engine with a non-default `materialized_postgresql_schema` (supported for years, restart-covered by `test_2.py::test_database_with_single_non_default_schema`) and equally the single-table engine with a non-default schema, which works since 26.6 (#107423 schema-qualified the single-table publication, and that commit is in `v26.6.1.1193-stable`), so single-table deployments with legacy names exist in the wild too. A new `adoptLegacyReplicationIdentityIfNeeded` runs on attach, before the publication is consulted (in `startSynchronization` and, for the database engine, in `fetchRequiredTables`, which reads the publication first): when the schema-aware objects do not exist but the legacy schema-blind ones do, the handler switches to the legacy names and keeps using them, including for `DROP` cleanup in `shutdownFinal`. The slot carries the evidence (its loss is what triggers the destructive re-sync); when the slot name does not depend on the schema (a user-managed slot or a unique replication consumer identifier), the publication carries it instead. Adoption is fail-closed: the legacy names are schema-blind and can belong to a coexisting engine over the default schema of the same PostgreSQL database, so the legacy publication is adopted only when every table it publishes belongs to this engine's schema; otherwise the schema-aware identity is kept and the attach proceeds as a fresh setup instead of hijacking another engine's slot and publication (two consumers on one slot/publication would cross-talk, the very failure the schema-aware identity removes). Adds `test_legacy_identity_adopted_on_attach_table_engine` and `test_legacy_identity_adopted_on_restart_database_engine`, which turn a freshly created deployment into a legacy one on the PostgreSQL side (while the table is detached / the server is stopped) and assert that after `ATTACH`/restart the legacy identity is adopted: rows written meanwhile arrive through the legacy slot, the schema-aware objects are not recreated, and `DROP` cleans up the adopted objects. Adds `test_legacy_identity_not_adopted_for_foreign_publication`: a schema-scoped database that lost its schema-aware slot must not adopt the legacy slot/publication of a coexisting default-schema database — it re-syncs under its own identity and both databases stay isolated. Addresses the AI review blocker on the database-engine identity change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/continue-pr session summaryAddressed the AI Review ❌ Request changes blocker on Fix: legacy-identity adoption on attach. A new Scope: the table engine needs the same story. The blocker cited only the database engine, but #107423 (which made the single-table engine work with a non-default schema under the legacy names) is in Fail-closed against hijacking. The legacy names are schema-blind and can belong to a coexisting engine over the default schema of the same PostgreSQL database ( Tests (all pass locally, release build, together with the four existing
Also updated the PR description with the compatibility story. The previously green CI was for |
Address the AI Review Block finding on the attach-time legacy-identity adoption (#107425). `adoptLegacyReplicationIdentityIfNeeded` refuses to adopt a legacy publication that publishes a table from a different schema, because it belongs to another engine and adopting it would make the two consumers cross-talk. But it only logged and returned, and since the schema-aware slot is gone at that point, `startSynchronization` then fell into `initial_sync` and reloaded a snapshot into the already-existing nested tables (`createNestedIfNeeded` is a no-op once they exist). That silently re-snapshots a populated replica on restart, duplicating on-disk data and turning the attach into a full reload. The `MaterializedPostgreSQL` wrapper reads with `FINAL`, so the duplicate rows are hidden at read time, but the extra parts are still written. Per the fail-close principle, surface the error instead of performing a consequential action on the fallback path: throw a `POSTGRESQL_REPLICATION_INTERNAL_ERROR` so the initial sync (and any re-snapshot) never runs and an operator can resolve the slot/publication conflict. On both the table-engine attach path (`checkConnectionAndStart`) and the database-engine startup path (`tryStartSynchronization`) the throw is caught and logged, so the server keeps running; replication simply does not start. Rework `test_legacy_identity_not_adopted_for_foreign_publication` accordingly: it now asserts the attach fails closed (the fail-closed message is logged and the schema-aware slot is never recreated), which proves the initial sync did not run - the earlier logical row-count check went through the `FINAL` wrapper and could not have detected a duplicate snapshot. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the fail-close on attach in `adoptLegacyReplicationIdentityIfNeeded`. The legacy replication slot and publication names are schema-blind, so the mere existence of the legacy slot does not prove the legacy objects belong to this engine: a same-database deployment over the default schema (or another schema targeting the same bare table) owns identically-named objects. When the schema-aware slot is gone but the schema-blind legacy slot survives (`Branch A`), the previous code only ran the ownership check inside `if (publication_exists(legacy_publication_name))`. If the legacy publication was missing or empty, the check was skipped, the engine adopted the ambiguous legacy slot, and `createPublicationIfNeeded` recreated the legacy publication for this engine's schema, hijacking another engine's replication identity instead of failing closed. Now the legacy identity is adopted only when the legacy publication exists and every table it publishes belongs to this engine's schema. If it is missing, empty, or publishes a table from another schema, ownership cannot be proven: the attach fails closed with an exception and the legacy slot is left untouched. Adds a regression test for the "foreign legacy slot present, publication missing" case (`test_legacy_identity_not_adopted_when_publication_missing`). Addresses the `clickhouse-gh` AI Review blocker at `PostgreSQLReplicationHandler.cpp:457`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…able engine The AI review of #107425 flagged that the new attach-time fail-close in `adoptLegacyReplicationIdentityIfNeeded` (`POSTGRESQL_REPLICATION_INTERNAL_ERROR`) is recoverable once an operator resolves the PostgreSQL-side replication-slot/publication conflict, but the standalone table-engine attach path never retried it: `checkConnectionAndStart` only rescheduled on `pqxx::broken_connection`, and the generic `catch (...)` logged the exception once and returned. So a legacy table hitting this branch on server start or `ATTACH TABLE` stayed permanently unsynchronized until another restart or re-`ATTACH`, while the database engine kept retrying via `DatabaseMaterializedPostgreSQL::tryStartSynchronization`. Reschedule the standalone table-engine startup task for this recoverable error too, so replication starts on its own after the operator resolves the conflict, without a restart or a manual re-attach. The conflict is thrown before anything destructive runs, and each retry re-checks ownership and refuses again while the conflict persists, so no re-snapshot can happen in the meantime. The exception is only ever raised on attach, so the diagnostic message is updated to state that startup keeps retrying and replication starts automatically once the conflict is resolved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…il-close Regression test for the retry gap flagged in review of #107425. A standalone MaterializedPostgreSQL table with a non-default schema attaches into a recoverable ownership conflict (schema-aware slot gone, an orphan schema-blind legacy slot present with no legacy publication), fails closed, and the retrying startup task resumes replication on its own once the operator restores the schema-aware slot — without a server restart or a manual re-`ATTACH`. The restored slot is reused (no re-snapshot), and new rows written after recovery reach the replica through it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ing created `isReplicationSlotExist` read `restart_lsn` and `confirmed_flush_lsn` from `pg_replication_slots` with `as<std::string>()` unconditionally. Both fields are NULL while the slot is still being created (PostgreSQL registers the row before assigning it a consistent snapshot point) and for a physical slot of the same name, so the read threw `pqxx::conversion_error` - a `std::logic_error` - and logging it from `checkConnectionAndStart` aborts sanitizer builds. The attach-time retry loop made this easy to hit: `test_table_engine_retries_recoverable_attach_conflict` probes the slot every few hundred milliseconds and occasionally lands inside `pg_create_logical_replication_slot`, which took the server down in 2 of 5 runs of the test in CI (Integration tests (amd_asan_ubsan, flaky) and Integration tests (amd_asan_ubsan, db disk, old analyzer, 1/6)). Throw a recoverable `POSTGRESQL_REPLICATION_INTERNAL_ERROR` instead: on the attach path the startup task retries and a slot caught mid-creation is ready by the next attempt, while the drop-only call sites already handle exceptions and log that the slot must be dropped manually. CI: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=107425&sha=5045eabee905afaccab15f35fb588d93c8472bba&name_0=PR&name_1=Integration%20tests%20%28amd_asan_ubsan%2C%20flaky%29 PR: #107425 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/continue-pr session summaryThe new Pushed
Remaining reds on the previous head were: the |
`checkConnectionAndStart` only rescheduled the standalone table-engine startup task on the attach path for `pqxx::broken_connection` and for an `Exception` whose code is `POSTGRESQL_REPLICATION_INTERNAL_ERROR`. Any other failure - in particular a `pqxx::sql_error`, which is not a ClickHouse `Exception` and so lands in the generic `catch (...)` - was logged and then silently dropped without rescheduling, so the retry loop died and the attached table stayed permanently unsynchronized until a server restart or a manual re-attach. `test_table_engine_retries_recoverable_attach_conflict` hit exactly this: after the operator restores the lost schema-aware replication slot, the retrying startup task adopts it and constructs the consumer, but `advanceLSN` occasionally throws `pqxx::sql_error` "replication slot ... is active for PID N" because a just-released connection still holds the slot for a moment. That threw out of `catch (...)` without rescheduling, the retry loop stopped, and the rows written after the slot was restored never arrived (`SELECT count()` stayed at 50 instead of reaching 100). Reschedule on any error on the attach path, matching the database-engine path, which retries on any exception via `DatabaseMaterializedPostgreSQL::tryStartSynchronization`. Each retry re-checks ownership and refuses again while a genuine conflict persists, so no re-snapshot can happen in the meantime; a transient holder such as "slot is active for PID N" is gone by the next attempt. CI: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=107425&sha=5eb324caf32d543b93e2cd8b736558f514bebd3a&name_0=PR&name_1=Integration%20tests%20%28amd_asan_ubsan%2C%20flaky%29 PR: #107425 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LLVM Coverage Report
Changed lines: Changed C/C++ lines covered: 97/128 (75.78%) · Uncovered code |
The standalone
MaterializedPostgreSQLtable engine did not work for a relation that lives in a non-default PostgreSQL schema (selected via thematerialized_postgresql_schemasetting).Update after merging
master. The original failure of this PR —CREATE PUBLICATION ... FOR TABLE ONLY <table>issued without the schema, sopqxx::undefined_table: ERROR: relation "<table>" does not exist— is now fixed onmasterby #107423 (commit190a65c1556). That change quotes the single-table publication's tables list viadoubleQuoteWithSchema, which also schema-qualifies it. The redundant publication tables-list change originally made here has therefore been reverted, leavingcreatePublicationIfNeededidentical tomaster.What this PR fixes now: the replication identity.
getPublicationNameand the defaultgetReplicationSlotNamederived their names from<postgres_database>_<table>only, ignoring the schema. Two standaloneMaterializedPostgreSQLtables that replicate a table with the same name from two different schemas of the same PostgreSQL database therefore shared one publication and one default replication slot. The secondCREATEdropped and recreated the shared publication, and because the publication carried only the bare relation name (schema_as_a_part_of_table_namestaysfalsein this mode), the two consumers cross-talked — one replica could stop receiving its schema's changes or ingest the other schema's rows.Fix. For the single-table engine, derive both the publication name and the default replication slot name from a bounded, collision-resistant hash of the full
(database, schema, table)identity whenmaterialized_postgresql_schemais set to a non-default schema, so the two configurations get distinct PostgreSQL objects and stay isolated. A plaindatabase_schema_tableconcatenation is not injective (schema = a_b,table = candschema = a,table = b_cboth fold todatabase_a_b_c), and the replication-slot name is additionally lower-cased with-mapped to_, so even PostgreSQL-distinct schemas such asFoo/fooora-b/a_bwould otherwise collide. Hashing a length-prefixed (hence unambiguous) serialization of the identity keeps the generated name injective in practice and fixed-length, and the human-readable database prefix kept in front of the hash (purely for recognizability inpg_replication_slots/pg_publication) is itself capped, so the slot stays within PostgreSQL's identifier limit regardless of the database, schema and table length. The default schema is left out of the identity: an emptymaterialized_postgresql_schema, or the explicit defaultmaterialized_postgresql_schema = 'public', keeps the legacy schema-unaware names, so existing single-table deployments keep their current publication and slot names andATTACHstill finds them.Database engine. The same schema-blind identity affected the
MaterializedPostgreSQLdatabase engine (empty remote table name):getPublicationNameand the defaultgetReplicationSlotNamederived the name from<postgres_database>alone, so twoCREATE DATABASE ... ENGINE = MaterializedPostgreSQLstatements over the same PostgreSQL database but different non-defaultmaterialized_postgresql_schemashared one publication (and, without a unique replication consumer identifier, one default slot) and cross-talked in the same way. The fix reuses the same collision-resistant, length-bounded identity for the database engine when it replicates a single common non-default schema — the remote table name is empty, so the identity is over(database, schema, ""), which stays distinct from any single-table identity. The default-schema database-engine path keeps the legacy<postgres_database>names, and existing non-default-schema deployments are covered by the legacy-identity adoption described below. Thematerialized_postgresql_schema_list/materialized_postgresql_tables_listfilters are intentionally left unchanged: keying the identity on them would rename the publication and slot of every existing filtered database-engine deployment and break itsATTACH(a re-sync would reload a snapshot into the already-existing nested tables, duplicating data), so hardening those filters needs a separate compatibility/migration story.Backward compatibility: legacy-identity adoption on
ATTACH. Deployments created before the identity became schema-aware already exist for both modes: the database engine's single-schema mode is years old (restart-covered bytest_2.py::test_database_with_single_non_default_schema), and the single-table engine works with a non-default schema since 26.6 (#107423 is inv26.6.1.1193-stable). They own the legacy, schema-blind slot and publication, so switching the generated names alone would break them on upgrade: restart /ATTACHwould look for the schema-aware slot, miss the existing one, run an initial sync that reloads a snapshot into the already-populated nested tables, and leave the legacy slot orphaned on the PostgreSQL side, retaining WAL forever. A newadoptLegacyReplicationIdentityIfNeededtherefore runs on attach (instartSynchronizationand, for the database engine, infetchRequiredTables, which consults the publication first): when the schema-aware objects do not exist but the legacy ones do, the handler adopts the legacy names and keeps using them, including forDROPcleanup. The slot carries the evidence (its loss is what triggers the destructive re-sync); for slot names that do not depend on the schema (a user-managed slot or a unique replication consumer identifier), the publication carries it instead. Adoption is fail-closed: the legacy names are schema-blind and can belong to a coexisting engine over the default schema of the same PostgreSQL database, so the legacy publication is adopted only when every table it publishes belongs to this engine's schema; otherwise the schema-aware identity is kept and the attach proceeds as a fresh setup instead of hijacking another engine's slot and publication.Tests. Adds
test_two_schemas_same_table_name_single_storage, which replicates the same table name from two schemas into two standalone engines and asserts that both the initial snapshot and ongoing replication stay isolated. Addstest_two_schemas_same_table_name_database_engine, the database-engine counterpart: it replicates the same table name from two schemas of one PostgreSQL database into two standalone database engines (no unique consumer identifier, so isolation comes solely from the schema-aware naming) and asserts that the initial snapshot and ongoing replication stay isolated and that the two engines own distinct publications and replication slots within the 63-character limit. Addstest_schema_aware_identity_publication_separator_collision(a_b.cversusa.b_c) andtest_schema_aware_identity_slot_hyphen_distinct(a-b.tversusa_b.t), which assert that the formerly-colliding identities now own distinct publications and replication slots and replicate in isolation. Keepstest_single_table_engine_with_non_default_schema(single non-default schema). Addstest_default_schema_preserves_legacy_identity, which creates the single-table engine with the explicit defaultmaterialized_postgresql_schema = 'public'and asserts (viapg_replication_slotsandpg_publication) that the legacy schema-unaware publication and slot names are used, soATTACHof tables created before the identity became schema-aware keeps working. Addstest_schema_aware_identity_long_database_name, which replicates a non-default-schema table under a long PostgreSQL database name (whose unbounded slot name would have exceeded the identifier limit) and asserts that the initial snapshot and ongoing replication succeed and that the generated publication and slot names stay within PostgreSQL's 63-character limit. Addstest_legacy_identity_adopted_on_attach_table_engineandtest_legacy_identity_adopted_on_restart_database_engine, which turn a freshly created deployment into a legacy one on the PostgreSQL side (while the table is detached / the server is stopped) and assert that afterATTACH/restart the legacy identity is adopted: rows written meanwhile arrive through the legacy slot, the schema-aware objects are not recreated, andDROPcleans up the adopted objects. Addstest_legacy_identity_not_adopted_for_foreign_publication: a schema-scoped database that lost its schema-aware slot must not adopt the legacy slot/publication of a coexisting default-schema database — it re-syncs under its own identity and both databases stay isolated.Closes: #59950
Related: #49045
Related: #107423
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
Fix the
MaterializedPostgreSQLtable and database engines so that a single table or database can be replicated from a non-default PostgreSQL schema (materialized_postgresql_schema), including the case where tables with the same name exist in several schemas of the same database (previously they would share a publication and replication slot and cross-talk).Documentation entry for user-facing changes
🤖 Generated with Claude Code
Version info
26.7.1.971(included in26.7and later)