Tests for MySQL Parallel snapshot#37819
Draft
peterdukelarsen wants to merge 6 commits into
Draft
Conversation
A designated leader worker locks all tables, computes PK-range split boundaries, and broadcasts them (with the GTID frontier) to all workers via a timely feedback loop. Each worker then reads only its assigned half-open PK range in parallel, so the snapshot of a large table scales with worker count. Single-column integer/char PKs are supported; other PKs (composite, unusual types) fall back to single-worker-per-table snapshotting. Builds on the lock/GTID helper refactor: the leader reuses lock_tables_and_read_gtid_set. Re-adds the ANALYZE calls and the stale-count test dropped from the tests commit, and bumps the MySQL container max_connections to accommodate the extra parallel connections. The leader opens its lock connection only after boundary sampling completes, so it no longer overlaps with the sampling probe connections and inflates the peak connection count. Also raises the per-source connection budget in the limits test MySqlSources scenario. The parallel path holds more upstream connections per source (a lock connection, up to two parallel snapshot reads, and a replication connection), so the scenario's SET GLOBAL max_connections is bumped from COUNT*2+2 to COUNT*4+2. Without this the scenario starves for connections and stalls. Raises the MySqlInitialLoad feature-benchmark MEMORY_MZ regression threshold from 20% to 30%, since the parallel snapshot path has a wider memory envelope during the initial load. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A BIT column maps to the UInt64 scalar type but carries a MySqlColumnMeta::Bit marker. The PK-range sampler rendered its split boundary from the scalar type alone, taking the integer path CAST(id AS CHAR). For a BIT column that returns the value's raw bytes, not a decimal literal, and the text was spliced unquoted into the worker range predicates. A crafted BIT value could then inject SQL so every worker read the whole table, duplicating rows. Exclude PK columns that carry a column-meta marker from range splitting, so they fall back to a single-worker whole-table read. Also validate that integer-path boundary literals are decimal before using them, guarding against any other type whose CAST-to-text is not a plain integer.
verify_schemas reads information_schema, which is not part of the repeatable-read snapshot, so the per-worker check after the lock was released compared each output against a live schema that need not match the snapshot the worker actually read. It also ran after PK sampling, so a primary-key column renamed or dropped upstream failed the sampling probe with a transient error that re-ran before verification every restart, wedging the whole source in a loop. Verify on the leader before sampling and locking. Outputs whose schema drifted are dropped and excluded from sampling, so a renamed or dropped column surfaces as a per-output definite error while the rest of the source keeps ingesting, with no retry loop. Each worker still re-verifies in its read transaction, but any incompatibility found there now means the schema moved after the leader's check, so it returns a transient error and the snapshot retries from a consistent state. The PK-boundary collation check moves off the leader's lock path the same way. Instead of the leader validating monotonicity under the lock and downgrading drifted tables to whole-table reads, each worker validates its ranged tables in its read transaction and fails transiently on drift, so the retry re-samples under the new collation. The leader's critical section shrinks to lock, read GTID, broadcast. DDL after the worker checks is caught by the reads failing with ER_TABLE_DEF_CHANGED.
Drop a redundant modulo in worker_pk_range that read like an off-by-precedence bug, remove an unnecessary loop label, fix a typo and a contradictory comment about the COUNT(*) fallback, and collapse a double blank line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add the mysql_source_snapshot_parallelism dyncfg, enabled by default, as a kill switch for parallel PK-range snapshotting. When disabled the leader skips boundary sampling, so every table takes the existing single-worker whole-table fallback and the read-phase upstream connection footprint drops back to one connection per responsible worker. The leader-based lock/broadcast protocol is unaffected. Registering the dyncfg automatically exposes it as a system var, so it is settable via ALTER SYSTEM SET and manageable via LaunchDarkly. Lists the flag in the test-infra parameter tables so CI can vary it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up tests for the parallel PK-range snapshot work: * A BIT(64) primary key whose value decodes to the SQL fragment "0 OR 1=1", covering the literal-injection fix that excludes column-meta-marked PKs from range splitting. * A primary-key column renamed upstream between source planning and the initial snapshot, covering leader-side schema verification: the renamed table surfaces a per-output definite error while other tables keep ingesting. * A snapshot with the mysql_source_snapshot_parallelism dyncfg disabled, covering the single-worker whole-table fallback path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
def-
approved these changes
Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remove these sections if your commit already has a good description!
Motivation
Why does this change exist? Link to a GitHub issue, design doc, Slack
thread, or explain the problem in a sentence or two. A reviewer who has
no context should understand why after reading this section.
If this implements or addresses an existing issue, it's enough to link to that:
Closes
Fixes
etc.
Description
What does this PR actually do? Focus on the approach and any non-obvious
decisions. The diff shows the code --- use this space to explain what the
diff can't tell a reviewer.
Verification
How do you know this change is correct? Describe new or existing automated
tests, or manual steps you took.