Repository navigation
Conversation
erezrokah
self-requested a review
October 14, 2026 09:39
Member
|
Can we flush pending checkpoints while the source is idle and split large checkpoint batches to stay within the gRPC message limit? |
Member
Author
|
@erezrokah Both done in 118eac4:
Checkpoint messages from the source are not split yet: a single |
- Validate checkpoint column types instead of panicking on an unexpected version type - Report a pipeline's error instead of a cancelled commit when a pipeline fails during a commit - Name the destination whose flush failed - Log when the source sends checkpoints, instead of claiming they are enabled up front - Test the checkpoint flow through syncConnectionV3 with fake plugins, and the switch that enables it
Member
|
Could we apply |
Member
Author
|
@erezrokah Done in 30ee29e: the receive-error path now goes through the same check (renamed to |
erezrokah
approved these changes
Oct 14, 2026
- Build and send state records one at a time, and count every column in the record size estimate - Report a duplicate state column as such, and route receive errors through receiveError - Test that a flush waits for pipeline delivery, idle commits through syncConnectionV3, backend errors across several records, and prefix matching of the state backend
- Flush every destination that has not failed yet at the end of the sync, so later failures skip delete-stale too - Keep syncing other sources when a sync finishes without saving its state, and report every such source at the end - Treat a failed state backend flush like a failed destination flush - Report destination commit failures together with hard flush errors - Prefer the pipelines' errors over the receive loop's instead of filtering each loop error - Word the error differently for sources that sent no checkpoints - Warn about skipped delete-stale only for destinations that run it - Bump plugin-pb-go to the regenerated flush_error wording
- Skip delete-stale only for the tables a destination failed to write, when it names them - Ignore the end of a write stream after the destination reported a failed flush, and close every destination before returning - Keep the unsaved state errors of earlier sources when a later source fails - Pin the preference of pipeline errors over the receive loop's in a helper with a deterministic test - Warn append mode destinations that the next sync appends the unsaved rows again - Stop UpdateTableNames from removing tables from its input, which skipped delete-stale on every destination after the first - Bump plugin-pb-go to the failed_tables field
- Flush destinations again at the end of the sync after they failed, so delete-stale skips the tables they fail later too - Add stream errors after a failure to the recorded failure, as records of any table may then be lost - Skip delete-stale for the whole relation tree of a failed table - Keep the state-not-saved cause on later errors, which still stop the next sources - Warn append mode destinations about synced again rows only for sources that sent checkpoints - Test a Send on a stream that ended without a reported failure, and several sources that save no state - Drop the table name changer subtest that main already has
- Treat a failed table the source did not sync as any table, and bound the walk up a relation tree - Say the sync stopped, not finished, when a later error joins the unsaved state - Report a failure once when a hard flush error follows it - Test a stream end first seen by a flush and a later failure that names no tables - Check the delete-stale decision in the checkpointer tests instead of a test-only helper
Adds state_error to the sync summary, so a sync that finished but lost records differs from one that crashed. The summary gained fields before without a destination migration.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When every destination supports flush, the CLI flushes them before it writes each source state checkpoint to the backend.
Tested with a local SQLite sync: after a source crash, the stored cursor never passed the committed rows.
Uses the protocol from plugin-pb-go v1.28.2.
Checkpoints stay off with transformers or destinations without flush support.
If a destination or the state backend reports a failed flush, the CLI saves no more state for that source, finishes the sync and exits with an error, which the sync summary also gives as
state_error. The next sources still sync.A destination that ends its stream after it reports the failure still lets the sync finish. A writer that stops on a failed write before it reports it on a flush, such as the mixed batch writer, still ends the whole run, as on main.
At the end of the sync, the CLI flushes failed destinations again, so it knows every table they lost records of.
Side effects of this policy:
Also, with or without checkpoints, the CLI now closes every destination before it returns a close error.