feat: surface mid-stream parse errors via ParseWithErrors - #9
Open
vaibhav8a wants to merge 1 commit into
Open
Conversation
Parse discarded the error when a fragment failed to parse and let the deferred close(out) end the stream, so an abandoned parse was indistinguishable from a well-formed patch that simply ended. Consumers scanning patch content therefore reported success over the part they had managed to read (gitleaks#1338). Add ParseWithErrors returning a buffered, always-closed error channel carrying the error that ended the parse. Parse delegates to it and drains the channel, so its contract is unchanged and the parse goroutine is never left blocked on an unread send. Refs gitleaks/gitleaks#1338
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs gitleaks#1338.
The problem
Parsedrops the error when a fragment fails to parse:The deferred
close(out)then closes the file channel exactly as it would at the end of a well-formed patch. A caller ranging over that channel cannot tell a patch that ended from one abandoned part-way through.That is the worst shape a failure can take for a consumer that scans patch content: nothing surfaces, and the answer is the reassuring one. In gitleaks it means a secret scan silently covers only the part of the diff that parsed, and exits 0 — the report in gitleaks#1338, where a hunk header of
@@ -231,18446744073709551615 +231,1 @@failedparseRange.What this adds
ParseWithErrors(r) (<-chan *File, <-chan error)— the same parse, with the error that ended it surfaced. The error channel is buffered, yields at most one value, and is closed when parsing finishes, so a caller that drains the files and then reads the error channel seesnilfor a patch that ended normally.Parseis unchanged for existing callers. It now delegates toParseWithErrorsand drains the error channel in a goroutine, so no caller has to change and the parse goroutine is never left blocked sending an error nobody reads.TestParseKeepsItsOldContractpins that for both a good and a malformed patch.Why not detect it in the consumer instead
I tried that first, in gitleaks: track whether the reader ever reached EOF, on the reasoning that the parse loop only ends normally at EOF. It does not work —
newParserwraps the reader in abufio.Reader, so read-ahead pulls a small patch entirely into the buffer before the parse error is hit, and EOF is observed even on an aborted parse. The signal would have been real for large diffs and silently wrong for small ones, which is worse than none. The error has to come from where it is raised.Tests
7 tests, all passing, plus the existing suite and
-race:Parsekeeps its old contract, and its goroutine finishes on both good and malformed input;go test ./...,go test -race ./gitdiff/andgo vetare clean.gofmtreports onlyapply.goandtestdata/apply/bin.go, which are untouched by this change and already unformatted onmaster.Follow-up
Once this is released,
gitleaks/sources/git.gocan switch its twogitdiff.Parsecall sites toParseWithErrorsand surface the error instead of reporting a clean scan. Happy to open that PR against gitleaks when you tag a version.