Conversation
…rose
Structured output already recovers a JSON object wrapped in prose, but only
when exactly one syntactically valid top-level object is present. A model that
drafts its answer inside a reasoning preamble emits a second valid object, and
even the sentence "return JSON, not {}" is enough, so recovery gave up and a
validation attempt was spent on text that could never parse.
Extraction now returns every balanced candidate and the task's own decoder
chooses among them: exactly one passing the schema is accepted, zero or several
keep the existing error. A candidate that fails DisallowUnknownFields, or
carries the wrong thread or schema version, was never a possible answer, so
this changes which bytes reach the decoder without changing what the decoder
accepts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #608
Problem
Structured-output decoding already recovers a JSON object from surrounding prose, via
extractSingleJSONObject. It requires exactly one syntactically valid top-level object, deliberately, so ambiguous output is never guessed at.A reasoning preamble routinely breaks that. Running the resident extractor against representative inputs:
no braces heredraft: {"decision":"skip"}I must return JSON, not {}.{}is a candidateThe budget is two validation attempts and an exhausted task is blocking, so a model that drafts before answering loses half its retry budget systematically rather than occasionally.
Change
Extraction returns every balanced valid candidate, and the task's own decoder picks among them: exactly one passing the schema is accepted, zero or several keep the existing error. A lone failing candidate still reports its own schema error, since that describes the real violation.
This cannot loosen validation. A candidate failing
DisallowUnknownFields, or carrying the wrongthread_idorschema_version, was never a possible answer — only the bytes reaching the decoder change, never what the decoder accepts. Trailing-data rejection is unaffected: each candidate is a single balanced object, and the raw-text check still runs first.Evidence
Every behavior change mutation-checked by reverting it alone:
recovers_the_schema-valid_object_when_the_preamble_drafts_anotherpassed >= 1rejects_two_schema-valid_objectsi = endadvanceTestExtractJSONObjects/nested_objects_and_arraysjson.Validprose_braces_alongside_one_valid_object,valid_object_nested_in_invalid_prose_bracesgo build ./...,go vet ./internal/...andgo test -count=1 ./...are clean.Note on scope
The originally filed issue claimed no extraction existed; that was wrong and the issue has been corrected. A truncated response remains unrecoverable by extraction and is the retry path's job, not this one's.