Skip to content

Fix uncaught pydantic ValidationError crash on malformed LLM response#255

Open
Ruanyuxi1337 wants to merge 1 commit into
NVIDIA:mainfrom
Ruanyuxi1337:fix-validation-error-crash
Open

Fix uncaught pydantic ValidationError crash on malformed LLM response#255
Ruanyuxi1337 wants to merge 1 commit into
NVIDIA:mainfrom
Ruanyuxi1337:fix-validation-error-crash

Conversation

@Ruanyuxi1337

Copy link
Copy Markdown

Fixes #250. Wraps the per-batch LLM request and parse loop in a try-except guard. If a parsing or network validation failure occurs, it warns, retries exactly once, and falls back gracefully, preventing the entire scan from crashing and losing all report progress.

@koriyoshi2041

Copy link
Copy Markdown

Thanks for taking this on. One gap worth covering before merge: this only wraps the async arun_batches() path, but semantic_security_discovery still calls analyzer.run_batches(batches).

In that sync loop, a malformed structured response from one batch still raises out of run_batches; the outer node catch then returns zero findings for the whole semantic analyzer, including any earlier batches that parsed successfully. A focused regression would be: three sync batches, middle one raises a Pydantic ValidationError from LLMAnalysisResult.model_validate(...), and the first/third batch results are still returned.

So I think the same per-batch isolation should be added to run_batches() too, otherwise #250 can still reproduce through that analyzer path.

@keshprad keshprad left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes because the async retry still aborts the scan after two malformed responses and changes the documented handling of ordinary ValueError. I also agree with @koriyoshi2041 that the synchronous run_batches() path used by semantic_security_discovery remains fail-fast and needs equivalent per-batch isolation. Please add regressions for persistent malformed async responses, ordinary ValueError propagation, and a three-batch sync run with a malformed middle response.

batch.file_label,
retry_exc,
)
raise retry_exc

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A persistent malformed response still reproduces #250 here. Pydantic ValidationError subclasses ValueError, so after the retry re-raises it, the aggregation loop matches it as ValueError and aborts the entire scan. Please handle exhausted Pydantic validation failures as an isolated batch failure while preserving propagation for ordinary configuration ValueError, and add a regression where both attempts are malformed.

except NotImplementedError as exc:
# Do not retry on configuration / code errors
raise exc
except Exception as exc:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This broad catch changes the documented ValueError contract. If the first call raises an ordinary configuration ValueError (for example, a missing API key) and the retry succeeds, arun_batches() returns successfully instead of propagating the configuration error. Please catch Pydantic ValidationError explicitly for retry and let non-Pydantic ValueError bypass retry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Single malformed LLM response aborts the entire scan (uncaught pydantic ValidationError) — triggered consistently by the default nv_build model

3 participants