feat: add stream_validate() hook to Requirement (#900)#925
Draft
planetf1 wants to merge 14 commits intogenerative-computing:mainfrom
Draft
feat: add stream_validate() hook to Requirement (#900)#925planetf1 wants to merge 14 commits intogenerative-computing:mainfrom
planetf1 wants to merge 14 commits intogenerative-computing:mainfrom
Conversation
Adds mellea/stdlib/chunking.py with ChunkingStrategy ABC and three built-in implementations: SentenceChunker, WordChunker, ParagraphChunker. split(accumulated_text) returns complete chunks, holding back trailing fragments for the next call. Closes generative-computing#899 Part of generative-computing#891 Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
Adds PartialValidationResult to mellea/core/requirement.py — a tri-state result type for per-chunk streaming validation. Follows the same private-fields-with-property-accessors pattern as ValidationResult. success: Literal["pass", "fail", "unknown"]. __bool__ returns True for "pass", False for "fail" and "unknown". "pass" is informational only in Phase 1; orchestrators call validate() at stream end for all non-"fail" results. Closes generative-computing#898 Part of generative-computing#891 Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
…esult._success Pyright infers self._success as str without the annotation because it doesn't narrow Literal types through bare instance attribute assignment. The explicit annotation makes the property return type verifiable. Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
Pyright flagged import as unaccessed; no pytest.* calls in the file. Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
…hunkingStrategy - Fix SentenceChunker whitespace leak: lstrip() after match.end() so double-space / tab separators don't bleed into the next chunk as leading whitespace - Add end-of-stream contract to ABC docstring (callers responsible for trailing fragment after stream terminates) - Fix incorrect comment "end-of-string" → "whitespace" - Compile _WHITESPACE / _PARA_BOUNDARY / _PARA_BOUNDARY_END at module level (consistent with _SENTENCE_BOUNDARY; avoids per-call recompile) - Expand SentenceChunker char class to include right curly double/single quotes (U+201D / U+2019) for common LLM output patterns - Document CRLF limitation on ParagraphChunker - Re-export ChunkingStrategy + chunkers from mellea.stdlib.__init__ - Add __all__ to chunking.py - Add tests: closing paren, double-space separator, tab separator, abbreviation edge case (known-bad split), WordChunker leading-whitespace Assisted-by: Claude Code
…tialValidationResult - Update mellea/core/__init__.py module docstring to mention PartialValidationResult - Update mellea/core/requirement.py module docstring to mention PartialValidationResult - Document ValidationResult/PartialValidationResult API asymmetry in class docstring - Add runtime validation of success parameter (fail fast on invalid values) - Expand as_bool() docstring with streaming-context warning for "unknown" → False - Add __repr__ for opaque debug output in streaming pipelines - Replace test_as_bool_matches_bool loop with parametrised test_as_bool_correctness - Add test_invalid_success_raises and test_repr_shows_state Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
…acy and curly-quote test - Fix misleading comment on _SENTENCE_BOUNDARY: was "processed by re engine as \u escapes" but the file contained literal Unicode chars. Now uses chr(0x201d) + chr(0x2019) for Python 3.12 compatibility (U+2019 is treated as a string delimiter in single-quoted raw strings on 3.12). - Add test_sentence_chunker_curly_quotes to verify U+201D/U+2019 matching. Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
- Add __repr__ to ValidationResult for parity with PartialValidationResult - Add test_thunk_field and test_context_field to close the test coverage gap for keyword-only constructor arguments (previously only default-None was verified) Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
Merges feat/898-partial-validation-result (PartialValidationResult) and feat/899-chunking-strategy (ChunkingStrategy ABC) into a single base for Wave 2 and beyond. Assisted-by: Claude Code
96f1919 to
0c030fb
Compare
planetf1
added a commit
to planetf1/mellea
that referenced
this pull request
Apr 24, 2026
…eam_validate - Remove "In Phase 1" temporal qualifier from docstring — reworded to timeless statement about orchestrator responsibility - Add type annotations (str, Backend, Context) to test subclass overrides - Add idempotency test: multiple calls on the same Requirement instance leave state unchanged Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
…mputing#923) - Simplify _SENTENCE_BOUNDARY regex to use \u escapes instead of chr() concatenation (cleaner, same semantics, Python 3.12-safe) - Document that SentenceChunker discards inter-sentence whitespace via lstrip() - Add test_chunking_strategy_is_abstract to document the extension-point contract Assisted-by: Claude Code
…#900) Add an async `stream_validate(chunk, backend, ctx)` method to the base `Requirement` class. The default implementation returns `PartialValidationResult("unknown")`; subclasses override to inspect the accumulated chunk and return `"pass"` or `"fail"` early. Per the Phase 1 design: `"pass"` is informational and does not short-circuit the final `validate()` call. The method must not mutate `self` — state isolation is the orchestrator's responsibility. Signed-off-by: Nigel Jones <jonesn@uk.ibm.com> Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
…eam_validate - Remove "In Phase 1" temporal qualifier from docstring — reworded to timeless statement about orchestrator responsibility - Add type annotations (str, Backend, Context) to test subclass overrides - Add idempotency test: multiple calls on the same Requirement instance leave state unchanged Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
Prevents positional confusion and makes future parameter additions to the signature non-breaking for existing subclass overrides. Assisted-by: Claude Code
d922a2c to
58128a7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a stacked PR (Wave 2 of streaming epic #891). The diff against
mainincludes code from the two Wave 1 PRs that this branch builds on:PartialValidationResulttri-state typeChunkingStrategyABC + built-in chunkersstream_validate()hook onRequirementSuggested review order:
What to ignore in the diff:
mellea/stdlib/chunking.py,test/stdlib/test_chunking.py,mellea/stdlib/__init__.py(covered by #923), and thePartialValidationResultclass inmellea/core/requirement.py(covered by #924). Those files appear in this diff because the branch is stacked, not because this PR changes them.What to focus on:
mellea/core/requirement.py— thestream_validate()method added toRequirement(one new method, ~10 lines)test/core/test_stream_validate.py— 7 new unit testsSummary
Adds
async stream_validate(chunk, *, backend, ctx) -> PartialValidationResultto the baseRequirementclass as a per-chunk streaming validation hook. The default implementation returnsPartialValidationResult("unknown")— subclasses override to inspect accumulated chunks and signal"pass"or"fail"early. The method is explicitly documented as non-mutating; state isolation is the orchestrator's responsibility.Design decisions
Per the agreed Phase 1 spec:
stream_validate(notavalidate)PartialValidationResult(tri-state:"pass","fail","unknown") — introduced in feat(core): add PartialValidationResult with tri-state semantics #898 / feat(core): add PartialValidationResult with tri-state semantics #924backendandctxare keyword-only: prevents positional confusion and makes future parameter additions non-breaking"pass"is informational: does not short-circuit the finalvalidate()call in Phase 1reset()method: state management is delegated to the orchestrator via cloningNo LLM-as-a-Judge logic is added here. This is a pure hook for custom validation overrides.
Test plan
test_default_returns_unknown— base class always returns"unknown"test_default_returns_partial_validation_result_instance— correct return typetest_stream_validate_is_coroutine— method is asynctest_subclass_can_return_pass— subclass override returning"pass"workstest_subclass_can_return_fail— subclass override returning"fail"with reason workstest_does_not_mutate_requirement— calling the method leavesselfunchangedtest_stream_validate_idempotent— multiple calls leaveselfunchangedRelated
Closes #900. Part of streaming epic #891. Stacks on #898 / #924 and #899 / #923.