feat: add stream_validate() hook to Requirement (#900)

[planetf1]

Summary

Adds async stream_validate(chunk, *, backend, ctx) -> PartialValidationResult to the base Requirement class as a per-chunk streaming validation hook. The default implementation returns PartialValidationResult("unknown") — subclasses override to inspect accumulated chunks and signal "pass" or "fail" early. Stateful implementations may accumulate state on self; the orchestrator clones the requirement before each attempt so state does not bleed across retries.

Design decisions

Per the agreed Phase 1 spec:

• Method name: stream_validate (not avalidate)
• Return type: PartialValidationResult (tri-state: "pass", "fail", "unknown") — introduced in feat(core): add PartialValidationResult with tri-state semantics #898 / feat(core): add PartialValidationResult with tri-state semantics #924
• backend and ctx are keyword-only: prevents positional confusion and makes future parameter additions non-breaking
• "pass" is informational: does not short-circuit the final validate() call in Phase 1
• No reset() method: state isolation is handled by the orchestrator cloning requirements (copy(req)) before each attempt

No LLM-as-a-Judge logic is added here. This is a pure hook for custom validation overrides.

Test plan

• test_default_returns_unknown — base class always returns "unknown"
• test_default_returns_partial_validation_result_instance — correct return type
• test_stream_validate_is_coroutine — method is async
• test_subclass_can_return_pass — subclass override returning "pass" works
• test_subclass_can_return_fail — subclass override returning "fail" with reason works
• test_does_not_mutate_requirement — calling the method on base class leaves self unchanged
• test_stream_validate_idempotent — multiple calls on base class leave self unchanged
• test_stateful_subclass_accumulates_state — stateful subclass (bullet counter) correctly updates internal state across calls
• test_stateful_subclass_clone_isolation — copy() of a stateful requirement produces an independent clone, confirming the orchestrator clone pattern

Related

Closes #900. Part of streaming epic #891. Builds on #924 (merged) and #923 (merged).

[planetf1]

@jakelorocco how does this look for you - I have another stacked PR behind this one too. (I'll just go one level deep)

[jakelorocco]

@planetf1, I might've missed this in the original proposal, but passing the accumulated chunks to the requirements is different from what I thought the proposal was. Could you please elaborate more on the design choice?

[jakelorocco]

This worries me. @nrfulton, your initial proposal was to have the requirement only see new chunks. This would show all accumulated chunks so far.

This forces all streaming requirements to be stateful; all requirements must now keep track of what chunks they have processed. The alternative would be to only provide new chunks to requirements; then streaming validation would be stateless except when needed. Requirements can choose whether they need to store and process multiple chunks, or just check each chunk independently.

I guess the checking each chunk independently is unlikely to be helpful, so I can see why accumulating and forcing requirements to track their own progress doesn't actually add much complexity.

If so, I think we should actually pre-define functions to help with this (either as a new class of requirements or functions that implementors can draw on).

Also, if we are passing the accumulated chunk through, I almost think we should just pass in some point-in-time copy of the model output thunk. Ie one that doesn't get streamed the new chunks but has all the data fields from the point-in-time it was copied at.

[planetf1]

I would suggest reverting to the simplicity of the original proposal. The change was made incorrectly when reviewing the initial PR. The initial approach is simple - and requirements can maintain state if the need to work on accumulated chunks. If we need to support accumulated content generally we can consider it in a later phase.

Which works best will vary by use case. Per-chunk like works better for

• checkinf for forbidden words/phrases
• ensuring a paragraph or sentence is coherent
• structural checks - code fencing
• format validation

especially when the MoT manages the semantic chunking (later)

There will be cases where accumulation is better -- these are probably more complex checks - does a story line flow, are we taking the response in an unexpected direction, do we have a complete enough response

Importantly if we stick to per-chunk we could still implement this second approach - albeit not as cleanly.

So in summary - I'll revert to original -- but if you now think that's wrong we can adjust?