Skip to content

fix(retain): constrain occurred_start/occurred_end so grammar-constrained models can't emit prose - #3845

Merged
nicoloboschi merged 4 commits into
vectorize-io:mainfrom
nickanderson:fix/occurred-timestamp-pattern
Aug 31, 2026
Merged

fix(retain): constrain occurred_start/occurred_end so grammar-constrained models can't emit prose#3845
nicoloboschi merged 4 commits into
vectorize-io:mainfrom
nickanderson:fix/occurred-timestamp-pattern

Conversation

@nickanderson

Copy link
Copy Markdown
Contributor

Problem

occurred_start and occurred_end are LLM-facing fields typed as bare
str | None, so the generated JSON schema advertises "any string". Their
descriptions say "ISO timestamp", but under grammar-constrained decoding
(response_format: json_schema → GBNF) a description is not a constraint
the grammar is the only thing standing between the model and an arbitrary
string.

When the model is asked for a timestamp it cannot derive — e.g. a narrative
duration with no start time — it reasons inside the string value:

"occurred_start": "2026-08-20T00:00:00Z/N/A (duration 40 mins implied within the
 incident window, but specific start time not provided, using event date as
 reference) -> 2026-08-20/2026-08-20 ... Actually, I will use ... Wait, the prompt says "

Two failures from one root cause:

  1. Corrupted record — a stored fact whose occurred_start is a paragraph of
    the model's reasoning rather than a date.
  2. Runaway generation — it doesn't stop. Observed runs burned the entire
    completion budget (2500/2500, and 4,598+ tokens on a larger cap) producing an
    unterminated JSON body with finish_reason: "length", which the retain path
    then re-sends byte-identical. That is the failure class described in finish_reason: "length" on the non-streaming .create() path is treated as a parse error, not OutputTooLongError — the fact-extraction auto-split (#2579/#3174) is still dead code for the truncation subclass #3811
    and openai_compatible_llm: bare json.loads + identical retries — parse_llm_json/json_repair never reached, deterministic failures burn all 4 attempts and drop the chunk #3683; this PR removes one of its triggers.

Reproduced with gemma-4-26B-A4B-it (Q4, Q6 and Q8) w/ llama.cpp.

Field emission order is significant here and is not always the client's to control. llama.cpp compiles properties into the grammar in order and enforces it.
re-serializes the request body with a sorted-map JSON type silently alphabetizes properties — which puts occurred_end/occurred_start ahead of
what — and the client cannot detect this, because the schema is still ==
equal as an object. (I hit this with one such proxy.
this PR fixes the symptom.)

@strix-security

strix-security Bot commented Aug 27, 2026

Copy link
Copy Markdown

Strix Security Review

Warning

This pull request has 30 commits after the last Strix review (6772cb3). Strix has not reviewed these changes.
Automatic review on push is off for this repository. To review the latest changes, tag @strix-security in a comment, or turn on re-review on push.

No security issues found.

Updated for 6772cb3.


Reviewed by Strix
Re-run review · Configure security review settings

@nickanderson
nickanderson marked this pull request as draft August 27, 2026 17:58
@nickanderson
nickanderson marked this pull request as ready for review August 27, 2026 22:46
nickanderson and others added 3 commits August 31, 2026 11:43
…p pattern

Both fields are LLM-facing but typed as bare `str | None`, so the generated
JSON schema advertises "any string". Under grammar-constrained decoding
(response_format: json_schema -> GBNF) a description is not a constraint, and
the model can put prose in the field.

When asked for a timestamp it cannot derive -- e.g. a narrative duration with
no start time -- it reasons inside the string value and runs to the completion
cap, producing a corrupted record plus a truncated body with
finish_reason: "length" that then gets retried byte-identical (vectorize-io#3811, vectorize-io#3683).

Constrain the string branch with a pattern permissive about precision (date
only, date+time, optional seconds/fraction, optional Z or offset) and strict
about everything else. null stays valid as the escape hatch for "no derivable
date"; _infer_temporal_date() already backfills from the text.

The internal Fact model is deliberately left lenient: it is the post-parse
storage model, not an LLM contract.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Assert the LLM-facing schemas expose a pattern-constrained string branch and
keep null, that real timestamp formats still validate, that prose (including
the observed runaway string) is rejected, and that the internal Fact model
stays lenient.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ity flag

The `pattern` keyword is only usable on backends that accept it, and
rejection is a hard 400 at request time rather than a degraded response:
Bedrock validates structured-output schemas against an allowlist that
excludes it, and OpenAI errors on unsupported keywords under `strict`.
Backends that neither enforce nor reject it (native Anthropic tool
schemas) gain nothing from it either.

So the constraint is no longer declared on the fact models. It is layered
on at schema-build time by `_with_iso_timestamp_pattern()` when
HINDSIGHT_API_LLM_SUPPORTS_STRING_PATTERN is set, mirroring the existing
`llm_supports_max_items` escape hatch that exists for the same reason.
Constraining the Pydantic model rather than post-processing the serialized
schema is what makes it uniform across providers -- Gemini is handed the
response model itself, not a schema dict.

Tests now pin both halves: no `pattern` in the schema by default across
every extraction mode, and when enabled, a constrained string branch that
stays nullable, composes with the entity-labels model, accepts the
timestamp formats real models emit, and rejects prose.
@nicoloboschi
nicoloboschi force-pushed the fix/occurred-timestamp-pattern branch from 6772cb3 to 9685f27 Compare August 31, 2026 10:03
Two CI follow-ups to the capability flag:

- skills/hindsight-docs is generated from hindsight-docs; the new
  configuration.md row has to be regenerated into it or
  verify-generated-files fails on the drift.
- test_retain_cacheable_prefix_invariant_to_per_bank_freetext builds its
  config as a SimpleNamespace, which raises AttributeError on a field it
  doesn't declare. Adding the field to the stub keeps the failure mode
  honest -- production passes a real HindsightConfig, so a stub that
  silently defaulted via getattr would hide a genuine config-shape error.
@nicoloboschi
nicoloboschi merged commit 5c78b81 into vectorize-io:main Aug 31, 2026
106 of 108 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants