LLM reference segmenter and citation engine - #702
Draft
de-code wants to merge 12 commits into
Draft
Conversation
ScienceBeam Parser EvaluationOverall (55 docs across 6 corpora)grobid 0.9.0-crf (default): 60 docs | sciencebeam-parser:main-41940126-20260820.0246 (grobid_crf): 60 docs | sciencebeam-parser:pr-702-bee85626-20260820.1336 (llm_citation): 55 docs
biorxiv (9 docs)grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-41940126-20260820.0246 (grobid_crf): 10 docs | sciencebeam-parser:pr-702-bee85626-20260820.1336 (llm_citation): 9 docs
ore (9 docs)grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-41940126-20260820.0246 (grobid_crf): 10 docs | sciencebeam-parser:pr-702-bee85626-20260820.1336 (llm_citation): 9 docs
pkp (10 docs)grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-41940126-20260820.0246 (grobid_crf): 10 docs | sciencebeam-parser:pr-702-bee85626-20260820.1336 (llm_citation): 10 docs
scielo_br (10 docs)grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-41940126-20260820.0246 (grobid_crf): 10 docs | sciencebeam-parser:pr-702-bee85626-20260820.1336 (llm_citation): 10 docs
scielo_mx (9 docs)grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-41940126-20260820.0246 (grobid_crf): 10 docs | sciencebeam-parser:pr-702-bee85626-20260820.1336 (llm_citation): 9 docs
scielo_preprints-jats (8 docs)grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-41940126-20260820.0246 (grobid_crf): 10 docs | sciencebeam-parser:pr-702-bee85626-20260820.1336 (llm_citation): 8 docs
|
A third sequence-model engine alongside wapiti and delft, reachable only through one of three opt-in profiles: llm_reference_segmenter, llm_citation, or llm_references for both. One profile per model so that a failing run says which model produced it. The shipped default stays grobid_crf, and a default install acquires no network dependency, credential requirement or new failure mode — asserted by test rather than intended, including that every other model in each profile stays on wapiti. No text reaches a document that was not in the source. The segmenter returns line numbers into numbered input and never text at all. The citation model returns field values, which the engine locates back in the token sequence: a value that cannot be found, or whose tokens an earlier field already claimed, raises. Either way the existing seam in Model independently rejects any result whose tokens are not the input tokens. The two shapes are opposite on purpose. Whatever the model must emit verbatim is what breaks at scale, so the shape follows the ratio of output to input: a citation response is shorter than its input, a segmentation response quoting reference text is as long as it. response_shape is therefore configuration, so comparing shapes is defining a second profile rather than building a second evaluation route. An invalid response raises rather than falling back to a CRF engine, since a score is only meaningful if every label came from the model under test. A response cut off at the output limit raises its own error naming finish_reason and the completion token count, because "truncated" and "malformed" want different fixes and reporting the first as the second sends you looking for a parsing bug. Every request pins zero data retention and fails closed, and a :free model id is refused at load because that tier requires allowing training on prompts. Three details that came from measurement rather than taste. The line_status column is resolved by name through the task's own data generator, because the feature layout differs per model and a wrong column produces plausible output with no exception. Predicted boundaries snap onto a preceding bare-label line, because models point at the line where reference text starts while the training convention puts the boundary on the line holding the number — and stating that in the prompt did not fix it. Values are matched on word tokens at the earliest unclaimed occurrence rather than in document order, because models write "Treble-Barna" where the tokeniser emits three tokens, and emit date out of position often enough that requiring order rejects values the source contains. The citation label vocabulary is read from TRAINING_XML_ELEMENT_PATH_BY_LABEL rather than restated, so it cannot drift from what the extractor understands. Checked end to end against qwen3.5-9b: 33 references against a gold of 33 over 1502 tokens for the segmenter, and 0.865 token accuracy over 223 tokens for citation, with tokens returned unchanged in both.
Spans follow OpenTelemetry's GenAI semantic conventions, so they are meaningful in any OTLP backend: gen_ai.operation.name, gen_ai.request.model, temperature and max_tokens, gen_ai.usage.input_tokens and output_tokens, response model and finish reasons, and — configurably — gen_ai.prompt and gen_ai.completion. OpenInference names are emitted alongside them so a backend reading those instead still renders the span as an LLM call. Nothing in the engine names a backend; Phoenix is only what listens in development, and the collector is configured with plain OTEL_EXPORTER_OTLP_ENDPOINT. This answers the question nothing else did. The response body says what a run cost; it never said what the model actually said, and every question of that kind so far has cost a re-run. The response body is attached to the span even when it fails to decode, which is the point: a truncated or malformed response becomes visible rather than inferred from an exception. PHOENIX_COLLECTOR_ENDPOINT is deliberately not read. An earlier version accepted it, which put one backend's configuration in the engine and, worse, disagreed with the exporter: the exporter reads only the OTEL_ variables and otherwise defaults to localhost:4318, so tracing switched on and then exported to a closed port. Phoenix is a compose service rather than a bare docker run, matching the convention that dev- targets use the venv and docker- targets use compose. It sits behind the telemetry compose profile so a plain docker-start does not bring an observability server up with it, is pinned rather than tracking latest, and keeps its sqlite db in a named volume so traces survive a restart — without one, comparing two runs would mean never restarting between them. dev-start-with-telemetry runs the host parser with the OTLP endpoint set, and nothing else: which profile is active is set the usual way with SCIENCEBEAM_PARSER__PROFILE, which is a general mechanism rather than anything to do with tracing. dev-install now includes the telemetry extra, because a dev environment that cannot see the responses is the situation this is meant to end. Optional in both directions. Without the extra installed, or without an endpoint in the environment, the engine emits nothing and behaves identically — a default install acquires no tracing dependency, and emission stays gated on the endpoint rather than following the extra. An existing tracer provider is left alone rather than replaced. The optional imports go through importlib rather than guarded static imports, so neither mypy nor pylint needs telling to ignore a module that is absent by design. A span carrying prompt text is a copy of manuscript text. Sending it to a collector on localhost is not a new disclosure when the same text already goes to the model, but sending it elsewhere is, so record_trace_content switches the text off while keeping the metrics.
processor.py hands the engine every reference of a document at once, and the engine was making one HTTP call per reference. It now sends max_references_per_request (default 10) per call, each numbered in the prompt, with the response nesting one entry per reference so a batch stays a transport decision rather than growing the index space the model has to track. Requiring an answer for every reference sent is what makes this safer than a flat list, not just faster: a merged or omitted reference fails validation rather than scoring, as does a duplicated or out-of-range index. Values are located within their own reference's tokens only, so a value belonging to reference 3 cannot be matched against reference 5. Measured on 10 real references: one batched call took 22s against 48s for ten single calls, at 0.906 token accuracy against 0.904. A first attempt measured 320s for the batched case, which turned out to be a provider timeout returned as HTTP 200 with an error body — the retry only inspected the status code, so it never fired and the failure surfaced as "response has no choices". Error bodies are now inspected for a retryable code, and the 320s was the bug rather than the batching. The tail remains: a long generation draws that timeout occasionally, and the retry costs more than the batching saves, so the bound is the lever on a flaky provider. The prompt is a new version rather than an edit, since its shape changed; values-v1 stays on disk so an earlier number can still be reproduced.
The model returns a line number per reference and the first words on that line. The number is still the answer; the words are only a check on it, so a wrong quote costs a check rather than the reference — which is the failure mode the anchor shape had, where the quoted text was the locator and a mismatch lost the reference entirely. Not the default. The pilot behind this measured evidence at about 0.19 partial_list above lines on a 12B and 0.003 above it on the 9B shipped here, at roughly four times the output tokens. On one real reference list both find 33 references against a gold of 33, in 5.3s and 6.9s. It earns its cost on a weaker model, so it is available rather than assumed. A quote is accepted against the line it names or the one below it, because models name the line holding the reference number while quoting the words underneath — which is the training convention rather than an error. Anything else increments a span attribute and logs; evidence_mismatch_raises makes it load-bearing. Off by default because a mismatch is a well-formed answer whose evidence disagrees, not a protocol violation, and in the pilot a 12B mismatched on 158 of 210 references against the 9B's 1 of 118. Word splitting moves into decode.py and the value shapes import it, rather than each shape carrying its own: two implementations of "does this text match these tokens" drifting apart is how a match silently stops matching, which a test caught here before it could.
The batched prompt introduced each reference with `[n]`, which is what a numbered reference list looks like, and paired `[0]` with `note: "1"` in its worked example. The model duly returned the marker as a `note` — on an unnumbered list, or on a mislabelled region with no bibliographic content, that value then appears nowhere in the reference and the request fails. Measured over 12 documents: 4 failed this way, all with `note` equal to the index. values-v3 introduces each reference with `REFERENCE n` on its own line, says the marker is not part of the reference and must never be returned, and uses an example whose indices differ from its notes with a second reference that is unnumbered and so has none. 11 of the same 12 documents now pass, and token accuracy over those documents rises from 0.877 to 0.930. Values are matched on characters rather than word sequences, because models join and split words differently from the tokeniser: `Moreno - San Segundo` comes back as `Moreno SanSegundo`, which no word comparison can match. A match must still begin at a token boundary, so a value cannot be located mid-word. The one remaining failure in those 12 is the model attributing reference 5's author to another reference in the batch. That is caught rather than silent precisely because values are located within their own reference's tokens — a flat field list would have found it elsewhere in the document and labelled it. The error now names the reference index, the label and the reference text, which is what made this diagnosable at all. The segmenter prompts gain a v2 telling them to report only bibliographic references and to return an empty list if the region contains none, since the region is sometimes a table and deciding what is a reference is the segmenter's job. An empty answer is now accepted rather than rejected.
Two fields claiming the same span was raising, which cost every reference in the document over one confused field. It also contradicted the rule this engine is built on: protocol violations raise, well-formed but implausible labellings are scored. A value absent from the source violates the guarantee and still raises. A second field over the same text does not — the text is in the source either way — so the later claim is dropped with a warning naming the reference, the label and the text. Found on a region the segmentation model mislabelled: the model was handed table rows, labelled a whole row as a booktitle and then again as a title, and the document failed. wapiti given the same input produces poor output rather than failing, so raising there also biased any comparison towards the engine that gives up.
Four reported failures in a row were the same class: the region handed to the citation model was not a reference list, the model invented a plausible field value, and one such field destroyed every reference in the document. Each was patched at the symptom — the batch marker, character-level matching, overlapping spans — and the class stayed. The mistake was conflating decision 4's guarantee with raising. The guarantee is that no text reaches a document that was not in the source, and discarding an unlocatable claim satisfies it completely: the value never becomes a label. Raising is stronger than the guarantee needs, and it fails precisely the documents where the CRF path degrades instead of failing, which biases any comparison towards the engine that gives up. So: a response the engine cannot parse raises — bad JSON, a missing reference entry, an out-of-range index. A claim it cannot honour is dropped, counted on sciencebeam.dropped_fields, and logged with the reference, label and text. dropped_field_raises restores strictness for a run that wants it. The remaining problem is upstream and unchanged: these documents are results tables that segmentation labelled as references. The engine now degrades to a countable number instead of an exception, which is what makes the scale of that visible.
Batches are independent, so the wall-clock should be the slowest batch rather than their sum. On 33 references in 4 batches, 4 concurrent calls took 111s down to 58s with accuracy unchanged. Order is preserved by mapping rather than completion order, and the first exception propagates. max_references_per_request and max_concurrent_requests are now in config.yml rather than only in the dataclass, with the reasoning next to them: raising the batch size is the wrong lever, because decode cost is linear in output tokens however they are grouped, so a bigger batch generates the same amount in one long stream that cannot be overlapped — and long generations are what draw the provider timeouts seen earlier. More, smaller batches running concurrently is the direction that helps. Recorded next to the setting: lower the concurrency if the provider answers with 429s, since the client retries with backoff but a rate-limited provider makes concurrency a net loss.
Measured on 33 references in 4 batches: 12s against 39s, at the same fp8 quantisation and the same price, with 100% reported uptime against 96% over the last 30 minutes. On the reference segmenter the same document took 2.1s against 3.9s with identical output. It costs about 0.007 token accuracy — 0.860 across three runs against 0.867 — which sits inside the run-to-run spread of most things measured for this engine, while latency has been the binding constraint throughout. Venice was also the steadiest thing measured today, at 11.9/12.1/11.9 seconds and 0.859/0.859/0.862. Records that concurrency beyond 4 stops helping on either provider, so the bound was provider throughput rather than parallelism, and that not every provider is reachable under the zero-retention pin: Parasail answered 429 immediately, so a candidate has to be tried rather than looked up.
The parser runs in a container in CI, so the key has to reach the container. It is passed as a bare `-e OPENROUTER_API_KEY` with the secret in the step environment: the value never appears in a command line, and an unset secret leaves the variable unset in the container rather than empty. Every non-LLM profile ignores it, so it is passed unconditionally and the engine is selected the usual way, with a profile: label or the workflow_dispatch input. PRELOAD_ON_STARTUP is already set there, so an llm profile validates its endpoint and model id while the container starts. A missing key fails at "Wait for parser" with the reason in the logs rather than part-way through a run, and that step now says so. benchmarks/run.py refuses an llm profile together with plos-manuscripts, naming both. This is the case that matters: on main the workflow adds --include-corpus plos-manuscripts automatically, so without the guard an llm profile on main would send non-redistributable manuscripts to a third party. Provider zero-retention does not cover the intermediary, which is why the exclusion cannot be satisfied by routing flags. The check is in run.py rather than only the workflow so it also protects a run from a laptop.
The native-test job runs pytest over tests/ alone with only the dev group installed, so a test there cannot import benchmarks.run: that pulls in benchmarks.fetch, benchmarks.corpus_source and then pyarrow, which the benchmark group provides and that job does not install. benchmarks/tests is collected separately and already imports benchmarks.run freely, which is why run_test.py was fine and this was not.
de-code
force-pushed
the
llm-reference-segmenter-engine
branch
from
August 20, 2026 13:35
5d481c0 to
350fc65
Compare
de-code
added
profile:llm_citation
benchmark:smoke
and removed
profile:llm_reference_segmenter
labels
`bool(x) and any(... x.values())` does not narrow `Any | None`, so the bake build's lint-mypy stage rejected it. Two isinstance checks with early returns read better anyway. It escaped local checking because I ran mypy over sciencebeam_parser and tests while every lint target — dev-flake8, dev-pylint, dev-mypy and the Dockerfile stages — also covers benchmarks.
Collaborator
Author
|
Next steps:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
part of https://github.com/eLifePathways/ScienceBeam2.0/issues/143