Rebuild training data B-/I- from element boundaries - #700
Merged
Conversation
AbstractTrainingTeiParser assigned the entity prefix only when the label changed, and skipped whitespace-only text nodes without touching the previous label. Two sibling elements sharing a label, with only whitespace between them, therefore came back as one entity: the second element's start was lost, and generate_delft_data maps our B- to GROBID's I- start marker, so a start the parser never emitted could not reappear in the artifact. Where a boundary did survive it survived incidentally - any non-whitespace separator is labelled O, which changed the previous label and restored the next start, so two idno elements separated by a comma were kept apart and the same pair separated by a space were not. is_start on TeiTrainingText already means "first text run in this element", including across <lb/>, so the guard is all that had to go. A whitespace-only run no longer spends the flag either, which is what a single space before a nested <label> used to cost. The semantics is one entity per element occurrence: text returning to a labelled parent after a labelled child continues the parent's entity. That is the inverse of what the writer does, where a B- opens a new sibling element, and it keeps <paragraph> whole across the markers the fulltext model nests inside it. GROBID's SAX parsers declare begin inside writeField and so start a field per text run, which would split a paragraph at every citation marker. Over the committed generated corpus the reference segmenter gains 1786 reference starts - 624 to 1678 for ore, 911 to 1643 for scielo_preprints-jats - and the header model 10. Citation, affiliation-address and segmentation are unchanged, every label and every document. On a collapsed document the delft output differs in the label column alone: 152 <reference> become I-<reference> and no token moves. The citation identifier test asserted the merged value as known behaviour; it now round-trips both identifiers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
part of https://github.com/eLifePathways/ScienceBeam2.0/issues/147
AbstractTrainingTeiParser assigned the entity prefix only when the label changed, and skipped whitespace-only text nodes without touching the previous label. Two sibling elements sharing a label, with only whitespace between them, therefore came back as one entity: the second element's start was lost, and generate_delft_data maps our B- to GROBID's I- start marker, so a start the parser never emitted could not reappear in the artifact. Where a boundary did survive it survived incidentally - any non-whitespace separator is labelled O, which changed the previous label and restored the next start, so two idno elements separated by a comma were kept apart and the same pair separated by a space were not.
is_start on TeiTrainingText already means "first text run in this element", including across , so the guard is all that had to go. A whitespace-only run no longer spends the flag either, which is what a single space before a nested used to cost.
The semantics is one entity per element occurrence: text returning to a labelled parent after a labelled child continues the parent's entity. That is the inverse of what the writer does, where a B- opens a new sibling element, and it keeps whole across the markers the fulltext model nests inside it. GROBID's SAX parsers declare begin inside writeField and so start a field per text run, which would split a paragraph at every citation marker.
Over the committed generated corpus the reference segmenter gains 1786 reference starts - 624 to 1678 for ore, 911 to 1643 for scielo_preprints-jats - and the header model 10. Citation, affiliation-address and segmentation are unchanged, every label and every document. On a collapsed document the delft output differs in the label column alone: 152 become I- and no token moves.
The citation identifier test asserted the merged value as known behaviour; it now round-trips both identifiers.