Ben Yehuda dictionary#3497
Draft
EliezerIsrael wants to merge 19 commits into
Draft
Conversation
Adds scripts/run_linker_save_links.py, a thin wrapper around the canonical link_segment_with_worker task body that runs synchronously (no Celery required). Iterates segments under a target Ref, populating both marked_up_text_chunks (so the React reader wrap_all_entities path displays inline citations) and links rows. Includes two monkey-patches for local-dev quirks: permissive Link._set_available_langs for incomplete VersionState aggregates, and a TextChunk-as-list workaround in MarkedUpTextChunk._validate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Some indexes (e.g. Encyclopedia Talmudit) return TextChunk.text as a list of paragraph strings even when called on a segment-level Ref. The MUTC validator slices that with the span char-range, which yields '[]' instead of the citation substring and aborts every save with InputError. Coerce the list to the segment string by indexing on the segment position before slicing. No-op for indexes that already return a plain string. Also drops the now-redundant local monkey-patch from scripts/run_linker_save_links.py. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ntation
Adds a new DictionaryEntry subclass for the Encyclopedia Talmudit import.
Required attrs (content, rid) match the kovetz/krupnik pattern; optional
attrs (volume, chapter_list, running_header) capture the structured
metadata our parser extracts so it stays queryable.
Unlike other lexicon entries, as_strings() returns one string per
paragraph instead of one joined string — so each paragraph becomes its
own segment in the Reader. Layout mirrors the print encyclopedia: lemma
folds into the first definition paragraph, the chapter list (הפרקים)
gets its own segment, and chapter headings fold into their respective
first paragraphs.
Registered in LexiconEntrySubClassMapping so Lexicon('Encyclopedia
Talmudit') instances resolve to this class.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Ref.all_subrefs(lang=...) was ignoring the lang param in the virtual-node branch — it always called self.text() (default English). For dictionary entries backed by a single-language Version (e.g. Hebrew-only), this returned an empty TextChunk, so size=0 and all_subrefs returned [], which in turn made Ref.all_segment_refs() empty for multi-segment dictionary entries even when the segments existed. Now honour the requested lang first, and fall back to whichever language has content. Surfaces multi-segment refs correctly for any single- language virtual node. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…schema (#3395) When the disambiguator's LLM rejects the sole candidate for a non-segment citation, _update_linker_output_resolution_fields records it by setting span["llm_resolved_ref_but_rejected"] (sefaria/helper/linker/tasks.py), then calls linker_output.save(). But LinkerOutput's spans schema never declared that field, so Cerberus rejected it as an unknown field: InputError({'spans': [{N: [{'llm_resolved_ref_but_rejected': ['unknown field']}]}]}) crashing the link_segment_with_worker task on that path. Declare the field alongside its sibling llm_* diagnostics as an optional, nullable string, matching disambiguator.NonSegmentResolutionResult.rejected_ref (Optional[str]). Additive schema widening; cannot invalidate existing records. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* chore: enable debug mode and adjust sampling parameters * chore: add "anti_talmud_no_book" tag to traceable functions in disambiguator * chore: add CSV test cases for disambiguator functionality * test: update disambiguator_test_set.csv * chore: update disambiguator_test_set.csv to include additional tokens for ambiguous entries * chore: add more rows to "optional" * chore: update disambiguator.py to clarify keyword spelling guidelines * chore: add another correct option * chore: update parameters of test * chore: update disambiguator.py for new model version and debugging tags * refactor: major refactor to reduce code duplication by opus 4.6 * chore: normalize text before sending to llm to reduce tokens * chore: update LANGSMITH_DEBUG_TAG and remove unused expanded query logic * chore: update LANGSMITH_DEBUG_TAG and add highlight feature to candidates * chore: implement caching for _llm_form_prior to improve performance * chore: implement bag-of-words scoring to reduce candidates before LLM processing * chore: add support for Google Generative AI and update token reduction logic * chore: implement caching for debug samples to improve efficiency * chore: reduce DEBUG_LIMIT for sampling in debug mode to improve performance * chore: add script to export documents from MongoDB to CSV with enhanced reference handling * chore: add .db to gitignore * chore: disambiguator scripts * chore: change debug limit * refactor: use base text caching and remove base texts for whole books * chore: update disambiguator scripts * chore: add prior to candidate selection, refactor base text calculation * fix: use English node titles for complex/complex base ref matching and build section-level base refs _get_commentary_base_ref previously matched nodes by internal key (e.g. 'OrachChaim'), causing misses when the key diverges from the English display title (e.g. Prisha → Tur). It also did not handle the case where the base index has an extra SchemaNode wrapper around its content leaf. Changes: - Match nodes by primary English title instead of schema key - After a successful match, append non-default node titles to base_title to form a valid complex ref (e.g. 'Tur, Orach Chayim') - Navigate through any intermediate SchemaNodes to the content leaf, then zip with the leaf's addressTypes[:-1] to produce a section-level (not segment-level) base ref - Return early from the complex/complex branch so the generic section loop only runs for simple bases Add disambiguator_helpers_test.py covering None/empty inputs, non-commentary, multiple base titles, book-level citing refs, both-simple Torah and Talmud cases, both-complex with matching and mismatching titles, complex/simple XOR cases, and missing base_text_titles. * chore: more disambiguator scripts * chore: update tests * chore: update scripts * fix: add high-score Dicta verification step and cache few-shot examples High-score Dicta matches (score >= 8) previously bypassed LLM confirmation, causing false positives when the citation was a vague book/folio reference with no direct quote. Key changes: - Add _verify_high_score: a dedicated LLM check for high-score candidates with a focused prompt warning about book-title citations, bare folio references, and incidental topical similarity - Add VERIFICATION_EXAMPLES: 6 few-shot examples (4 NO, 2 YES) covering book titles, folio refs with/without inline quotes, paraphrases, and whole-book references; placed in SystemMessage with cache_control - Add _get_candidate_text_for_confirmation: pads short candidates with neighboring segments for better confirmation context - Update _llm_form_prior to flag whole-book/chapter citations early - Move examples into SystemMessage with explicit cache_control in both _llm_confirm_candidate and _verify_high_score Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: skip already-resolved spans when dispatching disambiguation tasks Skip non-segment spans that already have `llm_resolved_ref_non_segment` set, and treat ambiguous losers (llm_ambiguous_option_valid=False) as resolved so they are not re-grouped and re-dispatched. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: gate dicta_high_score on phrase proximity to citation (≤60 chars) False positives occurred when Dicta matched a high-score quote elsewhere in the document body rather than adjacent to the actual citation marker. Now only bypass LLM verification when the matched phrase is within 60 characters of the citation in the windowed text, measuring from the nearest edges of the phrase and citation intervals. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: persist rejected candidate ref on LinkerOutput as llm_resolved_ref_but_rejected When _confirm_candidate returns NO (in both the Dicta and fallback search paths), save the rejected candidate ref on the span under llm_resolved_ref_but_rejected. Adds rejected_ref to NonSegmentResolutionResult and threads it through the apply functions so it is written even when no resolution is ultimately found. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: add lang param * chore: strip footnotes if citation is outside footnote * fix: correct phrase-distance calculation and narrow Dicta baseMatchedText to best-matching window - _dicta_phrase_distance: replace abs-distance formula with proper interval logic so any overlap between the resolution phrase and the citation returns 0 (previously a phrase containing the citation could return a large positive distance) - resolution_phrase: when Dicta provides both baseMatchedText and compMatchedText, use Levenshtein sliding-window search (normalized to strip nikkud/cantillation) to find the sub-span of baseMatchedText closest to compMatchedText, then map positions back to the original string so the result is always a true substring Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: update scripts * chore: add levenshtein as requirement * chore: tests * fix: re-enable Dicta auto-approve with data-driven thresholds Replace the disabled `if False` gate with a rule derived from 1,600-row statistical analysis: - section_level=True AND score>=8 AND dist<=10: 99.3% precision (2 misses/280) - score>=15 AND dist<=5 (any section level): 100% precision (0 misses/101) Previous rule (score>=6 / score>=8 with no distance gate) had ~4.3% miss rate and included a risky non-section-level branch at only ~88% precision. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: improve dicta_high_score heuristic * chore: include base text summary in prior and remove base text from future llm calls * chore: add citing ref to prompts * chore: update scripts * fix: move prior to after dicta. remove google * fix: shufflin * feat: allow specifying debug parent ref * chore: add args.limit * chore: upgrade to 4.6 * chore: add export script * chore: add import script * chore: add dicta proxy * chore: remove langsmith tracing for full run * fix: fix dicta proxy server * chore: update review script * chore: add resume flag * chore: hotfix startup probe for task pod * chore: remove old scripts and dicta proxy * chore: create scripts/linker_disambiguator dir * fix: lower ram usage --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
# Conflicts: # envs/staging/helmrelease.yaml # promotions/staging # sefaria/model/text.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A brief description of the PR
Code Changes
The following changes were made to the files below
Notes
Any additional notes go here