feat: honest source contract for tweet ingestion#136
Merged
Conversation
One core, resolve_thread / resolve_tweet, derives everything a geoloc tweet resolves to: coordinates (with quoted-text fallback), the footage source (quote, then explicit x / telegram / youtube link, then self), the source vs proof media split, title, proof text and dates. The human import (parse) and the machine detection (detect) become thin mappers over it, so the two paths cannot drift. Acquisition now carries the signals the resolution needs: QuotedTweet and host-classified source links from both syndication and the X archive, an optional syndication chase of third-party quotes and linked source statuses, and a hybrid media fetcher (archive disk plus X CDN for chased footage). Assemble persists the source and proof roles and embeds proof images in the proof document. The superseded one-off extractors are removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
A taxonomy of geolocation-tweet shapes, a sanitizer that distils real tweets into committed synthetic fixtures, and a two-tier recall harness: golden fixtures run in CI; corpus recall runs locally over the gitignored sample under backend/datasets/ (make typology-weights regenerates weights.json and fixtures.json). Baseline on a 480-tweet external corpus: 84.4% recall at 99.7% precision, with the image-only share quantified. planning/next.md switches to the neutral corpus wording. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
A detection only carries what its tweet declares. resolve_source drops the self-source fallback (quote, then footage link, else no source), split_media always routes the thread's own media to proof, and the epoch sentinel dies: source_url, source_posted_at and event_date are nullable end to end (ResolvedTweet, DetectedGeoloc, ParsedTweet, the event / search / duplicate / import reads). events.source_url and events.source_posted_at go nullable in the DB with ck_events_source_url_status requiring a source on requested and geolocated rows, and the geolocate transition returns source_url_required when promoting a draft without one. The archive reader now maps video and animated_gif media (highest-bitrate mp4 variant, fail-soft on a missing file). Existing tests updated to the contract, plus new CHECK and promotion-gate tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
tests/ingest_contract pins the brick per typology: 12 synthetic syndication fixtures with expected resolutions, a consolidated archive builder, and offline chase coverage (28 cases, full suite at 750). The frontend adopts the nullable source contract (api-types regeneration, the SourceLabel "To confirm" state, null-safe date formatting that would otherwise resurrect the 1970 epoch, form prefills). New docs/ingestion.md documents the contract; api.md and data-model.md drop the stale always-present claims; tracker and changelog updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
A profile link (x.com/handle) listed before the real status link won the source slot, producing events sourced to a profile with no media. classify_source_host now returns "x" only for status URLs, via the shared _X_STATUS_URL_RE that the archive chase reuses; a profile link falls through to "other". New x_profile_link typology fixture and resolution tests pin the behavior, and ingestion.md states the rule. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
The Subject tooltip described the camera position ("where the footage
was filmed"); it now states the actual semantics: the ground location
the footage shows. section_location repeated the same mix-up.
source_posted_at claimed a post always has a time, false since the
honest source contract (NULL without a dated source), and status now
covers a rejected detection, not only a withdrawn request.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: vidit-admin <admin@vidit.app>
Existing media tiles in the edit form were inert: a video could not be played during review. FileManager items now take an optional viewContent: the tile opens a lightbox (backdrop click, Escape, or the close button dismiss it) while the remove control stays a plain sibling, so view and remove never collide. MediaManager mirrors the read view treatment: video with controls and the poster-frame fragment, image enlarged uncropped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
The archive-import endpoint ran the backfill without chase, so tweets referencing their footage by quote or a Source status link imported with no source media. The analyst on-ramp now chases via syndication (fail-soft: an unreachable status degrades to a link-only source). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
Only Conflict and Capture source turned both their label and their field block red on a failed submit; every other required field only outlined the block. FORM_INVALID_LABEL (previously an unnamed literal duplicated in TagPicker) is now the shared label-side companion to FORM_INVALID_FIELD, SectionHeading takes an invalid prop, and Title, event date, source posted, source URL, coordinates, Subject, source media and proof all mark uniformly. Palette entry added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
The tweet-import pre-fill posted proof images to the removed POST /events/proof-images endpoint (405), so the seeded proof had no images. Imported images now follow the upload-at-publish contract: downloaded through the existing import proxy, named with safeProofFilename, staged as proof files, and seeded as placeholder:// image nodes. The editor resolves initial proof files to data: URLs for preview (object URLs die under Strict Mode's mount/cleanup/remount replay). Seed template em dashes replaced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
Halves the per-page card count so each page's source-media previews load faster; the sidebar count probe and other lists are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
A link to the analyst's own post is a cross-reference, not footage: resolve skips X status links matching the thread owner's handle, and the archive chase excludes status ids present in the archive itself. No positional guessing: a single remaining third-party footage link (deduplicated by URL, all hosts) is the source; several distinct candidates are ambiguous and leave the source empty for review. The one-source-media persistence cap is pinned by a test. New self_reference_link typology fixture mirrors the real-world case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
A sole t.me footage link is chased on the archive import paths: the embed's post date fills source_posted_at and, when the embed serves it, the footage lands as source media; sensitive posts degrade to link plus date, fail-soft throughout. TelegramFootage rides the record parallel to the quote and never overrides quote precedence or the ambiguity rules. The media allowlist admits the Telegram CDNs by strict dot-boundary suffix match, and the fetch guard only ever requests a public t.me channel post URL. The human single pre-fill stays chase-free. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
Chase candidacy now shares one source of truth with resolve (footage_candidates): host-gated X status extraction (an archive.org citation embedding a status path no longer matches), cross-host ambiguity honored by the chase (an X status plus a Telegram link chases nothing), chased self-links discarded by owner handle, and candidates deduplicated by status id across x/twitter and URL variants. The CDN chase fetch streams under the shared 110 MB cap the media proxy uses. Non-image proof media is skipped at persist so no unreachable rows land. The human pre-fill stops guessing: the response exposes the resolved source_posted_at (quote or Telegram date, never the OP's), a sourceless re-import clears the stale URL, and identical imported proof images no longer collapse into one placeholder. Telegram media capture wins over footer chrome. Docs aligned (self-quote carve-out, Telegram date exception, required source date wording) and added dash punctuation swept. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tweet ingestion now rests on one honest brick: a tweet id resolves to a pre-filled event where every field is either right or empty, never guessed.
The brick and its contract
resolve_thread/resolve_tweetare the single resolution core; the human import, the machine detection, and the archive backfill are thin mappers over it, so they cannot drift on coordinates, source, dates, or media (8d5d466).source_url,source_posted_atandevent_dateare nullable end to end,ck_events_source_url_statusrequires a source onrequestedandgeolocatedrows, and promoting a draft without one returns 400source_url_required(bf3d4d7).source_posted_atand, when the embed serves it, the footage becomes the source media; sensitive posts degrade to link plus date. The media allowlist admits the Telegram CDNs by strict dot-boundary suffix match (2d400df).Test surface
tests/ingest_contract: 13 synthetic syndication typologies with expected resolutions, plus a consolidated-archive end-to-end run, fully offline.tests/typology: sanitized recall harness for the coordinate parser over a gitignored external OSINT corpus, re-runnable viamake typology-corpus/make typology-weights(26b7f08).Frontend
SourceLabel, null-safe date formatting (anew Date(null)would silently resurrect the epoch), null-safe forms (22a83e6, bb8eca1).Docs
docs/ingestion.md: the contract, the recognized signals, what is never deduced, the bot input format, the archive-accepted formats.api.mdanddata-model.mdupdated to the nullable semantics and the previously undocumentedsource_url_requirederror code.Test plan
make hygiene,make gen-api-typesidempotent🤖 Generated with Claude Code