Skip to content

docs(design): status and findings after P27 to FX-19 and the newcomer pass - #162

Merged
tunjayoff merged 1 commit into
mainfrom
docs/design-v3-update-6
Oct 6, 2026
Merged

docs(design): status and findings after P27 to FX-19 and the newcomer pass#162
tunjayoff merged 1 commit into
mainfrom
docs/design-v3-update-6

Conversation

@tunjayoff

@tunjayoff tunjayoff commented Oct 6, 2026

Copy link
Copy Markdown
Owner

Docs only: docs/design/ (seven files), plus the push server's address removed from docs/push-channel/README.md and docs/all-sports/README.md (four lines, nothing else changed in those files). No code, test, config, CHANGELOG.md or docs/api/** change.

This is the docs PR after the batch that rule 7 of the implementation plan describes, the sixth after #51, #67, #81, #96 and #138. It records P27 #134, ST-28 #135, FX-18 #139, P28 #140, FX-17 #145, FX-13 #152 and #153, FX-14a #154, FX-15 #155, FX-19 #156 and FX-14b #161; the pull requests that were not plan items (#136, #137, #143, #146 to #149, #159) and the changelog catch-up #158; the first-time-user review of the web UI (2026-10-06) and its follow-up items; and the decisions of 2026-10-03 and 2026-10-06. Statements about code were read at b6caf2f (origin/main when this was written; the branch is rebased on it).

What changed, per document

03-implementation-plan.md

01-storage.md

Sixth-revision header; ST-28 as built in 2.1 to 2.4 and 4.4 (no ratchet, FS_ALLOWLIST with seven more modules, NAMED_EXCEPTIONS table with reasons and removers, src/config_files.py, league_dir_name); the shadow hooks gone (3.5, FX-15); P28's owner slices and odds history (season history written but never pruned); the odds provenance rule; close_data_dir (6.4); old backup scope names deprecated (9.1); the API restore that keeps the job record through the staged copy and one backup step (9.2, #153, with the remaining edge); Store.purge (9.3: what it deletes and keeps, the maintenance lease, the in-place rebuild, the empty bucket folders); Scope.followed still tournament-only for reads; section 12 items 128 to 136; two new risks in 13.

02-services.md

Sixth-revision header; follows always in the follows table (leagues.txt read-only legacy, adopt via PATCH {origin:"api"}); search kinds; team, player and event follow sync (window from the follow's seasons, MAX_LAST_PAGES = 5, lists not stored, FailedListing kinds); the selection chain per event with the player link; P27 rules (unset defaults = registry, required only counts); P28's odds and owner slices (7-day pre-match window, groups, body_key, routes); clear job by tournament/season and delete_data; connection state (per process); export names <label>_<UTC date>_<last 8 of job id> and /exports listing ssc export files; scheduler rules (every from the last run, prune-history off by default); FX-13's job specs and routes, with the recorded spec differing from the request body; FX-18's cancel rules (2.4, 3.4); FX-15's removals; CLI 4.1/4.3 (--only seasons, --follow KIND:ID, status --disk, the doctor's config codes, odds_country). Section 11 items 102 to 117.

04-schema-v1.md

Sixth-revision header; a new subsection "Odds and standings" with blocks for Odds, OddsMarket, OddsChoice, OddsLine and StandingsRow. Their field tables are exactly what render_fields prints for the models, without the generated markers: the document test compares the marked blocks with schema.MODELS, and the models are still in PENDING_MODELS. Moving them, regenerating the tables and tests/golden/schema/* is code: FX-21. The country is only in slice meta (opt-in), the provider in meta.provider_id. The heading ## 9. Open questions stays (its rename moved from P28 to FX-21).

05-web-ui.md

Sixth-revision header (the review at b567409, "the review wins" where it disagrees, the 12-task recheck of #161); as built for FX-14a (the applied rename table in TR/EN, "Add league" entry points, Help and glossary, toggletips, getting-started card, finish toasts, grey "not tried" state, localized typed confirmation) and FX-14b (team/player/match follows, seasons before the first download, checklists, exports, real restore, per-league delete, adopt, /status.connection, job kinds by spec); classic views gone (/classic redirects; client.ts removed; decision 22 done); browser floor Safari 16.4 / Chrome 111 / Firefox 128; vue-i18n 11 without the JIT define. Section 7: every G1–G24 with its state at b6caf2f, new G25–G33 with owners (FX-20, P30, or section 17 after 3.0.0). Section 11 items 27 to 53.

00-platform.md, README.md, the two push-channel pages

00: sixth-revision paragraph, state column of section 1, notes in 3, 5, 6 and 10, the live validation's wider scope (11), two rows in 13. README: revision line, status (90 items), decisions. The push server's address is replaced by a neutral wording on the four lines FX-15 named; it still stands on one line of docs/all-sports/progress.md, which FX-16 now owns (section 16).

Decisions recorded (section 13)

  • Owner: D1 — the package becomes sofascore_scraper; REN-1 goes ahead when no other branch is open. Team, player and single-match follows work in 3.0.0. A type-ahead search like the site's (FX-20).
  • Delegated: S11 closed; the four old backup scope names deprecated in 3.0.0 and removed in P30; the two ST-27 rules kept as built; the scheduler's every counts from the last run; the odds country opt-in ([client] odds_country), the provider id always recorded; prune-history off by default; dependabot ignores pydantic-core with a pydantic group of its own (manual constraints step).

Decisions that need the owner

  • The three slice proposals of feat(sports): each sport requests the detail slices SofaScore offers for it #121 (after the live validation; FX-16).
  • The direct step of the live validation (approval in that session), and whatever the validation raises.
  • The tag.
  • Proposed without asking, easy to undo: FX-21 as a new item; REN-1 before the live validation (FX-16 after it); FX-20's scope widened by the small UI leftovers listed above; the owners of the new section 16 and 17 rows (several UI reads of fields the API already offers are left for after 3.0.0 in section 17).

Feedback accounting

Input: lines 3 to 25 of ~/.local/share/sofascore-orch/design_feedback.txt (everything after "[orchestrator] MARKER-4"; the last line processed is line 25, "[FX-14b #161] merged …"), the "Design mismatches", notes, "Needs live validation", "Not done", "API gaps" and "Finding" bullets of #134 to #161 (168 units), and the follow-ups of the first-time-user review (20 units). Scratch ledgers (not committed) hold 211 units; a script finds a phrase of each placement in the documents: 211 of 211 units placed (674 verified placements), none unplaced. Work notes (how a PR was run) are in the closing paragraph of section 17.

Where the code at b6caf2f contradicts a line, the documents follow the code, among them: export names end with the last 8 characters of the job id; P28 merged before FX-13 (dependency reversed); P28 did not edit src/store/history.py and the first caller of prune is FX-15's task, which prunes event history only; ST-28 already removed league_stats, system_stats and summary_paths; FX-14a/b did not remove the retired save_empty_rounds control (now FX-20).

Checks

  • Plan consistency script (scratch, items/edges/levels/states/owners): origin/main 86 items, 194 edges, 0 problems; this branch 90 items, 205 edges, 0 problems (one false positive, "P3", a decision name).
  • tests/test_schema_v1.py and tests/test_store_catalog.py::test_schema_file_is_the_ddl_printed_in_the_design: 594 passed. These are the only tests that read docs/design/.
  • Table rows checked for their closing pipe; grep for the push server's address in docs/design, docs/push-channel and docs/all-sports/README.md: none.

Not verified

  • No product code was run, no SofaScore request, no browser. Measurements and CI times are quoted from the pull requests.
  • 01, 02/04, 05/00/README, the briefs region of 03 (As built, new briefs), sections 6 and 15–17 of 03 were drafted by helper agents from a shared brief of mine, each checking its statements against the code at b6caf2f; I wrote the rest of 03 (status, sections 1–5, 7–9, 11, 13, 14, 18), checked their reports, the ledgers and the plan's consistency, and reconciled owners. I did not re-read every changed line.

Section 18 of the plan (release readiness), verbatim

State on 2026-10-06, at b6caf2f: 85 of the 90 items are merged or done, one is in progress (FX-20), and
four are to do (FX-21, REN-1, FX-16, and P30, which comes after the release). What remains before the
v3.0.0 tag, in order:

  1. FX-20, in progress. A type-ahead search like the site's (the owner's decision of 2026-10-06: local
    suggestions first, a SofaScore search after two characters with a short pause, stale requests cancelled,
    answers kept for the session and counted in the request budget, the same in quick search), one search
    across tournaments, teams and players, the follow's name on a job ("Takım research: SofaScore push channel — coverage, lag, cost, direct connection (Talimat 06+07) #42" today), and small gaps of
    the UI (live watching offered for player follows, which ssc watch skips; the long sport list of
    Settings › Data; the retired fetch.save_empty_rounds control and the old text of
    fetch.only_finished; Health without the scheduler's next runs).
  2. FX-21, the schema follow-up. The five models of P28 (Odds, OddsMarket, OddsChoice,
    OddsLine, StandingsRow) move from models.PENDING_MODELS into MODELS, and Odds, OddsLine and
    StandingsRow into RECORDS; the blocks this revision added to 04-schema-v1.md become generated,
    test-checked tables (REGEN_SCHEMA_DOC=1); the JSON Schema golden under tests/golden/schema/ and what
    describe schemas prints are regenerated; the new slice keys join Slice.key's known values. It can run
    next to FX-20 (no shared file).
  3. REN-1, when no other branch is open (decision D1, owner, 2026-10-03): the package becomes
    sofascore_scraper, in one mechanical pull request, with the console-script entry and the Dockerfile
    (section 14). It goes before the live validation, so that the validation runs the code that is
    released.
  4. The live validation (Talimat 07). Once, at the end, in a busy match window, by the coder session
    with the request budget; its direct step needs the owner's approval given in that session. It checks:
    • the detail slices of every sport: one finished and one live match page of each of the 21 sports, with
      the detail endpoints each page requests and their status; this regenerates
      tests/fixtures/sport_slices/evidence.json (python tests/sport_evidence.py) and is FX-16's evidence,
      including the sports without usable evidence today (American football, Aussie rules, badminton, table
      tennis, rugby, minifootball);
    • status and scores of the new sports: the live status codes of the 18 new sports, the ice hockey
      shoot-out field, table tennis /event with every set and codes beyond 12, volleyball's
      defaultPeriodCount, a legs-only darts /event, the inning, cricket-break, MMA and e-sports codes not
      seen yet, and the near-end rules of the new sports;
    • the slice phases: which slices exist before kick-off, so that SliceSpec.phases can be narrowed and a
      pre-match selection can be asked for (P27);
    • odds and non-match data (P28): which odds provider ids answer without login (every sample used 1, the
      default of [client] odds_provider); the shape of winning_odds and season_odds (only 404 samples
      so far); player_statistics and standings home (catalog only); the WTA id of rankings (5 is the
      ATP page's); the sports in which each owner slice exists; where useful, better fixtures for
      tests/fixtures/p28/;
    • team, player and match follows (FX-19, FX-14b): /search/all (tournament hits, an empty answer as
      404 or an empty list, other result types) and /search/unique-tournaments/{q} for an unknown name;
      a team's events/last/{n} and next/{n} (page size, hasNextPage, next/0 as 404 for a team
      without fixtures, sports beyond the nine of the catalog); a player's events/last/{n} (the event id
      field, other sports; no next is known); the cost of MAX_LAST_PAGES = 5 for a team follow with
      all; reading upcoming matches by /event only; better fixtures for tests/fixtures/fx19/ where
      useful;
    • the type-ahead of FX-20: /search/all with short prefixes, and the requests a typed name costs;
    • the connection state of /status (FX-19) on the browser-first path, and the stop of a download on the
      async bridge path with a real browser (FX-18 was checked against fakes only);
    • the two push sources: page, and, with the owner's approval in that session, direct;
    • an end-to-end run on a copy of real data: ssc sync (a league, a team, a player and a match follow),
      export (every dataset and format), backup and a restore, and serve with the web UI's twelve
      newcomer tasks.
      What the validation finds becomes fix items before the tag.
  5. FX-16: the owner's decision on the three slice proposals of feat(sports): each sport requests the detail slices SofaScore offers for it #121 (pregame_form optional for
    football, basketball and tennis; tennis lineups and incidents not requested; tennis point_by_point
    required), with the evidence of the validation, and any row of another sport that the new evidence
    changes.
  6. The Docker image. It has not been built since P25 changed its entrypoint, because release.yml
    builds and smoke-tests it only when a tag is pushed. Build it once from the release commit and run
    docker/smoke-test.sh (offline, --network none: --version, the HEALTHCHECK, /health, the web app
    and a headless Chromium start), so that the tag does not find the first failure.
  7. The version bump. pyproject.toml still says version = "2.0.0" (pyproject.toml:3 at
    b6caf2f); src/version.py reads it, and the web UI shows it (v2.0.0 in the rail,
    frontend/src/app/SideRail.vue:71). The release pull request sets 3.0.0, and the checks of release
    point D (section 8) run on that commit: a green Linux CI and a local run of the full suite (Windows and
    macOS best-effort), the Docker smoke test, every behaviour change in the changelog, direct off in
    every default and its warnings in the README. The same pull request corrects two texts that no item
    owns: docs/deploy/ describes only the systemd timer, not the in-app scheduler of P29, and
    .github/workflows/audit.yml:42 still names pandas (section 16).
  8. The changelog close. docs(changelog): entries of #115 to #156 #158 added the entries of feat(sports): five set-based sports — volleyball, badminton, table tennis, padel, snooker [SP-2] #115 to feat(follows): team, player and match follows that download; editable web follows; per-league delete [FX-19] #156. The release pull request adds the
    entries of what merged after it (feat(web): every function in the new UI — team/player follows, seasons, data types, exports, restore; classic views removed [FX-14b] #161, FX-20, FX-21, REN-1, FX-16 and the fixes of the validation),
    states the deprecations of 3.0.0 (the four old backup scope names, the legacy /api routes and flags
    that P30 removes), and turns [Unreleased] into [3.0.0] with its date: release.yml refuses a tag
    whose version has no ## [3.0.0] section (scripts/release.py notes).
  9. The tag. v3.0.0 is the owner's (00-platform.md section 1, "Merging"); release.yml checks that
    the tag equals the version in pyproject.toml, runs CI, builds, smoke-tests and pushes the image and
    creates the GitHub Release.

Decisions the owner takes on the way: the three slice proposals of #121 (after the validation; FX-16),
whatever the validation raises, approval of the direct step in the validation session, and the tag.
Proposed by this revision without asking, easy to undo: FX-21 as a new item, and REN-1 before the live
validation. P30 follows once 3.0.0 has shipped with its deprecation notices for one release.

@tunjayoff
tunjayoff merged commit bb7cf4f into main Oct 6, 2026
3 checks passed
@tunjayoff
tunjayoff deleted the docs/design-v3-update-6 branch October 6, 2026 20:56
This was referenced Oct 6, 2026
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tunjayoff