fix(0228): scale the pivot by the measured USDC/USD rate, pivot-leg reset tooling, XLM snapshot - #311
Open
adamkoot wants to merge 9 commits into
Conversation
The pivot tier's ref_usd is a price in USDC, not in dollars, so every XLM- and USDT-quoted candle carried the USDC = $1 assumption task 0268 had just removed from the USDC leg itself. USDC closed at 0.9681 on 2023-03-11, so the stored value is ~3.2% high across the whole pre-epoch pivot population. pivot_sql now ASOF-joins prices.usd_rate for canonical USDC at the candidate's bucket END and writes close_usd as close x vwap(ref/USDC) x usdc_usd. The two method-specific rate legs are NESTED, as queries_ch::peg_series_sql proves, so a valid oracle reading wins outright without an argMax across methods. A bucket with neither rate is left unpriced rather than written as 0 x close. The candidate scan moved into a subquery because ASOF needs a materialized column for its inequality, which moves the partition bound onto the bare column and reorders the first two binds.
…d runbook Appendix C The scaled pivot only reaches rows still at close_usd = 0, so the ~118 M pre-epoch XLM-quoted and ~1.6 M USDT-quoted candles already carrying an unscaled value are inert. This adds the operator mechanism that re-opens them. UsdResetSpec gains require_pivot_usdc_rate, which appends the SAME external_rate_day_pred the 0268 mode uses and nothing else: a pivoted row never carried the peg tier's close_usd = close signature, so including it would select nothing. That also means the mode cannot be a fixed point across runs — it is value-idempotent instead, which is a recorded deviation from BRIEF decision C. The mode refuses the canonical USDC leg (that one is 0268's), a non-pivot leg, an oracle-shadowed span, a zero day-set, an empty window, and the two modes together. It is not gated on hourly rates: that refusal guards an irreversibility only the par signature creates. Appendix C carries the preconditions with their STOP conditions, the per-leg --reset-not-before derivation, the dry-run gate, the baseline, the expected runtime and the rollback. post_run_0228_it.rs is the operator's after-check, expected to fail until the campaign runs.
XLM's 52,607 Reflector readings live only in oracle_prices, which the dark
cleanup worker's 13-month policy will eventually prune, while usd_rate — never
pruned — holds none of them. peg_identities() is exactly canonical USDC by
design, and correctly so: XLM is not a peg.
measured_identities() is a second set beside it, named for what its members are:
measured, non-peg references. Its doc comment carries the identity evidence and
a test calls reflector_key_to_identity("XLM") so the claim is code rather than
prose, to task 0267's standard.
The two sets go to populate_usd_rate_from_oracle as SEPARATE calls. Its task
0139 identity pre-pass runs over the whole slice and returns before writing
anything for any identity, so appending XLM to the peg call would let a
collision on XLM's asset_id silently stop USDC's snapshot too.
The copy is gap-filling rather than watermarked, so the first run after deploy
copies all existing readings and no backfill tool is needed.
…ator checklist Implementation notes for the three commits, design decisions (D-01..D-09 from plan, 11 emerged), the D-06 deviation from BRIEF decision C with its evidence, the broken/modified test list, the acceptance-criteria status split between the branch and the operator run, Appendix C's checklist and the spawn list. Records that the #[ignore] ClickHouse tests were written but not run — no ClickHouse is reachable on this machine — and keeps the read-only prod probe that retired RESEARCH assumption A2.
Review WR-01: coarse-repair gated the --pivot-window-s minimum-width guard behind !dry_run, so Appendix C's "dry run first" would accept a window the real run then refuses. The window guards now run in dry-run mode as well; only the snapshot guard stays real-run-only, since a dry run discards nothing. Inherited from the 0182 shape. The task file records the review and verification results.
Two defects found by the prove run against ClickHouse 26.3.10.60. post_run_0228_it decoded a Nullable(Float64) ratio into f64 — the reference vwap's nullIf makes the whole median nullable — so RowBinary landed one byte off and the acceptance gate reported a carried factor of -3.9e230 on a table the tool had just repaired to 0.9681. It could never have passed on prod. The field is Option<f64> now, as in the 0268 twin; NULL behind matched rows is a finding that names the harness, and CI pins it. The ResetRequiresExternalRates refusal both --reset-require-* flags promised was unreachable through coarse-repair: an unloaded series does not fail the month enumeration, it empties it, so the run ended green with "0 month(s)". CoarseRepairDriver::run now checks it first, dry run included, through the one fn the pass already uses.
The #[ignore] ClickHouse suite was recorded as "written, NOT RUN"; it has now run against the prod build, together with a hand-built Appendix C campaign on all six tables and the XLM snapshot path. The task README carries the results, the two defects (Issues 8-9), the FREEZE-after-refusal caveat (Issues 10, spawn list 4) and decisions 21-22. Appendix B and C say what the no-rates refusal now actually does, and C warns about the snapshot a refused run leaves behind.
Three code-review findings, each reproduced before the fix. OracleUsdRatesSnapshotted summed the peg and measured snapshots. XLM lands rows every pass, so a stalled USDC snapshot, the pivot's only post-epoch rate and non-fatal by design, could never read zero. The series is peg-only again and XLM gets OracleMeasuredRatesSnapshotted. The pivot-leg gate checked pivot_ids() alone, while the pivot only runs when canonical USDC resolves in prices.assets. With USDC missing there and its rates loaded, every gate passed and the reset zeroed a leg no pivot refilled (rows_reset 1, rows_enriched 0). It now requires can_pivot(). Both leg refusals ran only inside the per-month pass, which a dry run never builds, so rehearsing the wrong leg ended green. The driver now runs them, with the no-rates check, before enumerating months.
Appendix B and C now list each reset refusal where it actually fires: before connecting, before month enumeration, or inside the pass, which is where the FREEZE-left-behind caveat still applies. The task README records the review (Issues 11), decisions 22-24 and the superseded WR-02 note.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pivot_sqlnow multiplies the ref/USDC vwap by the measured USDC/USD rate fromusd_rateat the bucket end (oracle over external via two nested method-specific ASOF legs, noargMax); a bucket with no rate is left unpriced, neverclose × 1. Fixes the last stored prices still assuming USDC = $1 after 0268 — every pre-epoch XLM- and USDT-quoted candle (2023-03-11 stored +3.2 %).--reset-require-pivot-usdc-rate) for the re-enrichment campaign: one predicate at three sites, five refusals (USDC leg, non-pivot leg, oracle shadow, empty day set, both modes at once), value-idempotent,price_ohlcv_1mstill refused; runbook Appendix C and thepost_run_0228_itfalsifier. Dry run now runs the pivot-window guard too (review WR-01).measured_identities()beside an untouchedpeg_identities()snapshots XLM Reflector readings intousd_rateasoracle/hops 0through a second, separately non-fatal snapshot call; the identity evidence is a test.notes/) rescoped the task: AC 1 as filed is met (pivot vs Reflector ±35 bps), the oracle tier already prices XLM legs inside its window, retention is the dark cleanup-worker's policy, not a TTL.-D warningsclean. The#[ignore]ClickHouse suite HAS run, against a rootlessclickhouse-common-static-26.3.10.60(the prod build):ch_enrich_it44/44,usd_rate_population_it5/5, plus a hand-built Appendix C campaign with the realcoarse-repairbinary on all six tables (0.588 → 0.5692428 = ×0.9681 on 2023-03-11, controls untouched, rerun value-idempotent at version 5) and the XLM snapshot path through the real writer.afe587e,c768b27): thepost_run_0228_itacceptance gate decoded aNullable(Float64)factor intof64and reported ±1e230 on a correctly repaired table — it could never have passed on prod (nowOption<f64>like the 0268 twin, NULL is a harness finding, CI-pinned); and theResetRequiresExternalRatesrefusal both--reset-require-*flags promised was unreachable through the CLI (an unloaded series empties the month enumeration →exit 0, "0 month(s)") —CoarseRepairDriver::runnow checks it before enumerating months, dry run included,#[ignore]-tested and confirmed with the binary. Task README Issues 8–10 and runbook Appendix B/C updated; a refused run still leaves its FREEZE behind locally (Issues 10, spawn list 4, prod unaffected)./code-reviewon the branch: 3 findings, all fixed in9fd74f3/fe452ab, no defect inpivot_sql. Metric: the snapshot counts were summed intoOracleUsdRatesSnapshotted, hiding a stalled USDC snapshot behind XLM's rows — now per set (peg series unchanged in meaning, newOracleMeasuredRatesSnapshotted). Gate: the pivot-leg reset passed with canonical USDC missing fromprices.assetsand zeroed a leg no pivot refills — now requirescan_pivot(). Dry run: both leg refusals (0228 and 0268 modes) ran only in the per-month pass — now before month enumeration.ch_enrich_it46/46 on 26.3.10.60, 725 CI tests.active, a CHORE follows ondevelop. Deploy isPrices-production-EventBridge(enrichment + oracle Lambdas), not Compute.