diff --git a/docs/runbooks/0286-reingest-history.md b/docs/runbooks/0286-reingest-history.md index ff07fb85..0f9cfb02 100644 --- a/docs/runbooks/0286-reingest-history.md +++ b/docs/runbooks/0286-reingest-history.md @@ -404,6 +404,352 @@ NOT include. For scale: task 0268's campaign re-priced 9.94 M candles in ~24 min 0228 estimated ~190 M candles at ~4 h 40 m. This is all of them, and it runs on the same worker under the same concurrency. +**7b-2. Re-price the coarse tiers. The drain above never reaches them.** The +enrichment worker prices `price_ohlcv_1m` and nothing else +(`CLICKHOUSE_TABLE: 'price_ohlcv_1m'`, +`infra/src/lib/stacks/eventbridge-stack.ts:483`). The coarse sweep covers only +the current and the previous month (`COARSE_SWEEP_LOOKBACK_MONTHS: '2'`, same +file `:627`). The rollup MVs re-aggregate only a recent window (`rollups.sql`). +Every coarse row that §4g and §7a rolled was built from a `1m` that had not been +priced yet, so it was written with `close_usd = 0`. Without this step every +15m/1h/4h/1d/1w/1M row older than two months keeps that zero for good, and +~130 unguarded `argMax(close_usd, …)` sites read it as a price. + +The fix is the ordinary fill-the-zeros mode of `coarse-repair` +([`repair-coarse-usd-values.md`](repair-coarse-usd-values.md), Steps 1–5), run +over the whole history on all six tables. It does not read `1m`. It prices each +coarse row from that table's own reference candles, with the same tiers as the +worker, including the measured USDC rate on the pivot. + +- **Start only when the `1m` drain is finished.** Both passes load the same + shared cluster, and 7c/7e measure the finished state. **The frontier gauge + alone cannot tell you this.** `EnrichmentFrontierMonthsPending` counts only + months that are NOT marked `exhausted` in `prices.enrichment_frontier` + (`frontier.rs`, `months_to_sweep`). Almost every historical month was marked + `exhausted` before the re-ingest, and dropping and re-ingesting its `1m` does + not clear the mark. The worker re-opens such a month only once its `swept_at` + is 7 days old (`ENRICH_HISTORICAL_RECHECK_SECS`), at most 4 per hourly run. + So a month re-ingested less than a week after its last re-stamp reads + `exhausted` while all its `1m` rows sit at `close_usd = 0`, and the gauge + reads `0`. + + "Finished" therefore needs two readings. Set `T7` to the UTC time the last + month's §4e re-ingest finished (from its log). Every `1m` month swept before + `T7` is suspect. + + **(a) [local machine] The gauge reads `0`.** No month is missing or + `pending`: + + ```bash + aws cloudwatch get-metric-statistics \ + --namespace Prices/Enrichment --metric-name EnrichmentFrontierMonthsPending \ + --dimensions Name=Environment,Value=production \ + --statistics Maximum --period 3600 \ + --start-time "$(date -u -d '-3 hour' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" + ``` + + You need at least one datapoint, and every datapoint must read + `Maximum` = `0`. An empty `Datapoints` list means the drain has stopped + publishing. It does not mean the drain is done. + + **(b) [fishuser-hero, after ①] Every month was confirmed after `T7`.** This + query must return **no rows**: + + ```bash + T7='2026-MM-DD hh:mm:ss' # UTC, the last §4e re-ingest's end + chadmin " + SELECT month, CAST(state AS String) AS state, swept_at, zeros_seen + FROM prices.enrichment_frontier FINAL + WHERE tbl = 'price_ohlcv_1m' AND month <= $END + AND (state != 'exhausted' OR swept_at < toDateTime('$T7')) + ORDER BY month + FORMAT PrettyCompactNoEscapes" + ``` + + The listed months still need a sweep. The worker gets to them without help: + each re-opens up to 7 days after its last stamp, and the drain then works one + month per hour. To skip the 7-day wait, rewind them as `prices_admin`. This + is the rewind the frontier's own docs prescribe ("an operator rewind needs a + `DELETE`"). A month with no frontier row goes back into the queue, so the + gauge in (a) rises and then drains: + + ```bash + chadmin "ALTER TABLE prices.enrichment_frontier DELETE + WHERE tbl = 'price_ohlcv_1m' AND month <= $END AND swept_at < toDateTime('$T7')" + ``` + + Re-run (a) and (b) until both pass. A month the drain swept after its own + re-ingest but before `T7` gets swept once more. That costs one hourly slot + and a cheap count, because its zeros are already gone. + +- **Pass NO `--reset-*` flag.** A reset discards stored values. Here nothing + is wrong, only missing, and 7d already rules the 0228 reset out. Every refusal + the reset modes carry is skipped when no `--reset-quote-asset-id` is given, + and none of them applies to this step. That covers `--snapshots-verified`, + the `--pivot-window-s` floor, the plain-reset-on-a-pivot-leg refusal and the + task-0208 epoch check (`assert_reset_is_admissible` returns at once). +- **Run with `--skip-snapshot`. The tool's own snapshot cannot be used as a + rollback here.** That snapshot is + `ALTER TABLE prices. FREEZE PARTITION WITH NAME 'repair_0114_…'` + (`repair.rs`, `freeze_partition`). It lands in the CH host's `shadow/`, and + the only way back from `shadow/` is `ATTACH PARTITION … FROM '/path/'`. That + needs filesystem access on the host, which no mTLS identity has, including + `prices_admin` (task 0286, "Rollback amendment"). Skipping it is safe for + two reasons. This mode only writes rows still at `close_usd = 0` and never + discards a value. And §4g/§7a can rebuild every month's coarse rows from its + re-ingested `1m`, so the tool's warning ("only safe when the 1m source can + rebuild this coarse table") is satisfied. **Exception:** a month recorded as + un-re-ingested (§5, precondition 9) whose `1m` is gone. Its coarse rows are + the only copy. Back them up in step ①b, before the dry run. + +- **Go from fine to coarse, one table at a time:** `15m`, `1h`, `4h`, `1d`, + `1w`, `1M`. The `1d→1w` (60 d) and `1d→1M` (400 d) MVs re-roll recent buckets + from `1d`. Once `1d` is priced, those re-rolls carry its rate rather than a + zero. + +**① [fishuser-hero, in `tmux`] Build, set the range, and define `chadmin`.** +Every step below runs in this one shell. `coarse-repair` writes as +`prices_writer`. Every read goes over mTLS as `prices_admin`, the uncapped +identity that already serves as the 0286 script's `--reader-cert`. Nothing here +goes through the CH host's own shell. + +```bash +ssh fishuser-hero +tmux new -s coarse0286 + +cd ~/stellar-prices-api && git checkout develop && git pull --ff-only +cargo build --release -p enrichment-worker --features aws-mtls --bin coarse-repair + +chadmin() { curl -sS --fail-with-body --cacert ~/prices-mtls/ca.crt \ + --cert ~/prices-mtls/prices-admin-production.crt \ + --key ~/prices-mtls/prices-admin-production.key \ + "https://ch.sorobanscan.rumblefish.dev/" --data-binary "$1"; } +chadmin "SELECT currentUser() FORMAT TSV" # must print prices_admin + +export CH_DOMAIN=ch.sorobanscan.rumblefish.dev +export MTLS_CERT_PATH=$HOME/prices-mtls/prices_writer.crt +export MTLS_KEY_PATH=$HOME/prices-mtls/prices_writer.key +export MTLS_CA_PATH=$HOME/prices-mtls/ca.crt + +# The sweep owns the current and the previous month, so stop two months back. +# CUTOFF is the first instant after END. It is fixed here, not re-read from +# now(), so ③ and ⑤ measure the same range even if the job crosses a month. +END=$(date -u -d "$(date -u +%Y-%m-15) -2 month" +%Y%m) +CUTOFF=$(date -u -d "$(date -u +%Y-%m-15) -1 month" +%Y-%m-01) +# From these months on, the 1d→1w (60 d) and 1d→1M (400 d) MVs re-roll 1w/1M +# from 1d every day. Once 1d is priced, those rows are priced without this step. +MV_1W_FROM=$(date -u -d '-60 day' +%Y%m); MV_1M_FROM=$(date -u -d '-400 day' +%Y%m) +echo "END=$END CUTOFF=$CUTOFF MV_1W_FROM=$MV_1W_FROM MV_1M_FROM=$MV_1M_FROM" + +# A 1w/1M row's reference can be the previous bucket, which a one-day ASOF +# window silently drops (coarse-repair --help, --pivot-window-s). +pivot_window() { case "$1" in price_ohlcv_1w) echo 604800 ;; price_ohlcv_1M) echo 2678400 ;; *) echo 86400 ;; esac; } +``` + +**①b [fishuser-hero, same `tmux`] Only if a month is recorded as +un-re-ingested: back up its coarse rows.** Its `1m` is gone, so these rows are +the only copy. This copies them into a backup table as `prices_admin` (the +phase-1 rollout's idiom, `0286-candle-definitions-rollout.md` §3). `chadmin` +fails on any ClickHouse error, so the loop stops at the first failed statement +instead of reporting a backup that does not exist: + +```bash +M=202607 # the un-re-ingested month, as YYYYMM +for T in 15m 1h 4h 1d 1w 1M; do + chadmin "CREATE TABLE IF NOT EXISTS prices.coarse_0286_bak_$T AS prices.price_ohlcv_$T" \ + && chadmin "INSERT INTO prices.coarse_0286_bak_$T SELECT * FROM prices.price_ohlcv_$T WHERE toYYYYMM(timestamp) = $M" \ + && chadmin "SELECT '$T', (SELECT count() FROM prices.price_ohlcv_$T WHERE toYYYYMM(timestamp) = $M) AS src, + (SELECT count() FROM prices.coarse_0286_bak_$T WHERE toYYYYMM(timestamp) = $M) AS bak FORMAT TSV" \ + || { echo "BACKUP FAILED at $T"; break; } +done +``` + +Each table must print `src` equal to `bak`, both above zero. Do not continue to +② on `BACKUP FAILED` or on a mismatch. A second run of the loop inserts the rows +again, so `bak` doubles. `TRUNCATE` the backup tables before you repeat it. +To restore the month from that copy, run `chadmin "ALTER TABLE prices.price_ohlcv_$T REPLACE PARTITION $M FROM prices.coarse_0286_bak_$T"`. + +`--start-month 201501` below is the lowest month the tool accepts. The tool +lists only months that hold enrichable zeros, so an empty month costs nothing. + +**② [fishuser-hero, same `tmux`] Dry run, all six tables. It writes nothing.** + +```bash +for TBL in price_ohlcv_15m price_ohlcv_1h price_ohlcv_4h price_ohlcv_1d price_ohlcv_1w price_ohlcv_1M; do + echo "===== $TBL =====" + ./target/release/coarse-repair \ + --transport hetzner --table "$TBL" \ + --start-month 201501 --end-month "$END" \ + --pivot-window-s "$(pivot_window "$TBL")" \ + --dry-run +done 2>&1 | tee /tmp/0286_coarse_dry.log +``` + +Expect every month from the table's first candle to `$END`, with `zeros_before` +close to the month's count of candles that have volume. **A table that reports +`0 month(s)` is a STOP**, not an all-clear. After a re-ingest there is nothing +already priced to explain it. Budget the run from the dry run's total: the rate +measured under task 0276 was ~7 k rows/s, and the repair runbook's Appendix C +explains why that is a floor. Time the first month and re-plan from that. + +**③ [fishuser-hero, same `tmux`] Baseline, as `prices_admin`.** Keep the +output. This function is also the after-check in ⑤. + +```bash +# Per coarse tier: every close_usd = 0 row OLDER than the coarse sweep's +# two-month lookback, split by why it is zero. Read-only. +coarse_zeros() { +for TBL in price_ohlcv_15m price_ohlcv_1h price_ohlcv_4h price_ohlcv_1d price_ohlcv_1w price_ohlcv_1M; do +chadmin " +WITH + (SELECT any(asset_id) FROM prices.assets FINAL + WHERE asset_code = 'XLM' AND issuer_address = '' AND contract_address = '') AS xlm, + (SELECT any(asset_id) FROM prices.assets FINAL + WHERE asset_code = 'USDC' AND contract_address = '' + AND issuer_address = 'GA5ZSEJYB37JRC5AVCIA5MOP4RHTM335X2KGX3IHOJAPP5RE34K4KZVN') AS usdc, + (SELECT any(asset_id) FROM prices.assets FINAL + WHERE asset_code = 'USDT' AND contract_address = '' + AND issuer_address = 'GCQTGZQQ5G4PTM2GL7CDIFKUBIPEC52BROAQIAPW53XBRJVN6ZJVTG6V') AS usdt, + (SELECT ifNull(minOrNull(timestamp), toDateTime('2106-01-01 00:00:00')) FROM prices.$TBL FINAL + WHERE asset_id = xlm AND quote_asset_id = usdc AND close > 0) AS xlm_ref_from, + (SELECT ifNull(minOrNull(timestamp), toDateTime('2106-01-01 00:00:00')) FROM prices.$TBL FINAL + WHERE asset_id = usdt AND quote_asset_id = usdc AND close > 0) AS usdt_ref_from +SELECT + '$TBL' AS tbl, + countIf(close_usd = 0) AS zero_usd, + countIf(close_usd = 0 AND volume_quote = 0) AS no_volume, + countIf(close_usd = 0 AND volume_quote > 0 AND close = 0) AS no_price, + countIf(close_usd = 0 AND volume_quote > 0 AND close > 0 + AND quote_asset_id NOT IN (xlm, usdc, usdt)) AS other_quote, + countIf(close_usd = 0 AND volume_quote > 0 AND close > 0 + AND ((quote_asset_id = xlm AND timestamp < xlm_ref_from) + OR (quote_asset_id = usdt AND timestamp < usdt_ref_from))) AS before_reference, + countIf(close_usd = 0 AND volume_quote > 0 AND close > 0 + AND (quote_asset_id = usdc + OR (quote_asset_id = xlm AND timestamp >= xlm_ref_from) + OR (quote_asset_id = usdt AND timestamp >= usdt_ref_from))) AS reachable_left +FROM prices.$TBL FINAL +WHERE timestamp < toDateTime('$CUTOFF 00:00:00') +FORMAT PrettyCompactNoEscapes" +done +} + +coarse_zeros | tee /tmp/0286_coarse_zeros_before.txt +``` + +The cutoff, `$CUTOFF` from ①, is the first instant the sweep covers on the +day you start. Everything below it belongs to this step. + +**Finish ② to ⑤ within the calendar month you ran ① in.** On the 1st, the +coarse sweep's window moves forward one month, and month `END + 1` is then +covered by neither the sweep nor this run. If the month turns before ⑤, re-run +the `END`/`CUTOFF`/`MV_*` lines of ①. Then run ④ once more on all six tables +with `--start-month "$END" --end-month "$END"`, the one new month, before ⑤. The +before/after comparison then covers that extra month too. Record that +alongside the numbers. + +**④ [fishuser-hero, same `tmux`] The real run, one table per command.** Read +each summary before you start the next table. + +```bash +TBL=price_ohlcv_15m # then price_ohlcv_1h, _4h, _1d, _1w, _1M +./target/release/coarse-repair \ + --transport hetzner --table "$TBL" \ + --start-month 201501 --end-month "$END" \ + --pivot-window-s "$(pivot_window "$TBL")" \ + --skip-snapshot 2>&1 | tee "/tmp/0286_coarse_${TBL}.log" +``` + +The repair runbook's "Log lines that look like failures but are not" applies +unchanged. In particular, every month before XLM/USDC's first candle +(2021-02) reports `enriched 0`, because nothing can price it. **Stop** on +either of these: + +- a month reporting exactly `200000` enriched, which means `one_shot` did not + take effect; +- **on the first run of a table only**, `enriched 0` on a month from 2021-03 + on that no MV has re-rolled from a priced `1d`. That covers every month for + `15m`/`1h`/`4h`/`1d`, months before `$MV_1W_FROM` for `1w` and months before + `$MV_1M_FROM` for `1M`. Every such month holds USDC-quoted candles, so a zero + there is a silent no-op. + +`enriched 0` is expected, not a stop, in two cases. First, `1w` months from +`$MV_1W_FROM` and `1M` months from `$MV_1M_FROM` on: the daily MV refresh has +already re-rolled them from the priced `1d`. The tool still lists them because +their unpriceable rows (`other_quote`, `before_reference`) match its candidate +predicate. Second, any month on a re-run of ④. Check both with ⑤'s +`reachable_left`, not with the summary line. + +**⑤ [fishuser-hero, same `tmux`] After-check, per tier.** Run +`coarse_zeros | tee /tmp/0286_coarse_zeros_after.txt`. Per table: + +| Column | Meaning | Required after the run | +| ------------------ | --------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `reachable_left` | Quoted in USDC, or in XLM/USDT from that leg's first USDC candle on. A tier can price these. | **~0.** Every remaining row must be explained with the drill-down below. | +| `before_reference` | Quoted in XLM/USDT before that leg's own USDC market exists in this table. Nothing can price them. | Unchanged from the baseline. This is the explained residue (the task-0208 boundary). | +| `other_quote` | No peg or pivot path. From 2025-09 the oracle tier prices some; the rest is the `no_reference` floor. | Equal to the baseline or lower. Never higher. | +| `no_price` | `close = 0`: a bucket with no price-forming fill. `0` is the correct USD value (ADR 0287, `rollups.sql`). | Unchanged. The repair selects such a row when its `volume_quote_usd = 0` (`CANDIDATE_PRED`) and may write that column, but `close_usd` stays `0`, so the count does not move. | +| `no_volume` | `volume_quote = 0`, which no tier prices. | Unchanged. | +| `zero_usd` | The sum of the five above. | Roughly the baseline minus the total the run's summary lines report as enriched. | + +If `reachable_left` is not ~0, list where the rows sit. Run this on +fishuser-hero, same shell, as `prices_admin`, and swap in the table: + +```bash +chadmin " +WITH + (SELECT any(asset_id) FROM prices.assets FINAL + WHERE asset_code = 'XLM' AND issuer_address = '' AND contract_address = '') AS xlm, + (SELECT any(asset_id) FROM prices.assets FINAL + WHERE asset_code = 'USDC' AND contract_address = '' + AND issuer_address = 'GA5ZSEJYB37JRC5AVCIA5MOP4RHTM335X2KGX3IHOJAPP5RE34K4KZVN') AS usdc, + (SELECT any(asset_id) FROM prices.assets FINAL + WHERE asset_code = 'USDT' AND contract_address = '' + AND issuer_address = 'GCQTGZQQ5G4PTM2GL7CDIFKUBIPEC52BROAQIAPW53XBRJVN6ZJVTG6V') AS usdt, + (SELECT ifNull(minOrNull(timestamp), toDateTime('2106-01-01 00:00:00')) FROM prices.price_ohlcv_1h FINAL + WHERE asset_id = xlm AND quote_asset_id = usdc AND close > 0) AS xlm_ref_from, + (SELECT ifNull(minOrNull(timestamp), toDateTime('2106-01-01 00:00:00')) FROM prices.price_ohlcv_1h FINAL + WHERE asset_id = usdt AND quote_asset_id = usdc AND close > 0) AS usdt_ref_from +SELECT toYYYYMM(timestamp) AS month, quote_asset_id, count() AS candles, + min(close) AS min_close, max(close) AS max_close +FROM prices.price_ohlcv_1h FINAL +WHERE timestamp < toDateTime('$CUTOFF 00:00:00') + AND close_usd = 0 AND volume_quote > 0 AND close > 0 + AND (quote_asset_id = usdc + OR (quote_asset_id = xlm AND timestamp >= xlm_ref_from) + OR (quote_asset_id = usdt AND timestamp >= usdt_ref_from)) +GROUP BY month, quote_asset_id ORDER BY candles DESC LIMIT 30 +FORMAT PrettyCompactNoEscapes" +``` + +Two shapes are an explained residue: + +- the leg's USDC market was silent for longer than `--pivot-window-s` before + the bucket (a thin month of 2021–2022, clustered by month); +- a `close` so small that `close × rate` truncates to 0 at 14 decimal places. + `max_close` shows this, as in the repair runbook's Appendix A triage. + +Anything else is a defect. This mode only ever selects rows still at zero, so +re-running ④ for that table is safe. It re-prices only what is still +missing. + +Record the before and after tables on task 0286. 7c's `post_run_0228_it` reads +the coarse tiers, so it can pass only after this step. + +**Rollback.** Nothing was discarded, so a rollback only ever rebuilds. What it +takes depends on the table: + +- **`15m`/`1h`/`4h`/`1d`, one month.** `DROP` that table's partition and re-run + §4g's pre-roll for the month, with only that table's statement from the + generated file. The re-roll reads the now-priced `1m`. +- **`1w`/`1M`.** There is no per-month undo. §7a is `TRUNCATE` plus a + full-range re-roll, so it rebuilds the **whole table** from `1d`. Only do it + when the whole table must go. Every un-re-ingested month's backup from ①b + has to be restored into it afterwards. +- **A month recorded as un-re-ingested (①b), any table.** Never `DROP` and + re-roll it, because its `1m` is gone and the re-roll would write nothing. + Restore it from the backup with the `REPLACE PARTITION` line in ①b. + **7c. The 0228 after-check**, on the now-repaired XLM/USDC pivot reference. Capture the `VERSION_BEFORE` values **before** the enrichment campaign starts: @@ -474,7 +820,9 @@ The phase-3 criteria of lore task 0286, in the order they can be checked: was 22 760 / 143 577 / 85 699 / 30 064 / 10 938 candles on 15m / 1h / 4h / 1d / 1w. After the re-ingest and pre-roll it is **zero**. 5. **The whole history re-enriched** — `close_usd > 0` wherever a reference - exists — and `post_run_0228_it` green. + exists, on `1m` (7b's drain) AND on every coarse tier (step 7b-2: + `reachable_left` ~0 on each of the six tables, every other column + explained) — and `post_run_0228_it` green. 6. **No USDT-quoted `1m` row carries the $1 peg** (step 7e): `peg_written = 0` and `pivot_written > 0` on `1m`, and on one coarse tier. This closes task 0212.