Skip to content

feat(scripts): add resumable bounded log scanner - #365

Draft
yzwooyi wants to merge 1 commit into
circlefin:mainfrom
yzwooyi:codex/resumable-log-scanner
Draft

feat(scripts): add resumable bounded log scanner#365
yzwooyi wants to merge 1 commit into
circlefin:mainfrom
yzwooyi:codex/resumable-log-scanner

Conversation

@yzwooyi

@yzwooyi yzwooyi commented Sep 6, 2026

Copy link
Copy Markdown

Summary

Issue #83 needs a client-side workaround for wide eth_getLogs requests. This adds a Python standard-library reference scanner that stores raw logs and its cursor in SQLite, commits each successful prefix atomically, and resumes safely after interruption.

  • Freezes latest to a numeric end block and verifies chain ID and end-block hash on resume.
  • Splits range-limited or timed-out requests adaptively without skipping blocks; bounded retries cover transient transport, HTTP 429/5xx, and Arc's concurrency limit.
  • Validates the complete response batch, deduplicates by (blockHash, transactionHash, logIndex), and rolls back rows and cursor together on write failure.
  • Documents the client-side workflow and adds dedicated Python CI.

This addresses #83 as a client workaround and does not change the RPC server. Related: #79 contains a simple pagination snippet; this adds an executable crash-resumable tool.

Validation

  • python3 -m unittest discover -s scripts -p test_scan_logs.py -v — 25 tests passed under Python 3.12.14, including localhost HTTP transport fixtures.
  • python3 -m py_compile scripts/scan_logs.py scripts/test_scan_logs.py — passed.
  • git diff --check — passed.
  • Live comparison: blocks 60786744..60786844 in 17-block chunks returned 1,836 raw logs exactly matching the direct RPC fixture (0 missing, 0 unexpected).
  • A live from-genesis request returned provider error 4444 (pruned history unavailable), which is correctly treated as fatal rather than a range timeout.

No Rust or protocol changes are included. Full Rust and contract builds were not run because this is a client-script change; dedicated Python CI remains pending.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant