feat(scenarios): cdk-erigon FEP to PP rollback scenario

[jhkimqd]

Why

Silicon (mainnet rollup ID 10, chain 2355, cdk-erigon v2.61.20 fork 12 validium, cdk-node 0.5.4-rc1) can no longer verify batches: agglayer 0.6.0 disabled interop_sendTx (-10009), the only settlement path of the legacy cdk-node aggregator. On 2026-09-17 it had lastBatchSequenced 76058 vs lastVerifiedBatch 75940, and the cdk-erigon FEP to PP runbook cannot start because AgglayerManager.initMigration reverts with AllSequencedMustBeVerified() while the counters differ. Agglayer 0.5.x is sunset, so the gap cannot be closed by verifying.

What

scenarios/cdk-erigon-fep-to-pp-rollback/ reproduces that situation on kurtosis-cdk with Silicon's versions and validates the way out end to end:

1. deposit L1 to L2, withdraw L2 to L1, legacy aggregator verifies through agglayer 0.6.0-rc.5, claim on L1
2. Polygon upgrades agglayer to 0.6.0: cdk-node logs -10009, sequencing continues, a withdrawal lands in a sequenced-but-unverified batch
3. initMigration reverts with AllSequencedMustBeVerified()
4. operator stops sequencing and calls rollbackBatches(rollup, lastVerifiedBatch): counters equal, both RollbackBatches events, L2 block hashes unchanged, erigon's zkevm_virtualBatchNumber follows on both nodes
5. cdk-erigon to v2.61.24 with the runbook's PP config (sequencer re-added with its persistent datadir), legacy components stopped
6. aggkit 0.8.1 aggsender in dry run, initMigration to AggchainECDSAMultisig, bootstrap certificate up to the last verified block, normal certificates, the in-window withdrawal claimed on L1

Around every one of those steps both cdk-erigon nodes are probed (erigon_health: block number, batch/virtual/verified counters, fork id, reference block hash, zkevm_isBlockVirtualized, zkevm_getBatchByNumber, eth_getLogs, debug_traceTransaction, eth_call) and made to do real work: transactions submitted through the RPC node and the sequencer, L1 to L2 deposits claimed on L2 after the rollback and after the migration.

RUNBOOK.md is the operator runbook for Silicon. Every observation block is rendered from the committed evidence (scripts/render-runbook-observations.py), which is the unmodified CI artifact of run 35286572861: command outputs, state snapshots, service logs, health probes and Playwright screenshots of the Kurtosis Enclave Manager, with key material redacted while recording.

Findings worth reading even if you skip the rest

• What rollbackBatches does to cdk-erigon (runbook section 3b, observed on both nodes at nine stages): once the RollbackBatches block is final, the L1Syncer stage deletes the local L1 sequence records, zkevm_virtualBatchNumber drops to the target, zkevm_isBlockVirtualized flips to false for the rolled-back range and zkevm_getBatchByNumber loses its sendSequencesTxHash. Block hashes, local batch numbers, eth_getLogs, debug_traceTransaction, transaction submission through either node, datastream sync and L1 to L2 deposits claimed on L2 are unaffected, before and after the migration. There is no L2 unwind. A read-only mainnet simulation from Silicon's admin succeeds today.
• initMigration on mainnet is callable by the Agglayer admin, a 5-of-9 Gnosis Safe holding UPDATE_ROLLUP_ROLE without timelock; the timelock path would add 3 days. The runbook makes the actor and the signing lead time explicit.
• aggkit 0.8.1 defaults to Mode = "Auto", which reads the consensus type from the rollup contract and exits on a legacy PolygonValidiumEtrog. Starting aggkit in sync-only mode before the migration needs Mode = "PessimisticProof".
• When the target rollup type was added on L1 before the cdk-erigon upgrade (Silicon: mainnet type 14), the v2.61.24 sequencer logs received UpdateRollupTopic for unknown rollup type once, keeps producing and keeps fork 12; it must not be restarted before its next clean L1 Sequencer sync finished. RPC nodes do not run that stage and are unaffected.
• kurtosis service update recreates containers without their persistent directories and dropped a mount; the scenario re-adds the sequencer and aggkit with Starlark that declares the same persistent_key.

Validation

• The author's Apple Silicon host cannot run the fork 12 zkevm-prover (amd64/AVX2: SIGILL under Rosetta; under QEMU the x86 agglayer binary exceeds kurtosis-cdk's 180 s agglayer vkey task), so validation ran on ubuntu-latest through the new workflow, about 25 minutes per run.
• Seven full runs passed with identical invariants: 35201843011, 35204567044, 35207232742, 35209534700, 35211930721, 35214396395 and, with the cdk-erigon health probes, 35286572861 (committed evidence). An eighth run on the final commit also passed: 35289107019.
• An adversarial validator reviewed the scenario, runbook and evidence in two passes (contract source, cdk-erigon and aggkit source, mainnet reads, artifact comparison) and rated it ship-ready after the fixes now included.

🤖 Generated with Claude Code