feat: rt-sim: deterministic simulated AsyncRuntime (draft, #206)

[pchrysostomou]

Draft for #206: a standalone rt-sim runtime, and t24_append_membership running on it by seed with a repeatable full OpenRaft log.

Seed demo: #2059 replayed

With #2059's wait reverted locally (git show 4eb14581 | git apply -R, not part of this PR) and OPENRAFT_NETWORK_SEND_DELAY=30, 24 of 300 seeds fail follower_answers_forward_to_leader the way CI did:

t24_append_membership.rs:241
  left: Some(6)
 right: Some(5)

Seed 9 fails in 5 of 5 fresh processes with a byte-identical 3,467-line DEBUG log. With the wait restored, all 24 seeds pass. Seed 9's passing log matches the failing one for the first 3,431 lines. At virtual 0.143 s new_cluster() returns while follower 1 still reports last_log:5: the failing run asserts there, and the fixed run waits for last_log:6 at 0.147 s.

cd tests-sim
OPENRAFT_RT_SIM_SEED=9 OPENRAFT_NETWORK_SEND_DELAY=30 OPENRAFT_SIM_LOG=/tmp/t24.log \
  cargo test --test membership -- --exact run1::follower_answers_forward_to_leader

CI runs this twice in a new tests-sim job and diffs the two logs.

select!: with and without the reseed

tests-sim patches in your futures-util fork (same tag as tests-turmoil), and rt-sim calls futures_util::reseed(seed) at the start of every block_on. It includes t24 three times, as run1..run3, so one process runs the scenario back to back. Seed 9, 30 ms delay:

• reseed on: the three runs write identical 3,558-line logs, sequentially and on 3 parallel threads, and match run1 alone in a fresh process.

• reseed off (--no-default-features): three different logs. They diverge at line 1,396, in the four-branch select! in replication/stream_state.rs:154. rt-sim's scheduling trace stays identical across all three: the shuffle changes which branch body runs, not which task is polled.

  OPENRAFT_RT_SIM_SEED=9 OPENRAFT_NETWORK_SEND_DELAY=30 OPENRAFT_SIM_LOG=/tmp/runs.log \
    cargo test --test membership -- --test-threads=1 follower_answers_forward_to_leader
  

The runs share one log file; the membership::runN span target tells them apart.

Changes

• rt-sim/, excluded, publish = false:
  • one thread, FIFO run queue; once it is empty the clock jumps to the earliest timer, and equal deadlines fire in registration order; panics on deadlock;
  • thread_rng() seeded from OPENRAFT_RT_SIM_SEED;
  • mpsc, oneshot and mutex over tokio::sync; its own FIFO watch, since tokio::sync::watch picks a random waker slot when tokio's rt feature is on;
  • opt-in sim-log feature: with OPENRAFT_SIM_LOG, block_on routes its thread's tracing events to a file, stamped with virtual time, without thread and span ids, with DisplayInstant's wall-clock times masked;
  • Suite::test_all passes, in a new rt-sim CI job.
• openraft-memstore: an rt-sim feature with a second, cfg-gated declare_raft_types! that sets AsyncRuntime = SimRuntime.
• fixtures: one line. The send delay comes from thread_rng() instead of rand::random(); tokio behaviour is unchanged.
• tests-sim/, excluded: the futures [patch], and t24 included by path. It is the only way to run a test on rt-sim; tests itself has no new feature or dependency.

Easy to cut

• sim-log. Without it seeds still reproduce pass/fail and the scheduling trace, but the select! drift above only shows in this log.
• The three-way include in tests-sim, which exists only for the in-process check.

Limits

• Only t24 is wired in. A test that declares its own type config (t23_custom_payload) or names TokioInstant (t61, t14, t15) can't switch runtimes as is.
• DEBUG logging is formatted on one thread, so CPU-bound tests can run slower than on tokio. With RUST_LOG=off rt-sim is faster.
• cargo package -p openraft-memstore fails as is: its optional dependency on rt-sim is a path without a version.
• Not yet: single-threaded; seeds other than from env vars; masking beyond DisplayInstant's two formats.

make verify passes, apart from two things this machine lacks: cargo-expand for the test_expand and test_since macrotest targets, and Python 3.10 for check-doc-links.py (its 129 links resolve under 3.9 with postponed annotations).

One question for you. Cargo won't package memstore with an unversioned optional dependency, so keeping the rt-sim feature on openraft-memstore means publishing openraft-rt-sim alongside it, the way rt-tokio is. The alternative is to leave rt-sim unpublished and move the runtime switch out of memstore. Which do you prefer?

Checklist

• Updated guide with pertinent info (may not always apply).
• Squash down commits to one or two logical commits which clearly describe the work you've done.
• Unittest is a friend:)

---

This change is 

[drmingdrmer]

This runtime alone cannot make OpenRaft fully deterministic. OpenRaft uses futures_util::select!, which uses its own RNG to choose a branch when multiple futures are ready.

I’ll keep this PR open for now and explore making branch selection part of the async runtime abstraction.