Hardcoded Semaphore(8) in CommittingState serializes large validator committees #1721

Problem

CommittingState.handle caps concurrent validator executions with a hardcoded asyncio.Semaphore(8) (backend/consensus/base.py:1925, introduced in #1272).

For large committees this dominates transaction latency: a 47-validator committee runs as ~6 sequential rounds of 8 (sliding window, but with similar per-validator LLM latencies it behaves like waves). This is the likely cause of the observed "47v is ~10× slower" behavior, and becomes increasingly relevant as appeals push committees to 23+.

Why the cap exists (and why 8 is arbitrary)

Bounding GenVM concurrency is legitimate:

• each validator execution spawns a GenVM subprocess → unbounded concurrency risks OOM
• uvloop fd-registry collisions from concurrent GenVM socket operations (hit on the RPC path, fixed in fix: bound rpc genvm calls and health scans #1648)
• DB connection-pool exhaustion

But the value 8 is not derived from anything — not container memory limits, host RAM, or committee size. It's a static default from #1272 with no recorded rationale. Meanwhile the RPC path's equivalent bound is already configurable via GENVM_MAX_CONCURRENT (backend/protocol_rpc/endpoints.py:71, default 8).

Proposal

1. Make the consensus-path bound configurable — either reuse GENVM_MAX_CONCURRENT or add a dedicated env var (e.g. CONSENSUS_VALIDATOR_CONCURRENCY), defaulting to the current 8 for backward compatibility.
2. Deployments with adequate memory can then raise it to match committee sizes (23–47+) instead of paying a ~6× serialization penalty.

References

• backend/consensus/base.py:1925 — hardcoded asyncio.Semaphore(8)
• backend/protocol_rpc/endpoints.py:67-73 — configurable RPC-path equivalent + rationale comment
• fix: validation function concurrency #1272 — introduced the semaphore (no rationale for the value)
• fix: bound rpc genvm calls and health scans #1648 — documented fd/DB-pool failure modes of unbounded GenVM concurrency