fix(rpc): prevent Studio database-pool starvation (v0.121 hotfix)

[MuncleUscles]

Problem and outcome

Hotfix for the v0.121 production line following Studio's September 14 database-health outage. Concurrent Explorer reads could occupy the shared application pool while a synchronous pool checkout on the API loop delayed response completion and session cleanup. The database itself remained available.

Keep database waits off the API loop, return Explorer connections before responses leave their worker, and bound repeated Explorer reads. This is based directly on released v0.121.23 (7d66e42b5dddddbecd2067df545107248d7c7c8c), excluding Studio Next and sibling PRs.

Implementation and validation

• Dispatch synchronous RPC handlers through the worker pool, preserving async handlers and awaitable results.
• Offload contract snapshot checkouts in schema, gen/sim call, and eth_call paths, plus the ConsensusData interceptor.
• Preserve websocket notifications from worker threads by publishing on the application's owning loop.
• Explorer query workers own and close read sessions before returning. Limit active queries to two per process.
• Cache counts for five seconds with one refresh; concurrent refresh callers may use previous counts. Overload returns HTTP 503 with Retry-After: 1 before opening another session.
• No migration, dependency pin, GenVM version, connection-pool setting, or infrastructure change.

Validation:

• Backend unit suite: 859 passed, including 32 new regressions for dispatch, responsiveness, context propagation, notifications, session cleanup, cache/admission behavior, and all Explorer routes.
• Frontend type check and unit suite: 196 passed.
• Repository Python/general pre-commit checks passed. Frontend ESLint passed with 290 warnings in unchanged files; Prettier check passed. Frontend hooks were run once as read-only equivalents to avoid duplicate rewriting runs.
• The first CI run exposed an optional Starlette TestClient dependency (httpx2). Explorer tests now exercise the in-memory ASGI app directly with no new dependency. The full 859-test suite also passes against CI FastAPI 0.135.1, Starlette 1.6.0, AnyIO 4.15.1, and typing_extensions 4.16.0.
• Initial sandbox run could not create two local Unix sockets; the complete host-level run passed.
• Local PostgreSQL integration setup could not start because the Docker Hub image fetch timed out. Real SQLAlchemy session return/rollback behavior is covered with SQLite; full database-backed stable-track E2E is required before landing.

Risks and rollback

Sync handlers now execute in worker threads; regression tests preserve dependency commit/rollback, client context, JSON-RPC results/errors, and websocket delivery. Explorer counts can lag five seconds, or briefly return the previous count during a refresh. Busy Explorer reads receive a retryable 503.

Rollback is the previous v0.121.23 image set; no schema rollback is necessary. Production recovery must still be verified after deployment; the hotfix does not resolve unrelated stuck consensus transactions.

Delivery context

Single-PR landing cut; no cross-repository dependencies or handoffs. Freeze head e8208389451cfe9ba392b4726bb0b56d928f24a7 and base 7d66e42b5dddddbecd2067df545107248d7c7c8c. Request full v0.5 all E2E once for this exact snapshot. User authorized E2E, admin merge after green, and release. Planned next patch is v0.121.24; release deployment scope is being confirmed because the standard lane also promotes Rally and Collective Memory.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: e9ceb583-1cdf-488f-9192-68aeb8d5102d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[MuncleUscles]

Landing checkpoint: freeze the single-PR v0.121 hotfix cut at #1766.

• Included PRs: fix(rpc): prevent Studio database-pool starvation (v0.121 hotfix) #1766 only; no lower layers, sibling composition, cross-repository Depends-On, or required handoffs.
• Head: 8bb5376e2d2b202bb2bbff54aeea34612af7bc9d
• Base v0.121: 7d66e42b5dddddbecd2067df545107248d7c7c8c
• Synthetic merge: fa2071acc85c0cf756e6ac4a2650d46b8abb809e
• Source tree: 1fefb88c08fd5dd3df3b65272ecd5b0e736d281c, verified identical to the locally reviewed/tested tree. Publication used the connected GitHub app because host Git transports were unavailable.
• Local validation: 859 backend + 196 frontend tests passed, type check passed, Python/general pre-commit and frontend formatting/lint passed (existing frontend warnings only). CI includes the database integration suite unavailable locally due a Docker Hub timeout.
• Full E2E selection: stable v0.5, default profile, all scope, all configured stacks; no force/cache/cloud/capacity overrides. No prior run exists for this snapshot.

The user authorized admin merge only after green and release afterward. Heads and base remain frozen while E2E runs. Standard release promotion scope still requires the user's pending confirmation.

[MuncleUscles]

/run-e2e v0.5 all

[ci-core-e2e-runner]

E2E status was updated. Follow the current E2E and merge checks on this PR. Detailed diagnostics are available internally.

[MuncleUscles]

CI diagnosis: the PostgreSQL integration job passed, but backend unit collection failed because Starlette 1.6.0's optional TestClient now requires httpx2, which this repository does not install. The new Explorer test file introduced that optional-client import. The fix is test-only: invoke the in-memory ASGI app directly, with no new dependency and no production-code change. Validating against CI's FastAPI/Starlette/AnyIO versions before publishing the revision. The original E2E snapshot will be superseded by one explicitly selected run after the revised head is frozen; it will not be used to merge a different head.

[MuncleUscles]

Revised landing checkpoint: #1766 only, with the test-client packaging correction. No production behavior or dependency changes since the initial cut.

• Head: e8208389451cfe9ba392b4726bb0b56d928f24a7
• Base v0.121: 7d66e42b5dddddbecd2067df545107248d7c7c8c
• Synthetic merge: 651952439541f70a05c98e9d3878bffff09c2249
• Source tree: 98f2ad2cb7d8d744a3715150bbdd1104711002b2, verified identical to the local reviewed tree.
• Full backend suite: 859 passed against both the original local environment and CI's web-framework versions. Existing frontend validation remains 196 passed plus type/lint/format checks. PostgreSQL integration passed on the preceding runtime-identical cut.
• No cross-repository dependencies or handoffs; no sibling composition.

The previous run 34873235915 tested the superseded head. A new full v0.5/all-scope/all-stack request now produces valid evidence for the corrected exact head and intentionally supersedes the old run. No force/cache/cloud/capacity override is needed. Do not merge until current CI and exact full E2E are green.

[MuncleUscles]

/run-e2e v0.5 all

[ci-core-e2e-runner]

E2E status was updated. Follow the current E2E and merge checks on this PR. Detailed diagnostics are available internally.

[sonarqubecloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
82.6% Coverage on New Code
1.1% Duplication on New Code

See analysis details on SonarQube Cloud

[MuncleUscles]

Hotfix landing verified

• Single-PR landing cut: fix(rpc): prevent Studio database-pool starvation (v0.121 hotfix) #1766 into v0.121; no cross-repository dependencies or outstanding handoffs.
• Full E2E: run 34874232719, default profile / v0.5 / all scope / all stacks; 86/86 scenarios passed, 79/79 expected feature entries covered, no skips or missing coverage.
• Tested PR head: e8208389451cfe9ba392b4726bb0b56d928f24a7; unchanged base: 7d66e42b5dddddbecd2067df545107248d7c7c8c; App candidate: becede63759775c45f08d8298f1e2ca754ded1f8.
• Local validation: 859 backend tests and 196 frontend tests; formatting/type checks and current required CI passed. DB integration passed on the runtime-identical initial hotfix commit; the follow-up changed tests only.
• Authorized squash merge completed: 2004fc4315f0d2a0266f2da730992ce1e33ecfcc. Verified v0.121 points to this commit and its tree 98f2ad2cb7d8d744a3715150bbdd1104711002b2 exactly matches the tested candidate.

Release remains pending deployment-scope confirmation. Planned patch is v0.121.24; no release tag or release workflow was created/dispatched. The normal release lane also promotes to Rally and Collective Memory, which is broader than the Studio Prod incident. Waiting for the owner to choose that standard lane or Studio Prod only. Studio Next #1762 is not part of this hotfix.

[MuncleUscles]

Standard release started

The owner confirmed the standard deployment lane, including Rally and Collective Memory, on 2026-09-14. The deployment-scope hold is resolved.

Started the existing manual-docker-release.yml workflow once for branch v0.121, version v0.121.24: release run 34880110370.

Verified the new annotated tag peels to merged hotfix commit 2004fc4315f0d2a0266f2da730992ce1e33ecfcc, whose tree matches the full green E2E snapshot. Tag validation passed; all five component images are now building for both architectures. Standard staging gates and production promotion remain intact. Studio Dev is preview-pinned and Studio Next is excluded.

Automatic follow-up is active for build, staging, production promotion and live health/image verification. Deployment is not yet complete.

[MuncleUscles]

Release handoff — published and promoted; verification blocked

v0.121.24 was created at merged commit 2004fc4315f0d2a0266f2da730992ce1e33ecfcc (tested tree 98f2ad2cb7d8d744a3715150bbdd1104711002b2). Build/tag run 34880110370 passed. Full pre-merge E2E run 34874232719 passed 86/86 scenarios, with no missing coverage; 859 backend and 196 frontend tests plus required CI passed. Single-PR hotfix; no cross-repository dependencies/handoffs in the tested cut.

The owner authorized the standard deployment scope. Staging GitOps PR #722 merged as de5351173661f6fba615b4be5adc517d079c568b; its sync/health gate passed. Production GitOps PR #723 merged as 5319f906399cd169dacec0a0d543fa2215b5bcc9, changing only the five released images in rally-studio-stg, studio-prd, rally-studio-prd and collective-memory-studio-prd. All promoted digests matched the build outputs. Studio Dev/Next was excluded.

Post-deploy verification is not green: run 34882106929, attempt 2.

• Collective Memory synced successfully to 5319f906399c and its RPC rollout/frontend/explorer were Healthy; migration succeeded. The app gate timed out because ScaledObject/studio-consensus-worker reports Triggers defined in ScaledObject are not working correctly; the worker deployment had 7/8 updated replicas available.
• Rally's first verifier could not establish its VPN, before deployment checks. A single isolated retry connected successfully. It then timed out on the same degraded worker ScaledObject. Rally is Synced to 5319f906399c; its other health-reported resources were Healthy.
• Both apps' aggregate Degraded timestamps are September 9, before this hotfix. That establishes pre-existing degraded application state, but does not by itself prove the exact trigger failure's root cause.
• Public /health checks on Studio, Rally and Collective Memory were healthy again around 20:09 UTC, including database/GenVM/consensus. This is not a substitute for a green rollout gate.
• Studio Prod's exact live image/pod-readiness verification remains unavailable: local PRD-DevExp AWS SSO is expired and refresh fails; Argo's browser access requires login. A sign-in refresh was requested from the owner.

No gates were bypassed. No infrastructure fixes, pool changes, restarts, rollback, cleanup or transaction repairs were performed. Automatic follow-up is being paused pending refreshed AWS access and investigation of the autoscaler health failure; do not rebuild/redeploy or blindly rerun these unchanged failed health gates.

[MuncleUscles]

AWS follow-up: the v0.121.24 hotfix is now actually rolled out on Studio Prod. Its current migration succeeded at 20:28:54 UTC after guarded removal of the stale finalizer from an already-completed/deleting August 25 Job. Studio Rollout is Healthy with 4/4 updated RPC replicas, workers are Ready, and Studio's full live verification job 104143613078 passed in run 34894088124.

Shared release blocker diagnosed and repaired: production Prometheus had exceeded its 1 GiB cap and was OOMKilled over 1,300 times, leaving all three KEDA triggers unable to connect. argocd-aws-apps#61 is merged (b9624a6) after green CI; replacement is Ready with zero restarts, and all three ScaledObjects are Ready=True. Live demand metrics return Studio=5, Rally=0, CM=0.

Release verification coverage is repaired by devexp-argocd-apps#724 (5259d2d), with green full Linux release tests. The remaining verifier failure is evidence retention: Rally/CM were Synced+Healthy with all four runtime images verified, but their successful migration Jobs expired after the one-hour TTL before the delayed check could inspect the fifth image. Follow-up #725 retains the latest named migration Job until the next sync replaces it; CI and all three post-merge live gates remain required. No new app release/tag or pool resizing; no health-gate bypass. Final completion is not yet claimed.