Skip to content

fix(deploy): Run the service on a dedicated-core VM - #74

Merged
dcramer merged 1 commit into
mainfrom
fix/dedicated-core-vm
Nov 22, 2026
Merged

dcramer merged 1 commit into
mainfrom
fix/dedicated-core-vm

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 22, 2026

Copy link
Copy Markdown
Contributor

Change the default machine_type from e2-small to e2-standard-2.

e2-small is a shared-core type. It sustains 0.5 vCPU and can go above that only while it has burst credit. The proxy uses CPU for each request (TLS, JSON key, redaction), so its throughput stops at the shared-core limit.

On 2026-11-21, from about 22:45 UTC, about 35 getsentry/junior eval runs used Roach in each 15-minute window. Before that, it was 5 to 15. The sentry/roach spans show the service, not the upstreams:

span 22:00–22:45 UTC p50 23:00–00:15 UTC p50
roach.request, replayed 38ms 495ms
roach.request, passthrough 80ms 478ms
roach.store.read 25ms 490ms

The total request rate stayed at about 14k per 15 minutes while latency went up about 10 times: the service was saturated. Replays (GCS) and passthrough calls (api.vercel.com) slowed at the same time. roach.key is synchronous CPU work with no I/O, and its p95 went from 17ms to 200ms. That points to CPU starvation on the VM, not to GCS or an upstream. The getsentry/junior integration evals went from about 4 minutes of test time per shard to about 26 minutes and hit the 30-minute job limit. Their Roach hit rate did not change (532 of 535 model requests replayed).

e2-standard-2 has 2 dedicated vCPUs, so the single Node process gets a full core and GC gets the other one. It uses the same boot disk type. One Node process is still limited to one core for JavaScript. If a load spike saturates it again, the next step is to run more than one process.

To deploy, run terraform apply -var-file=roach.tfvars in deploy/gcp/. The Deploy workflow does not apply Terraform. allow_stopping_for_update is set, so the apply stops the VM for about a minute to resize it, and eval runs in progress at that time fail.

via @dcramer.

--

View Session [Sentry]

The e2-small VM sustains only 0.5 vCPU. On 2026-11-21, when about 35
getsentry/junior eval runs used Roach at the same time, request p50 went
from about 70ms to about 500ms, and integration evals timed out.
@dcramer

dcramer commented Nov 22, 2026

Copy link
Copy Markdown
Member

What's this change from a cost profile

@sentry-junior

sentry-junior Bot commented Nov 22, 2026 •

Copy link
Copy Markdown
Contributor Author

The VM cost goes up about 4x, from about $12 to about $49 a month. These are on-demand list prices for us-east1 at 730 hours a month:

type vCPU memory $/hour $/month
e2-small (now) 0.5 sustained, shared core 2 GB 0.0168 ~12.23
e2-standard-2 2 dedicated 8 GB 0.067 ~48.92

So the change adds about $37 a month. E2 types get no sustained-use discount. A 1-year commitment would bring e2-standard-2 down to about $31 a month, and a 3-year one to about $22. The boot disk, load balancer, bucket, and egress don't change.

There's a cheaper option, e2-medium, at about $24 a month. It is still shared-core: it sustains 1 vCPU and bursts above that. That would roughly double what Roach can handle under steady load, but it can still run out of burst credit the same way. I'd keep e2-standard-2.

Prices are from https://gcloud-compute.com/e2-standard-2.html and https://gcloud-compute.com/e2-small.html, which mirror the GCP price list.

@dcramer
dcramer merged commit c1dcf32 into main Nov 22, 2026
17 checks passed
@dcramer
dcramer deleted the fix/dedicated-core-vm branch November 22, 2026 00:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant