Web UI pages (policies, integrations) take several seconds to display

[amondnet]

Summary

The web UI in apps/host-cloudflare (the executor-cloudflare Worker) takes several seconds to display data on pages like /<org>/policies and /<org>/integrations/<slug>. Investigation with Workers Observability + a real browser waterfall shows the slowness is not the JS bundle or render, but a combination of (1) a large fan-out of per-request API calls that each rebuild a scoped executor, (2) an unnecessarily heavy /api/tools path, and (3) geographic distance (KR users served from a US colo).

Evidence (measured)

Browser waterfall, warm load of the integrations page:

• Document + JS bundle is fast: domContentLoaded/load ≈ 388ms, chunks ~241ms (cached). Not the bottleneck.
• After mount the page fires ~8 parallel API calls (account/me, integrations, tools, policies, connections ×3, openapi/.../config).
• Per-call TTFB (warm): 433–580ms for most, /api/tools = 1168ms (the long pole). Data-complete ≈ 1.77s warm, worse cold (server-side 0.5–1.7s per call).

Geography (Workers Observability, last ~3.5h, 182 /api/* requests):

• cf.country = KR, cf.colo = LAX for 100% of requests — Korean users are served from a US-West colo.
• This matches the gap between server-side span (~145ms for /api/policies) and browser TTFB (~433ms): ~290ms is the KR↔LAX round trip.

CPU vs wall: median CPU ~40–60ms while wall time is seconds → the worker is waiting (I/O / network / round trips), not computing.

Root Causes (with code references)

1. /api/tools long pole. toolsList runs syncStaleConnectionTools on every list call, which issues 2 unconditional D1 queries even when nothing changed (packages/core/sdk/src/executor.ts:2475 call, :2438 definition). It also calls listOperations, which reads the entire plugin_storage operation set and filters in JS (packages/plugins/openapi/src/sdk/store.ts:150-156). D1 caps bound params at 100, so wide reads batch (apps/host-cloudflare/src/db/d1.ts).
2. Page fan-out rebuilds a scoped executor per call. Each of the ~8 calls independently builds a scoped executor and re-reads the catalog/policies (packages/core/api/src/server/scoped-executor.ts:253). The D1 handle itself is shared and built once per isolate (apps/host-cloudflare/src/app.ts), and auth (Cloudflare Access JWT) is cheap and JWKS-cached (apps/host-cloudflare/src/auth/cloudflare-access.ts:73) — so the redundancy is the executor build + repeated D1 reads, multiplied by 8.
3. Geographic placement. No placement config; KR users are served from LAX (~290ms RTT). Paid ~once per page because calls are parallel, but still adds to every page.

Proposed Fixes (impact order)

• Fix A — cut /api/tools server time (long pole). Skip syncStaleConnectionTools on the read path when no integration has been revised (cache the check, or move sync to write-time / a background task). Make listOperations filter by integration at the DB layer instead of scanning all rows.
• Fix B — page bootstrap BFF endpoint. Add a single aggregated endpoint that authenticates once, builds one scoped executor, and returns account + integrations + tools + policies + connections in one response; switch the affected pages to a single call. Removes ~7 redundant executor builds per page load. Expected warm: ~1.77s → ~0.6–0.9s.
• Fix C — short-TTL cache of scoped executor / catalog reads per (org, subject) to amortize fan-out and repeated navigation (scoped-executor.ts:253).
• Fix D — geography (separate track). Investigate why KR users hit LAX (likely *.workers.dev subdomain routing; test a zone custom domain). If the worker is moved closer to users, D1 must also get a nearby read replica (D1 read replication / Sessions API), otherwise multi-query endpoints get slower.
• Fix E — (separate) MCP session DO 30s keepalive alarm runs every 30s at ~0 CPU / 30s wall, keeping the DO from hibernating (apps/host-cloudflare/src/mcp/session-durable-object.ts). Unrelated to page latency; review for cost/hibernation.

Notes

• Measurements taken 2026-06-25 via Workers Observability and a browser waterfall.
• Recommended order: A, then B, then C; D and E are separate tracks.

---

• To see the specific tasks where the Asana app for GitHub is being used, see below:
  • https://app.asana.com/0/0/1216031173051682