[rdimitrov]

Cyclic listing bursts queue application work when app CPU competes with other node workloads. This change releases listing rows before typed decoding, prepares versioned listing JSON during writes, and adds checkout timing, opt-in loopback profiling and configurable per-replica DB pool limits. Defaults remain 60 maximum / 10 minimum connections. Derived JSON preserves fresh metadata, invalidates on old-writer updates and falls back for unknown codecs; existing rows have an explicit bounded backfill tool.

Draft: neither latest staging layout passed the short offered-load gate. Production is unchanged. The experimental scaling configuration is not approved for promotion.

Generator calibration and staging results

The benchmark now separates scheduler/worker delay, connection acquisition, TLS, first-byte and body-transfer timing by request kind and bounded cache outcome. An HTTPS loopback fixture at 4,500 reads/s reproduced 6.40% dropped work on the four-vCPU client/fixture host. A temporary sixteen-vCPU host completed all 405,000 calibration requests without drops or failures. This resolves a local generator limitation; it does not independently certify external network capacity or attribute all older losses to the generator.

Both subsequent staging tests used the same larger generator, prepared-JSON image, four app replicas at 400m CPU request each, six ingress replicas, four nodes and three-CPU DB quota. Pools remained 30/5 per app, totaling 120/20. Each burst scheduled 810,000 reads over three minutes at 4,500/s with varied keys, updated listings and searches; neither short test published data.

Measurement	Shared nodes	DB isolated on one of the same four nodes
Reads dropped before HTTP	1.067%	2.892%
Attempt failures	9 timeouts	0
Page p95, scheduled to completion	328 ms	414 ms
Search p95	3,369 ms	1,635 ms
Updated-listing p95	3,935 ms	1,291 ms
Largest sampled pool queue on one app	758	460
Highest measured app CPU-stalled fraction	39.1%	30.7%
Dominant DB statement mean	1.72 ms	0.63 ms

The worst shared-layout app was on a node without PostgreSQL. DB isolation improved measured SQL time but app CPU stalls and queues persisted. Isolation also concentrates apps and ingress on three nodes, so this measures the complete placement change, not an isolated DB effect. Each layout was tested once. Different placements also limit comparisons with older generator runs.

Both two-minute recoveries completed 12,000 reads without drops or failures and ended with zero sampled pool queues. DB throttling did not increase within either captured run; neither app nor DB restarted. The failed short gates mean the conditional ten-minute burst with publishing and thirty-minute recovery was not run. No app HPA was implemented. These results do not justify increasing the DB pool or treating a dedicated DB node as a proven fix.

The next resource design needs to address app CPU allocation and placement alongside busy ingress workloads, including scheduling feasibility and the desired idle footprint. The specific production configuration is not yet validated.

Validation and restoration

• Race tests and lint passed; CI at driver source 08077e4 and CodeQL passed. Timing tests cover delayed headers/body, connection reuse, missing events, bounded labels and calibration restrictions. Existing tests cover API/OpenAPI equivalence, prepared-data freshness/invalidation, codec fallback, backfill concurrency and pool limits.
• Original staging app image/resources, two replicas, app/ingress placement and rollout strategies, ingress HPA specification and rate-limit annotation were restored. Profiling was disabled; the temporary generator VM and disk were deleted.
• All 48 sampled response hashes matched after restoration. No test publications were created. Additive prepared-JSON data remains in staging and is ignored by the original app.
• Final 15:51 UTC snapshot still had four nodes and six ingress replicas. Restored ingress bounds are 2–6; node-pool bounds remain 2–4. Natural return to two nodes is unproven.

See latest report and portable evidence, earlier fixed-budget comparison, CPU diagnosis, and benchmark/backfill instructions.