Add OTLP metrics for workers, gRPC, the scheduler, store tiers and health

[amankrx]

What and why

Fixes #2614. NativeLink had OTLP metrics for cache and execution only. Everything else lived in the older #[metric] system or nowhere.

Adds the missing categories: workers (connected, connects, disconnects, keepalives, paused/draining), gRPC (one duration histogram by service, method and status, giving rate, errors and latency together), scheduler matching duration, fast/slow store tier hits and bytes, and health check results.

Also the two execution metrics dropped in #2599 for want of a data source. Memory comes from ActionResourceUsage.peak_memory_kb, which the worker already reported. CPU time is new: the worker samples /proc/<pid>/stat for the process group every 250ms already, so it now reads utime/stime from the same line and reports it as cpu_time_ms.

Connection pools are covered too: free slots at acquisition, whether a caller had to queue, and Redis reconnects. A pool that keeps reporting zero free slots is the bottleneck whatever the latency elsewhere says.

Nothing is labelled by worker id or action digest; both grow without bound.

How this was verified

Unit tests for the metric helpers, the gRPC path splitter, the health status labels, and the /proc/<pid>/stat CPU parser.

Risk

Moderate, mostly in accounting rather than behaviour.

CPU time is cumulative per process and a process that exits disappears from /proc, so the sampler keeps the last value per pid and totals them at the end rather than summing the live group. cutime/cstime are excluded, since they only cover reaped children and would double count.

Worker counts are per scheduler instance; with several schedulers the fleet total is the sum. They are exact by construction: unbounded LRU so nothing is silently evicted, one removal path, and re-adding an id does not double count.

gRPC status comes from response headers. Tonic puts it in trailers when a call fails mid-stream, and trailers are not reachable from that layer without consuming the body, so error rate undercounts mid-stream failures. Rate and latency are exact.

The proto gains cpu_time_ms at tag 5, a new optional field, so an old worker reports zero.

Nothing from #2614 is left out.

---

This change is 

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
nativelink	Ready	Preview	Aug 14, 2026 4:02am
nativelink-aidm	Ready	Preview	Aug 14, 2026 4:02am

[MarcusSorealheis]

LGTM