feat(metrics): Add runtime-switchable DogStatsD UDS support, migrate Rust to metrics-exporter-dogstatsd, and propagate runtime global tags

[phacops]

Summary

• Replace the Rust metrics pipeline with metrics-exporter-dogstatsd (native DogStatsD protocol, client-side aggregation, UDP + Unix domain socket transports)
• Add Unix domain socket (UDS) support for DogStatsD on both the Python and Rust sides, selectable at runtime via the use_dogstatsd_uds sentry-option
• Add runtime global tag propagation so tags set via set_global_tag() (e.g. assigned_partitions, min_partition during Kafka rebalancing) are included in every DogStatsD metric payload
• Remove duplicate storage/consumer_group tag setting in the Rust consumers, since those are already passed as static labels to the DogStatsD builder

Transport selection

The use_dogstatsd_uds sentry-option (in the snuba namespace, read by both Python via snuba.state.sentry_options.get_option and Rust via sentry_options::options("snuba"), same key) selects the preferred transport. A deployment configured with only one transport uses that one regardless of the flag; the flag is authoritative only when both are configured (a configured socket never overrides an available UDP target while the flag is off), so host/port stay configured as the UDP rollback target.

Configuration	flag false (default)	flag true
host/port and socket	UDP	UDS
host/port only	UDP	UDP
socket only (no host/port)	UDS	UDS
neither	no metrics	no metrics

DOGSTATSD_HOST/DOGSTATSD_PORT come from SNUBA_STATSD_ADDR (or DOGSTATSD_HOST + DOGSTATSD_PORT). The socket address comes from SNUBA_DOGSTATSD_SOCKET_PATH and is a full address including the transport scheme (e.g. unixgram:///run/dogstatsd.sock for datagram, unix:///run/dogstatsd.sock for stream). It is passed verbatim to both runtimes — the Rust exporter parses the scheme, and the Python datadog client strips it and selects the socket kind — so no scheme is hardcoded on either side.

Switching between UDP and UDS (rollout / rollback)

1. Prepare (one deploy): configure both the UDP target and SNUBA_DOGSTATSD_SOCKET_PATH, and leave the use_dogstatsd_uds sentry-option at false. Everything runs on UDP.
2. Switch to UDS: set the use_dogstatsd_uds sentry-option to true, then restart the consumers (rolling restart — no redeploy).
3. Roll back to UDP: set it back to false and restart. Instant rollback, no redeploy.
4. Make UDS permanent: keep the flag on and leave host/port configured as the dormant fallback.

Notes

• With both host/port and a socket configured, the flag toggles between them and host/port is the UDP rollback target. A deployment with only a socket configured sends over UDS regardless of the flag (there is no UDP target to prefer or fall back to); one with only host/port sends over UDP.
• The flag takes effect on the next restart, not live: both runtimes build the backend once (metrics-exporter-dogstatsd's .install() and arroyo's metrics::init() are once-per-process; Python's metrics is a module-level singleton whose client factory resolves the transport once per process on first metric emission). A rolling restart with no code change is the intended mechanism.
• This toggles UDP↔UDS within the new metrics-exporter-dogstatsd backend; there is no runtime fallback to the previous statsdproxy pipeline (that needs a revert).

Runtime config source

The use_dogstatsd_uds flag lives in the sentry-options snuba namespace (schema: sentry-options/schemas/snuba/schema.json), the same store the rest of snuba's runtime config now uses. Both runtimes read the same key: Python through snuba.state.sentry_options.get_option("use_dogstatsd_uds", False) and Rust through sentry_options::options("snuba"). Reads are defensive — an uninitialized options client, a missing key, or a wrong type all default to false (prefer UDP).

Implementation

Python (create_metrics()): returns DummyMetricsBackend only when neither a UDP (host/port) nor a UDS (socket) transport is configured — a static decision, because create_metrics() runs at snuba.environment import time and importing snuba.state.sentry_options there would be a circular import. When a transport is configured it builds the DogStatsd client lazily; the factory resolves the transport once per process on first metric emission (by which point init_options() has run) — UDS when a socket is set and the flag selects it (or there's no UDP target), otherwise UDP. Resolving once per process keeps every thread on the same transport (DatadogMetricsBackend builds a client per thread). The socket address is passed to the datadog client verbatim (the client strips the scheme).

Rust: the new DogStatsDBackend adapts arroyo's Recorder trait to the metrics crate facade installed by the exporter. Histograms are emitted as DogStatsD histograms (h), not distributions, so the agent-side .avg/.count/.median/.95percentile/.max sub-metrics are unchanged — timers move from the ms wire type to h, which Datadog treats identically. A shared create_dogstatsd_backend() with a pure select_transport() applies the same gating as Python; both consumer.rs and accepted_outcomes_consumer.rs go through it. The socket address is passed to the exporter's with_remote_address verbatim (it parses the scheme). The use_dogstatsd_uds_enabled() reader uses sentry_options::options("snuba") (native, no pyo3), and init_with_schemas runs at consumer startup before the metrics backend is built.

Runtime global tags: a thread-safe global tag map (a sorted BTreeMap behind a LazyLock/RwLock) stores tags set at runtime. set_global_tag() writes to both this map and the Sentry scope; record_metric() appends every entry to each metric, so tags like assigned_partitions and min_partition appear in DogStatsD payloads.

Follows the same pattern as getsentry/relay#5675.

Test plan

• cargo test --workspace — passes (the only failures require ClickHouse/Redis/Python infra and fail identically on master in a bare environment)
• cargo clippy --workspace --all-targets -- -D warnings — no warnings
• cargo fmt --check — clean
• Unit tests for the global tag store: set/get, overwrite dedup, BTreeMap sort order (isolated per-test key namespaces)
• Unit tests for the Rust select_transport gating matrix (incl. socket-only → UDS)
• Python tests for create_metrics UDP/UDS/socket-only selection, process-wide transport resolution, verbatim socket address, and the no-transport dummy path
• Verify metrics flow over UDS in staging (set the use_dogstatsd_uds sentry-option to true + restart)
• Verify global tags propagate in DogStatsD payloads in staging

🤖 Generated with Claude Code

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 033eaf7. Configure here.}

[onewland]

This seems fine to me, I'm wondering if you were able to test locally at all; not with unit tests, but by starting a snuba consumer <storage> and seeing some data flow across the wire with the new metrics recorder, maybe with a UDP capture utility or something.

I've just seen that sort of test catch problems before merge before.

[phacops]

Good call. Short of a full snuba consumer <storage> run (that's the last unchecked box — I'll do it in staging), I verified the new recorder actually puts the right bytes on the wire: I pointed DogStatsDBackend at a local socket and captured the raw DogStatsD payloads emitted by record_metric for a counter, gauge, and timer (with a runtime min_partition global tag set), over both transports.

UDP:

snuba.consumer.messages_processed:3|c|#topic:events,min_partition:4,storage:spans,consumer_group:spans-consumers
snuba.consumer.assigned_partitions:7.0|g|#instance:0,min_partition:4,storage:spans,consumer_group:spans-consumers
snuba.consumer.process_latency_ms:30.0|h|#step:decode,min_partition:4,storage:spans,consumer_group:spans-consumers

UDS (unixgram://…): byte-for-byte identical.

Confirms:

• counter → c, gauge → g, timer → h (Datadog treats h and the old ms identically — same .avg/.count/.median/.95percentile/.max sub-metrics);
• prefix applied exactly once (no double-prefix);
• per-metric tags + static storage/consumer_group + the runtime min_partition tag on every line;
• no datadog.dogstatsd.client.* telemetry (disabled so the migration doesn't add metrics);
• UDS emits exactly the same payloads as UDP.

Still to do: the end-to-end capture from a running consumer in staging.

---

Generated by Claude Code