Skip to content

fix!: Bound the request metric attributes so long-running Relays stay under the cardinality limit - #907

Open
keelerm84 wants to merge 2 commits into
v9from
mk/SDK-3269/bound-request-attributes
Open

keelerm84 wants to merge 2 commits into
v9from
mk/SDK-3269/bound-request-attributes

Conversation

@keelerm84

@keelerm84 keelerm84 commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

The request metrics kept every attribute set they had seen until the process restarted. On a
long-running Relay Proxy those sets filled the cardinality limit, and every new request was then
reported in the attribute-less otel.metric.overflow series. On LaunchDarkly's own fdv2-stg tier
nearly every stream was reported as overflow after about 8 days.

http.server.active_requests is an UpDownCounter, which stays cumulative under the delta temporality
preference too. Under the SDK's default cumulative temporality, launchdarkly.relay.requests,
http.server.request.duration and the events-received counter grow the same way.

Three attributes let the sets grow without bound:

  • launchdarkly.application.version changes with every client deploy. It is removed from every
    request metric, not only from active_requests, since the other instruments grow the same way on a
    cumulative export.
  • http.request.method was recorded verbatim. It is now normalized as the semantic convention defines
    it: the methods it lists as-is, and anything else as _OTHER.
  • The status and not-found handlers need no credentials, but recorded the user agent and application
    tags the caller sent, so any caller could add a series per distinct value. Requests with no LD
    environment now record user_agent.original and launchdarkly.application.id as not_provided, which
    the docs already claimed they did. Requests in an environment keep both.

What remains grows only as SDKs are released, so the synchronous UpDownCounter is kept.

BREAKING CHANGE: launchdarkly.application.version is no longer reported on any metric. Dashboards and
alerts that group or filter by it lose that dimension. launchdarkly.application.id is unchanged, and
the version is still included in the usage data the Relay Proxy sends to LaunchDarkly.


Note

Overview
Long-running Relay Proxy instances were hitting OpenTelemetry’s per-instrument cardinality cap because cumulative request metrics retained every distinct attribute set until restart. This change caps cardinality on request-scoped OTLP metrics so new traffic doesn’t collapse into otel.metric.overflow.

Breaking: launchdarkly.application.version is removed from all request metrics (it still appears in usage data sent to LaunchDarkly). Dashboards that grouped or filtered on that label need another dimension.

http.request.method is now normalized per the HTTP semantic convention: listed methods in the expected case, everything else (including wrong-case get or invented verbs on unauthenticated routes) maps to _OTHER.

Unauthenticated traffic (status endpoints and unmatched routes) no longer records caller-supplied user_agent.original or launchdarkly.application.id; both are forced to not_provided via EnvironmentManager.scope, so scanners can’t mint unbounded series. Authenticated SDK requests keep those attributes unchanged.

Docs in docs/metrics.md were updated to match (user-agent header precedence, method normalization, and the unscoped not_provided behavior). Tests cover method normalization, unscoped attribute stripping, and an integration check that 20 distinct unauthenticated requests still produce only two request-counter series.

Reviewed by Cursor Bugbot for commit 8166c8e. Bugbot is set up for automated code reviews on this repo. Configure here.

… under the cardinality limit

The request metrics kept every attribute set they had seen until the process restarted. On a
long-running Relay Proxy those sets filled the cardinality limit, and every new request was then
reported in the attribute-less otel.metric.overflow series. On LaunchDarkly's own fdv2-stg tier
nearly every stream was reported as overflow after about 8 days.

http.server.active_requests is an UpDownCounter, which stays cumulative under the delta temporality
preference too. Under the SDK's default cumulative temporality, launchdarkly.relay.requests,
http.server.request.duration and the events-received counter grow the same way.

Three attributes let the sets grow without bound:

- launchdarkly.application.version changes with every client deploy. It is removed from every
  request metric, not only from active_requests, since the other instruments grow the same way on a
  cumulative export.
- http.request.method was recorded verbatim. It is now normalized as the semantic convention defines
  it: the methods it lists as-is, and anything else as _OTHER.
- The status and not-found handlers need no credentials, but recorded the user agent and application
  tags the caller sent, so any caller could add a series per distinct value. Requests with no LD
  environment now record user_agent.original and launchdarkly.application.id as not_provided, which
  the docs already claimed they did. Requests in an environment keep both.

What remains grows only as SDKs are released, so the synchronous UpDownCounter is kept.

BREAKING CHANGE: launchdarkly.application.version is no longer reported on any metric. Dashboards and
alerts that group or filter by it lose that dimension. launchdarkly.application.id is unchanged, and
the version is still included in the usage data the Relay Proxy sends to LaunchDarkly.
@keelerm84
keelerm84 requested a review from a team as a code owner October 5, 2026 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants