fix!: Bound the request metric attributes so long-running Relays stay under the cardinality limit - #907
Open
keelerm84 wants to merge 2 commits into
Open
fix!: Bound the request metric attributes so long-running Relays stay under the cardinality limit#907keelerm84 wants to merge 2 commits into
keelerm84 wants to merge 2 commits into
Conversation
… under the cardinality limit The request metrics kept every attribute set they had seen until the process restarted. On a long-running Relay Proxy those sets filled the cardinality limit, and every new request was then reported in the attribute-less otel.metric.overflow series. On LaunchDarkly's own fdv2-stg tier nearly every stream was reported as overflow after about 8 days. http.server.active_requests is an UpDownCounter, which stays cumulative under the delta temporality preference too. Under the SDK's default cumulative temporality, launchdarkly.relay.requests, http.server.request.duration and the events-received counter grow the same way. Three attributes let the sets grow without bound: - launchdarkly.application.version changes with every client deploy. It is removed from every request metric, not only from active_requests, since the other instruments grow the same way on a cumulative export. - http.request.method was recorded verbatim. It is now normalized as the semantic convention defines it: the methods it lists as-is, and anything else as _OTHER. - The status and not-found handlers need no credentials, but recorded the user agent and application tags the caller sent, so any caller could add a series per distinct value. Requests with no LD environment now record user_agent.original and launchdarkly.application.id as not_provided, which the docs already claimed they did. Requests in an environment keep both. What remains grows only as SDKs are released, so the synchronous UpDownCounter is kept. BREAKING CHANGE: launchdarkly.application.version is no longer reported on any metric. Dashboards and alerts that group or filter by it lose that dimension. launchdarkly.application.id is unchanged, and the version is still included in the usage data the Relay Proxy sends to LaunchDarkly.
kinyoklion
approved these changes
Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The request metrics kept every attribute set they had seen until the process restarted. On a
long-running Relay Proxy those sets filled the cardinality limit, and every new request was then
reported in the attribute-less otel.metric.overflow series. On LaunchDarkly's own fdv2-stg tier
nearly every stream was reported as overflow after about 8 days.
http.server.active_requests is an UpDownCounter, which stays cumulative under the delta temporality
preference too. Under the SDK's default cumulative temporality, launchdarkly.relay.requests,
http.server.request.duration and the events-received counter grow the same way.
Three attributes let the sets grow without bound:
request metric, not only from active_requests, since the other instruments grow the same way on a
cumulative export.
it: the methods it lists as-is, and anything else as _OTHER.
tags the caller sent, so any caller could add a series per distinct value. Requests with no LD
environment now record user_agent.original and launchdarkly.application.id as not_provided, which
the docs already claimed they did. Requests in an environment keep both.
What remains grows only as SDKs are released, so the synchronous UpDownCounter is kept.
BREAKING CHANGE: launchdarkly.application.version is no longer reported on any metric. Dashboards and
alerts that group or filter by it lose that dimension. launchdarkly.application.id is unchanged, and
the version is still included in the usage data the Relay Proxy sends to LaunchDarkly.
Note
Overview
Long-running Relay Proxy instances were hitting OpenTelemetry’s per-instrument cardinality cap because cumulative request metrics retained every distinct attribute set until restart. This change caps cardinality on request-scoped OTLP metrics so new traffic doesn’t collapse into
otel.metric.overflow.Breaking:
launchdarkly.application.versionis removed from all request metrics (it still appears in usage data sent to LaunchDarkly). Dashboards that grouped or filtered on that label need another dimension.http.request.methodis now normalized per the HTTP semantic convention: listed methods in the expected case, everything else (including wrong-casegetor invented verbs on unauthenticated routes) maps to_OTHER.Unauthenticated traffic (status endpoints and unmatched routes) no longer records caller-supplied
user_agent.originalorlaunchdarkly.application.id; both are forced tonot_providedviaEnvironmentManager.scope, so scanners can’t mint unbounded series. Authenticated SDK requests keep those attributes unchanged.Docs in
docs/metrics.mdwere updated to match (user-agent header precedence, method normalization, and the unscopednot_providedbehavior). Tests cover method normalization, unscoped attribute stripping, and an integration check that 20 distinct unauthenticated requests still produce only two request-counter series.Reviewed by Cursor Bugbot for commit 8166c8e. Bugbot is set up for automated code reviews on this repo. Configure here.