You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[kafka_consumer] consumer_groups_regex is applied after fetching offsets for all groups, causing a full admin fan-out #24911
Title: [kafka_consumer] consumer_groups_regex is applied after fetching offsets for all groups, causing a full admin fan-out
Type: Bug
Summary:
kafka_consumer enumerates every consumer group in the cluster, issues one AdminClient.list_consumer_group_offsets per group, and applies consumer_groups_regex only afterward. As a result a narrow regex (matching a handful of groups) still fans out one admin request per group across the whole cluster. On a Strimzi cluster with ~9,100 consumer groups, a two-group regex still enqueues ~9,100 offset requests.
Impact:
The regex does not scope the work, so it is not usable as a way to limit load on large clusters.
The mass simultaneous admin-request fan-out is the condition that triggers the librdkafka use-after-free crash on the shared admin coordinator path (rd_kafka_admin_coord_request), see librdkafka Don't assume nested tags exist #4605 / fix PR Remove MIB requirement in profiles #5397. The whole Agent process aborts (exit 139), which on a cluster check runner also takes down co-located checks (kubernetes_state_core, kube_apiserver_metrics, custom_consul).
Expected behaviour:
Filter the group list by consumer_groups_regex before issuing per-group offset requests, so only matching groups incur admin requests.
Notes:
This is fixable on our side without waiting for the upstream librdkafka fix, and it is the change that would let customers use a regex safely on large clusters.
Title: [kafka_consumer] consumer_groups_regex is applied after fetching offsets for all groups, causing a full admin fan-out
Type: Bug
Summary:
kafka_consumer enumerates every consumer group in the cluster, issues one AdminClient.list_consumer_group_offsets per group, and applies consumer_groups_regex only afterward. As a result a narrow regex (matching a handful of groups) still fans out one admin request per group across the whole cluster. On a Strimzi cluster with ~9,100 consumer groups, a two-group regex still enqueues ~9,100 offset requests.
Impact:
Expected behaviour:
Filter the group list by consumer_groups_regex before issuing per-group offset requests, so only matching groups incur admin requests.
Notes:
This is fixable on our side without waiting for the upstream librdkafka fix, and it is the change that would let customers use a regex safely on large clusters.
Environment:
kafka_consumer 6.13.0, Agent 7.76.3, confluent-kafka-python / librdkafka 2.11.1, Kafka 3.9.0 (Strimzi), ~9,100 consumer groups.
References:
ZD 2980971, AGENT-16802. Upstream: librdkafka #4605, PR #5397. Adjacent fan-out issues: integrations-core #18983, #19564, #858.