Skip to content

[kafka_consumer] consumer_groups_regex is applied after fetching offsets for all groups, causing a full admin fan-out #24911

Description

@prabhusiva619

Title: [kafka_consumer] consumer_groups_regex is applied after fetching offsets for all groups, causing a full admin fan-out

Type: Bug

Summary:
kafka_consumer enumerates every consumer group in the cluster, issues one AdminClient.list_consumer_group_offsets per group, and applies consumer_groups_regex only afterward. As a result a narrow regex (matching a handful of groups) still fans out one admin request per group across the whole cluster. On a Strimzi cluster with ~9,100 consumer groups, a two-group regex still enqueues ~9,100 offset requests.

Impact:

  1. The regex does not scope the work, so it is not usable as a way to limit load on large clusters.
  2. The mass simultaneous admin-request fan-out is the condition that triggers the librdkafka use-after-free crash on the shared admin coordinator path (rd_kafka_admin_coord_request), see librdkafka Don't assume nested tags exist #4605 / fix PR Remove MIB requirement in profiles #5397. The whole Agent process aborts (exit 139), which on a cluster check runner also takes down co-located checks (kubernetes_state_core, kube_apiserver_metrics, custom_consul).

Expected behaviour:
Filter the group list by consumer_groups_regex before issuing per-group offset requests, so only matching groups incur admin requests.

Notes:
This is fixable on our side without waiting for the upstream librdkafka fix, and it is the change that would let customers use a regex safely on large clusters.

Environment:
kafka_consumer 6.13.0, Agent 7.76.3, confluent-kafka-python / librdkafka 2.11.1, Kafka 3.9.0 (Strimzi), ~9,100 consumer groups.

References:
ZD 2980971, AGENT-16802. Upstream: librdkafka #4605, PR #5397. Adjacent fan-out issues: integrations-core #18983, #19564, #858.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions