Knative Architecture Setup - Activator Instability

[nardine-t]

Hey, we’ve been running into some unexplained instability with our Knative setup and are wondering whether our current architecture is ideal for production reliability.

We’re currently using Kourier along with Istio as an additional security layer. Is this a recommended setup, or would a simpler architecture—such as using Kourier alone—generally be preferable?

The recurring issue we’re seeing is that the Knative Activator appears to fail or become a bottleneck. When this happens, requests stop flowing end-to-end, even though the backend pods have already scaled up and there is still queued work waiting to be processed.

This has us questioning whether our current request path and scaling configuration are contributing to the problem. Has anyone encountered similar behavior?

We’d especially appreciate recommendations around:

• Whether Kourier + Istio is a reasonable architecture versus using Kourier alone
• Activator configuration or tuning
• Concurrency and target settings
• Ingress configuration/choice
• Scaling class or autoscaling adjustments
• Other approaches for ensuring traffic continues flowing reliably during load spikes

[Yash121l]

From the code, the activator is not scaling anything itself: per revision it holds a breaker with capacity containerConcurrency times the pods assigned to this activator, requests block in that breaker, and past a fixed 10000-deep pending queue it answers 503 "pending request queue full" (pkg/activator/net/throttler.go). The Istio part matters. The activator probes pod IPs directly, and if those probes fail in a way that looks mesh-caused it sets podsAddressable = false for the revision and falls back permanently to the ClusterIP; capacity then becomes containerConcurrency * readyBackends / numActivators and per-pod balancing is left to kube-proxy, which is the shape of your symptom, one activator pinned while pods sit idle. A second stale-state path: numActivators comes from the activator Service's healthy endpoints, and if this activator's own IP is missing from that set the update returns early and capacity is never recomputed.

Three things to pull during a spike: the activator log line Set capacity to N (backends: B, index: i/n), whether ClusterIP is successfully probed appears for the affected revision, and the revision's containerConcurrency next to the activator replica count. If the mesh fallback is what you are hitting, either exclude activator and queue-proxy from strict mTLS or pin the behaviour instead of letting it be inferred:

# config-network ConfigMap, knative-serving namespace
data:
  mesh-compatibility-mode: "disabled"

Does that match what you see?