EOF when running linkerd tap command against a category of resources on a k3d cluster

[cpretzer]

What is the issue?

Running the linkerd tap command against a control plane deployed to a k3d cluster results in an EOF error when targeting an entire category of kubernetes resources (e.g. Pod or Deployment).

The same command run against a specific resource (deploy/linkerd-prometheus, or a specific pod) works as expected.

How can it be reproduced?

• Provision a cluster using k3d (The number of nodes doesn't matter)
  -- k3d cluster create tap-test

• Deploy the linkerd control plane
  -- linkerd install | kubectl apply -f -

• Run tap against all resource types in the linkerd control plane
  -- Pods: linkerd tap -n linkerd po --verbose
  -- Deployments: linkerd tap -n linkerd deploy --verbose

Logs, error output, etc

DEBU[0000] Response from [https://0.0.0.0:43867/apis/tap.linkerd.io/v1alpha1/watch/namespaces/linkerd/pods//tap] had headers: map[Cache-Control:[no-cache, private] Content-Length:[205] Content-Type:[application/json] Date:[Thu, 12 Nov 2020 17:58:32 GMT]] 
DEBU[0000] Waiting for data...                          
error reading byte stream header: error while reading bytes from message: unexpected EOF

linkerd check output

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days

linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ tap api service is running

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match

linkerd-prometheus
------------------
√ prometheus add-on service account exists
√ prometheus add-on config map exists
√ prometheus pod is running

linkerd-grafana
---------------
√ grafana add-on service account exists
√ grafana add-on config map exists
√ grafana pod is running

Status check results are √

Environment

• Kubernetes Version: v1.18.9+k3s1
• Cluster Environment: k3d
• Host OS: "Pop!_OS 20.04 LTS" (Ubuntu 20.04)
• Linkerd version: Found on edge-20.11.1, also happens with stable-2.9 and stable-2.8.1

Additional context

When targeting a specific deployment or pod, the command works and the output is displayed:

linkerd tap -n linkerd deploy/linkerd-prometheus
req id=0:0 proxy=in  src=10.42.0.1:47096 dst=10.42.0.17:9090 tls=no_tls_from_remote :method=GET :authority=10.42.0.17:9090 :path=/-/ready
rsp id=0:0 proxy=in  src=10.42.0.1:47096 dst=10.42.0.17:9090 tls=no_tls_from_remote :status=200 latency=833µs
end id=0:0 proxy=in  src=10.42.0.1:47096 dst=10.42.0.17:9090 tls=no_tls_from_remote duration=66µs response-length=21B
req id=0:1 proxy=out src=10.42.0.17:41954 dst=10.42.0.14:4191 tls=true :method=GET :authority=10.42.0.14:4191 :path=/metrics
rsp id=0:1 proxy=out src=10.42.0.17:41954 dst=10.42.0.14:4191 tls=true :status=200 latency=3033µs
end id=0:1 proxy=out src=10.42.0.17:41954 dst=10.42.0.14:4191 tls=true duration=260µs response-length=2628B
req id=0:2 proxy=out src=10.42.0.17:48764 dst=10.42.0.10:4191 tls=true :method=GET :authority=10.42.0.10:4191 :path=/metrics
rsp id=0:2 proxy=out src=10.42.0.17:48764 dst=10.42.0.10:4191 tls=true :status=200 latency=1847µs
end id=0:2 proxy=out src=10.42.0.17:48764 dst=10.42.0.10:4191 tls=true duration=46µs response-length=4002B

I've checked the logs from all the pods in the kube-system and linkerd and nothing is logged when the tap command fails.

[adleong]

Just a guess, but perhaps k3d doesn't like the blank resource name in the API path:

/apis/tap.linkerd.io/v1alpha1/watch/namespaces/linkerd/pods//tap

(notice the blank segment between the resource type pods and the subresource tap)

[adleong]

If this works on non k3d clusters, I think we may just need to document this as a shortcoming of k3d.

[amarkdotdev]

Opened #15487 — collection taps now use .../pods/tap instead of .../pods//tap so k3d/k3s don't hit unexpected EOF.