History-limit pruning deletes TaskRuns owned by a still-running PipelineRun, causing the pipeline task to re-run

[l-qing]

Expected Behavior

A TaskRun owned by a PipelineRun must never be pruned independently of its parent. Child TaskRuns are meant to be reclaimed through ownerReference cascade when the parent PipelineRun itself is deleted.

The codebase already states this intent in three places:

• pkg/reconciler/taskrun/controller.go:83,92 — the informer event filter uses isStandaloneTaskRun to drop any TaskRun carrying a tekton.dev/pipelineRun label or a PipelineRun ownerReference.
• pkg/reconciler/taskrun/reconciler.go:45 — ReconcileKind checks it again, with the comment "will be handled by it is parent resource(PipelineRun)".
• pkg/reconciler/tektonpruner/controller.go:303 — the periodic GC only processes TaskRuns where !HasPipelineRunOwnerReference(), and the function comment states "checks if the TaskRun ... is not owned by a PipelineRun before processing".

Count-based cleanup (history limit) should honour the same rule.

Actual Behavior

Those ownership filters only gate which TaskRun triggers the limiter. They do not constrain what gets deleted. The deletion candidate set is re-listed from the API server with no ownership filter at all.

1. pkg/config/history_limiter.go:274-342 — for global- and namespace-level configuration (including the shipped default), the switch identifiedBy falls through to default:, which runs hl.resourceFn.List(ctx, resource.GetNamespace(), "") — every TaskRun in the namespace.
2. pkg/reconciler/taskrun/reconciler.go:124-136 — TrFuncs.List is a plain TaskRuns(namespace).List(...). It excludes neither tekton.dev/pipelineRun nor TaskRuns with a PipelineRun ownerReference.
3. The only subsequent filter is completion status (history_limiter.go:349-355). A completed child TaskRun of a still-running PipelineRun passes it.
4. The list is sorted newest-first (L362-371) and everything past historyLimit is deleted (L374-421).

Consequence: the deleted child TaskRun is recreated and the pipeline task runs again. On the tektoncd/pipeline side (verified against v1.14.1):

• pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go:832-837 — setTaskRunsAndResolvedTask swallows IsNotFound, so t.TaskRuns stays empty.
• Same file, L395-400 — isScheduled() returns len(t.TaskRuns) > 0, i.e. false. isSuccessful, isFailure and IsRunning short-circuit on len == 0 in the same way.
• The pipeline task is therefore treated as never started, re-enters the DAG execution queue, and is recreated by createTaskRuns (pkg/reconciler/pipelinerun/pipelinerun.go:1111).

The task executes a second time and its previous results and status are lost. For non-idempotent tasks — publishing artifacts, provisioning VMs, pushing tags — this is considerably worse than wasted compute.

This affects the out-of-the-box configuration. The shipped config/600-tekton-pruner-default-spec.yaml is enforcedConfigLevel: global with historyLimit: 100, which is exactly the path that lists the whole namespace without an ownership filter.

One precondition is worth stating clearly: this path only fires when a standalone TaskRun completes in the namespace. But a single such trigger sweeps all completed TaskRuns in that namespace beyond historyLimit, children of running PipelineRuns included.

Steps to Reproduce the Problem

1. Install the pruner and shrink the limit so the effect is quick to observe:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: tekton-pruner-default-spec
     namespace: tekton-pipelines
   data:
     global-config: |
       enforcedConfigLevel: global
       successfulHistoryLimit: 2

2. In namespace demo, start a long PipelineRun whose first few tasks finish quickly and whose last task sleeps for 10 minutes. Wait until at least 3 of its child TaskRuns have completed successfully while the PipelineRun is still running.

3. In the same namespace, create and complete one standalone TaskRun (a bare TaskRun not owned by any PipelineRun; a single true step is enough). This is the only thing needed to trigger the TaskRun history limiter.

4. Observe that the pruner deletes every successful TaskRun in demo beyond the newest 2 — including the completed children of the still-running PipelineRun.

5. The PipelineRun controller then recreates the deleted TaskRuns and the corresponding pipeline tasks execute again.

Note that step 4 is invisible at the default log level, because deletions are only logged at DEBUG (see #283). To observe it, either raise the log level or watch the tekton_pruner_controller_resources_deleted_total{resource_type="taskrun",operation="history"} metric.

Additional Info

Observed on a production build cluster (16-day window, pruner pod with no restarts, so the counters cover the full window):

• tekton_pruner_controller_resources_deleted_total{resource_type="taskrun",operation="history"} totals 884 across 5 namespaces, with 377 in the busiest one.
• The age-at-deletion histogram is telling. In one namespace, of 237 history-path deletions, 162 (68%) happened less than 30 minutes after the TaskRun completed, 39 under 10 minutes, and 6 under 5 minutes. Build pipelines on that cluster routinely run longer than half an hour.
• Simulating doResourceCleanup against live cluster state: in a namespace configured with successfulHistoryLimit: 10, 12 TaskRuns were in the deletion set and 8 of them belonged to 2 PipelineRuns that were still running. One more standalone TaskRun completing in that namespace would have deleted all 8.

Still present on main (05bebae): the only occurrence of PipelineRun in pkg/config/history_limiter.go is the metrics resource type at L383; TrFuncs.List is unchanged; the only ownership filters in the repository remain the two trigger-side ones.

Related issues

• Log deletions at INFO level instead of DEBUG #283 (Log deletions at INFO level instead of DEBUG) explains why this has stayed invisible: a deleted TaskRun leaves no log line, and users only see a pipeline task inexplicably running twice, which is easily mistaken for a transient retry.
• Garbage collection is not serialized: safeRunGarbageCollector locks a function-local mutex #352 / fix(controller): serialize garbage collection triggered by ConfigMap updates #353 (garbage collection not serialized) is a different defect, but it widens the window: concurrent sweeps multiply the chance of hitting a running PipelineRun's children. That issue characterises concurrent sweeps as "rather than data corruption", which suggests it was not known at the time that a single count-based sweep can already delete children of a running PipelineRun.

Suggested fix

Exclude PipelineRun-owned TaskRuns from the candidate set in doResourceCleanup, reusing the existing isStandaloneTaskRun criteria:

• server-side and cheap: append the negative label selector !tekton.dev/pipelineRun to TaskRun list calls;
• client-side as a safety net: re-check the PipelineRun ownerReference on the listed results, covering objects where the label is absent.

No change is needed on the PipelineRun side — child TaskRuns are already reclaimed by ownerReference cascade when the parent is deleted.

• Kubernetes version:

Server Version: v1.34.5

• Tekton Pipeline version:

v1.6.0

• Tekton Pruner version:

v0.3.4

[l-qing]

/kind bug

[yuzichen12123]

Thanks for tracking this issue. I investigated the behavior and opened a fix:

#424

The history limiter currently builds its deletion candidates from all completed resources in the namespace. This can include completed TaskRuns that belong to a still-running PipelineRun, either through the tekton.dev/pipelineRun label or a PipelineRun ownerReference. If such a TaskRun is deleted, the PipelineRun controller can recreate it and run the task again.

The PR excludes PipelineRun-owned resources from history-limit cleanup and adds regression coverage for both ownership signals. Local validation passes with go test ./....

@anithapriyanatarajan @infernus01 @pramodbindal @vdemeester could you please take a look when you have time?