GitHub check-run update 404s right after creation (read-after-write lag) aborts PipelineRun creation — needs retry

[gunther432]

Issue

update_check_run (PATCH /repos/{owner}/{repo}/check-runs/{id}) intermittently fails with 404 Not Found for a check-run that does exist — and the whole PipelineRun creation is aborted and reported as a failed check ("There was an error creating the PipelineRun: ... cannot use the API on the provider platform to create a in_progress status: PATCH .../check-runs/: 404").

Root cause (verified)

GitHub API read-after-write eventual consistency. Forensics from a burst on 2026-08-11 (PaC v0.45.0, GitHub App auth):

• Controller log shows the PATCH failing 200-630 ms after the check-run was created (same reconcile burst; operation":"update_check_run","duration_ms":630,...,"status_code":404).
• Querying the SAME four check-run IDs ~1h later returns 200 OK — the check-runs existed all along; the PATCH simply hit a replica that had not seen the fresh row yet.
• Reproduces under bursts (e.g. a merge queue doing update-branch on several PRs), where create-to-update gaps are milliseconds. 11 occurrences in 12h across two repos.

Impact

The PipelineRun is never created; the PR is left with a permanently-red check that no /retest can revive (CEL event == "pull_request" gates), so automation that treats red as terminal deadlocks. We work around it with a janitor that detects the failure signature and does close+reopen to re-emit the PR event.

Suggested fix

Retry with short backoff (e.g. 3x with 500ms-1s) on 404 responses to check-run update/create-status calls made shortly after check-run creation. A 404 immediately after a successful create is far more likely to be replica lag than a truly missing resource.

Environment

• Pipelines-as-Code v0.45.0 (images ghcr.io/tektoncd/pipelines-as-code/*:stable), GitHub App provider, self-hosted Tekton on k8s.

[chmouel]

Thanks for the detailed investigation. We agree this is a genuine transient-404
failure: PAC creates the check run, receives its ID, and immediately PATCHes that
exact ID. The later successful GETs for the same IDs show that these instances
are not ordinary missing-resource 404s and are consistent with read-after-write
lag.

We do not want to treat GitHub 404s generally as retryable. A 404 can still
represent a deleted check run, an invalid ID, or another real configuration or
permission problem, and a broad retry could hide those failures.

The implementation should therefore be limited to this correlation:

• track an explicit “created now” flag and retry only a short, bounded series of
  404 responses from the immediate check-run PATCH when PAC created that exact
  check run in the current operation;
• retain the normal terminal 404 behavior for IDs loaded from PipelineRun
  annotations or discovered through check-run lookup;
• keep the existing generic API retry transport unchanged: only this call site
  knows that the target ID came from a successful create in the same operation;
• return the original error if those bounded retries are exhausted;
• add tests for transient recovery and for the non-retryable 404 paths.

This preserves real-404 visibility while addressing the confirmed transient
create-to-update sequence.

and fyi we have started "retrying" since we merged this commit d54ff3e
but not on 404

[chmouel]

Tracked downstream in our Jira: SRVKP-13334 to ease our prioritisation.