Report cancelled and timed-out Dispatcher batches and jobs

[AAraKKe]

What does this PR do?

Reports cancelled and timed-out Dispatcher batches and jobs as their own outcomes. GitHub reports a job stopped by timeout-minutes as cancelled, so the Dispatcher tells the two apart itself.

• Timeouts: a completed job that ran (a runner was assigned, runner_name), concluded cancelled and lasted at least the test jobs' 120-minute limit timed out. It fails the job, its batch and the run, and counts as timed_out in metrics. TEST_JOB_TIMEOUT_MINUTES is tested against both jobs in test-batch.yml. A shorter cancelled job is a genuine cancellation (Status.CANCELLED). Cancelled jobs that ran now get a job.duration sample, so timed-out jobs show up in duration charts.

• Dense outcome counters (ResultMetric in execution_metrics.py), identical for batches and jobs: passed, failed, skipped, cancelled, inconclusive, timed_out. Each batch or job emits all six, exactly one of them 1.

• One owner for outcome counters: the gatherer. It counts each planned job once, at the first progress update showing it finished, or at gathering for jobs never observed finished (from their artifacts). A finished job keeps its outcome whatever happens to its batch later. The runner keeps job.duration and the completion log.

• Batch outcome: a genuinely cancelled batch is cancelled; otherwise a job failure outranks a timeout, which outranks the workflow's own conclusion.

• Dispatcher cancellation: at shutdown the Dispatcher hands the dispatched batches to TaskTestGatherer.report_unfinished. On a cancellation, every batch and job without an outcome is counted cancelled, instead of jobs.incomplete. Other shutdowns keep jobs.incomplete, excluding jobs whose outcome was already counted. Once stopping, the bus no longer delivers progress updates, so a job that finished in the last poll before a cancellation is counted cancelled.

• Run outcome and PR comment share DispatcherProgress.has_failure: a failed batch, or a failed job in a batch that was not cancelled. A cancelled batch without failures makes the run cancelled. Cancelled batches and jobs render as 🚫. The CLI aborts a cancelled run with "Dispatcher tests were cancelled." (same exit code).

• dispatcher.batch.id is no longer a metric tag. Batch IDs are plan positions (batch-00, batch-01, ...) and nothing queries metrics by them. The only metrics that needed the tag were the jobs.queued and jobs.running gauges, to keep concurrent batches from overwriting each other; the runner now emits their per-platform totals across in-flight batches instead.

• Cleanup: removed state with no production reader or writer: BatchFinished.timed_out (never set by the runner) with the gatherer branches and ProgressError.TIMED_OUT it fed, and the gatherer's _status_by_batch/_results_by_batch registries with WorkflowStatus. Tests that read the registries now assert the published progress or emitted metrics.

The commit status that test-batch.yml posts is unchanged: GitHub commit statuses have no cancelled state.

Motivation

conclusion_to_status mapped cancelled to FAILURE, but GitHub uses cancelled both for a genuine cancellation and for a job that hit timeout-minutes. Job 110448952220 ran 120.4 minutes and reports conclusion: cancelled; only its annotation says "The job has exceeded the maximum execution time of 2h0m0s". The job payload has nothing else that tells them apart, so this uses the duration.

The cancellations worth reporting are the Dispatcher's own: once it is cancelled it stops polling, so GitHub never tells it how its batches ended. In the last 7 days, 6,753 of 6,951 incomplete jobs (97%) came from 28 cancelled Dispatcher runs; they are now counted as cancelled.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[dd-octo-sts]

✅ Dispatcher tests: passed

Dispatcher beta: informational only
Existing CI remains the merge signal.

  2/2 jobs

✅ 2 passed · nothing failed

Batches · ✅ batch-01 2/2

_{Dispatcher finished on e6fa777 — GitHub Run · Dispatcher Logs.}

[cit-pr-commenter-54b7da]

evalya-impact-summary

evalya impact analysis

Impact analysis: RUN-ALL — every test task will run
Trigger:         empty diff (default branch, scheduled run, or shallow-clone fallback)
Test tasks:      0 (all selected)
Publish tasks:   2 (always emitted)
Diff:            empty (no diff information)

Learn more about CI impact filtering

[datadog-prod-us1-3]

 

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 89.51% (+0.10%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 818845c | Docs | View more details | Give us feedback!}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-06T13:10:12.314416Z	ff698d0	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e076b665ae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Let failed jobs outrank a cancelled batch

When a cancelled workflow contains an unconfirmed job whose JUnit artifacts contain failures, _unconfirmed_job_status marks the job as FAILURE, while _finished_batch_progress keeps the batch status as CANCELLED. This outcome calculation only inspects batch statuses, so it returns CANCELLED even though progress.failed > 0; the PR comment's _has_failure check consequently renders the same run as failed while the CLI and run metrics report it as cancelled. Check the collected job failures before selecting the cancelled outcome.

Useful? React with 👍 / 👎.

[AAraKKe]

Fixed, though not by letting failed jobs outrank the cancelled batch. A cancelled batch is discarded, so whatever it left unfinished should not count:

• An unconfirmed job in a cancelled run is now reported cancelled whatever its artifacts hold.
• The run outcome and the PR comment heading now share one rule, DispatcherProgress.has_failure: a failed batch, or a failed job in a batch that was not cancelled.

A job that completed with failure before the cancellation still counts in jobs.failed, but no longer fails the run, so the CLI, the run metrics and the comment agree.

[chatgpt-codex-connector]

Preserve failing reports from completed cancelled jobs

If tests have already written failing JUnit and the job is then cancelled during the workflow's if: always() report collection or upload, GitHub can return a completed job with conclusion cancelled alongside those reports. The completed-job path in _job_status applies this new mapping directly and never examines the reports, classifying the job as cancelled and dropping the real test failure; by contrast, the newly added unconfirmed-job path correctly lets failed artifacts outrank cancellation. Apply the same failed-report precedence for completed cancelled jobs.

Useful? React with 👍 / 👎.

[AAraKKe]

Not changing this. A cancelled job is discarded: anything it had not finished counts for nothing, including failing reports from steps the cancellation interrupted. Only outcomes that completed before the cancellation are reported.

[AAraKKe]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: ff698d061f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[dd-octo-sts]

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and code coverage settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
license-headers	Validate Python files have proper license headers	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
package	Validate Python package metadata and naming	✅
qa-label	Validate the pull request declares whether it needs QA for the next Agent release	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run