fix(cargo-anvil): preserve mutation runner resource diagnostics

[Ralf Biedert (ralfbiedert)]

Change

Linux PR mutation groups disable unused core dumps and stream memory, disk,
process CPU/RSS, and the latest cargo-mutants debug record every 30 seconds.
PR mutation recipes print caught and unviable mutations as they complete.
After the group finishes, the shared action attempts to preserve its logs and
mutants.out in an OS/architecture-specific artifact with three-day retention.

This supports the investigation of repeated Linux x64 and ARM64 runner losses
on microsoft/oxidizer#797. The cause of those losses remains undetermined.
Disabling unused core files removes one avoidable resource cost; this change
does not claim to fix every possible reason for a runner disappearing.

Mutation selection, parallelism, timeout policy, and authoritative exit status
are unchanged. Diagnostic upload is supplemental. A runner that disappears
before upload may still lose local files. The sampler is stopped on step exit.

Verification

• Added behavioral tests executing the actual Bash template for success,
  recipe failure, other groups, and non-Linux mutation groups.
• Existing cargo-anvil unit and integration suites passed; updated and checked
  all three emitted-tree snapshots and ran doctests.
• Ran repository formatting, README generation, and spelling checks, plus
  Windows and Linux Clippy.
• Regenerated this repository from the modified generator and confirmed a
  subsequent dry run has no pending changes.

[Copilot]

🟡 Changes recommended

The mutation contract documentation is stale, and the new test depends on host processes and timing.

2 open findings

• Avoid host-dependent race in sampler test · New
• Update design contract for new mutation command flags · New

What changed in this PR

Adds Linux mutation-run diagnostics and short-lived artifact preservation to investigate runner losses.

Changes:

• Samples runner resources and disables mutation core dumps.
• Uploads mutation logs and mutants.out.
• Streams mutation outcomes and adds behavioral coverage.

File	Description
.anvil.lock	Updates generated-file checksums.
.github/​actions/​anvil-run-group/​action.yml	Emits generated diagnostics behavior.
justfiles/​anvil/​checks/​mutants-diff.just	Prints caught and unviable mutations.
crates/​cargo-anvil/​templates/​github/​run-group-action.yml	Defines resource sampling and artifact upload.
crates/​cargo-anvil/​templates/​justfiles/​anvil/​checks/​mutants-diff.just	Defines mutation reporting flags.
crates/​cargo-anvil/​tests/​github_mutation_diagnostics.rs	Tests diagnostic behavior.
crates/​cargo-anvil/​tests/​snapshots/​snapshots__github_backend.snap	Updates GitHub output snapshot.
crates/​cargo-anvil/​tests/​snapshots/​snapshots__ado_backend.snap	Updates ADO output snapshot.
crates/​cargo-anvil/​tests/​snapshots/​snapshots__local_only.snap	Updates local output snapshot.
crates/​cargo-anvil/​docs/​design/​github.md	Documents mutation diagnostics.

🧠 Review effort: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.9%. Comparing base (b0fb0d1) to head (0324c0a).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #239   +/-   ##
=====================================
  Coverage   98.9%   98.9%           
=====================================
  Files        304     304           
  Lines      44436   44469   +33     
=====================================
+ Hits       43975   44008   +33     
  Misses       461     461           

Flag	Coverage Δ
linux	98.9% <ø> (+<0.1%)	⬆️
linux-arm	98.9% <ø> (-0.1%)	⬇️
scheduled	?
windows	98.8% <ø> (+<0.1%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🔵 Needs a closer look

The tests do not exercise debug-log streaming and use assertions that can accept malformed action outputs.

0 open findings

2 resolved since last review

• Avoid host-dependent race in sampler test
• Update design contract for new mutation command flags

Previously missed (2)

In code that hasn't changed since last review

Add coverage for streaming only the debug log's final record

crates/​cargo-anvil/​tests/​github_mutation_diagnostics.rs:54

No scenario creates mutants.out/debug.log, so the new branch that streams its latest record is never executed; a wrong path or use of more than the final record would go unnoticed. Add a synchronized case that makes a multi-line debug log appear between samples and verifies that the next sample contains only its last line.

Assert exact action output instead of permissive substring matches

crates/​cargo-anvil/​tests/​github_mutation_diagnostics.rs:114

These substring checks can pass for a misspelled output key such as not_exit_code=… and do not detect unexpected action outputs. Since the helper controls both lines and their order, compare the complete output so this test protects the composite action's exact output contract.

🧠 Review effort: Balanced

[Ralf Biedert (ralfbiedert)]

Follow-up 6dedb65 fixes the crash-handler behavior exposed by PR797's diagnostics. ulimit -c 0 did not prevent Ubuntu's piped apport handlers: the Linux job exhausted 16 GiB RAM and 3 GiB swap while many handlers occupied ~100 MiB each. Linux explicitly ignores RLIMIT_CORE for piped handlers: https://man7.org/linux/man-pages/man5/core.5.html.

Only ephemeral GitHub-hosted Linux PR mutation jobs temporarily use kernel.core_pattern=core, making the existing zero limit effective. An EXIT cleanup restores the original pattern; self-hosted runners and all mutation selection/assertions are unchanged. Configuration failure stops before the recipe; restoration failure warns without replacing the recipe result.

Nine synchronized Bash regressions cover the generated script, including restoration after recipe failure, no self-hosted changes, and setup/restore failures. All three generation snapshots, strict source workspace lint, pinned formatting, spelling, and the generator build passed. The prior isolated Windows MSRV retry also passed before this push; new-head hosted validation is now running.

[Copilot]

🟡 Changes recommended

The checked-in action omits the template’s hosted-runner core-routing mitigation.

1 open finding

• Regenerate action to disable piped crash handler · New

🧠 Review effort: Balanced

[Copilot]

🔵 Needs a closer look

Privileged kernel configuration and asynchronous shell cleanup warrant final human validation in GitHub-hosted CI.

0 open findings

1 resolved since last review

• Regenerate action to disable piped crash handler

🧠 Review effort: Balanced

[Martin Taillefer (geeknoid)]

No new actionable finding in the current diagnostics implementation. Probe and resource-log failures are guarded so later samples continue; the authoritative Just status is captured before supplemental reporting. Hosted-runner core routing is restored on exit, while self-hosted runners do not change the host-wide setting.

The prior sampling, documentation, and regeneration concerns have been addressed in the current head and are not refiled. The FIFO/scripted-probe tests check continuation after failures without relying on a fixed sampling sleep.

All five passes covered the source templates, behavioral tests, design contract, and generated wiring. No builds, tests, mutation campaigns, or shell fixtures were executed.