docs(changelog): Flaky Tests detect consistently failing tests as Broken

[samgutentag]

What shipped

Threshold monitors can now classify consistently failing tests as Broken, distinct from Flaky, via a "Detection type: Broken" setting. Test status priority is Broken > Flaky > Healthy, and the v2.test_case.status_changed webhook now emits BROKEN.

Source

• Eng PR: trunk-io/trunk2#3197 (Add broken state detector)
• Linear: TRUNK-17699 (no absorbed duplicates)
• Date basis: trunk2#3197 mergedAt = 2026-03-10

Wired into all 4 sites

• changelog/2026-03-10-flaky-tests-detect-broken-tests.mdx
• docs.json (Changelog 2026 group)
• changelog/index.mdx (March 2026 section)
• flaky-tests/changelog.mdx (March 2026 section)

Docs link target: https://docs.trunk.io/flaky-tests/detection/failure-rate-monitor (documents the Detection type: Broken setting directly).

🤖 Generated with Claude Code

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
trunk	🟢 Ready	View Preview	May 29, 2026, 5:25 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

[samgutentag]

Verification status (2026-05-28): live

Verified: customers can use this. Ready to publish.

• Eng PR: trunk-io/trunk2#3197 (merged 2026-03-10, merge commit on main)
• Flag: none (no LaunchDarkly gating found in the diff)
• Signals: eng PR merged and intact on main; Broken detection is already documented as shipped in the published detection docs (failure-rate-monitor, detection index)

Next: ready to mark non-draft and publish.

[samgutentag]

Code verification (2026-05-28): 3 confirmed / 0 contradicted / 0 ambiguous / 0 unverifiable

Claim	Verdict	Source
Detection type: Broken setting on threshold monitors marks tests as BROKEN	confirmed	reconcile.ts:411
Test status priority is Broken > Flaky > Healthy	confirmed	reconcile.ts:403-411
v2.test_case.status_changed webhook now includes BROKEN as a status value	confirmed	v2-test-case-status-changed.ts:8

Every factual claim in the entry matches source. The feature ships as a detectionType enum ("flaky" | "broken", default flaky) on the threshold monitor config, with a clickhouse migration adding BROKEN = 3 to the status enum. No contradictions.

---

Source #1 — Detection type Broken marks tests as BROKEN (confirmed)

File: trunk-io/trunk2/ts/apps/flake-detection/src/monitors/reconcile.ts#L411

// Priority-based status resolution: BROKEN > FLAKY > HEALTHY
const status = anyBrokenActive
  ? "BROKEN"
  : anyFlakyActive
    ? "FLAKY"
    : "HEALTHY";

Reasoning: A monitor's detectionType === "broken" produces a BROKEN classification (see detectionType === "broken" ? "BROKEN" : "FLAKY" in the same file). The reconcile step then resolves the test's overall status from active monitors. Confirms the entry's "set Detection type to Broken → tests marked Broken" claim against production code, not just the eng PR README.

Source #2 — Status priority Broken > Flaky > Healthy (confirmed)

File: trunk-io/trunk2/ts/apps/flake-detection/src/monitors/reconcile.ts#L403-L411

// Priority-based status resolution: BROKEN > FLAKY > HEALTHY
const status = anyBrokenActive
  ? "BROKEN"
  : anyFlakyActive
    ? "FLAKY"
    : "HEALTHY";

Reasoning: The ternary evaluates broken before flaky before healthy, so a test with both a broken-type and flaky-type monitor active resolves to BROKEN. Matches the entry's "shows as Broken until the broken monitor resolves" wording exactly. Backed by unit tests in the same PR ("should use BROKEN over FLAKY when both broken and flaky monitors are active (priority)").

Source #3 — v2.test_case.status_changed webhook includes BROKEN (confirmed)

File: trunk-io/trunk2/ts/packages/tools/svix-publish-schemas/src/events/v2-test-case-status-changed.ts#L8

"Emitted when the health status of a test case changes. Test status can transition between HEALTHY, FLAKY, and BROKEN. Learn how test health is detected: https://docs.trunk.io/flaky-tests/detection",

Reasoning: The event definition for v2.test_case.status_changed (registered under that exact key in event-types.ts:34) explicitly documents BROKEN as a possible status value alongside HEALTHY and FLAKY. The backend webhook builder (trunk/py/dagster/assets/flake_detection/webhooks.py:55) emits type: "v2.test_case.status_changed" with an uppercased new_status. Event name and the BROKEN value both confirmed.