Skip to content

Add 'Debugging from the UI' section with parallel-worker URLs and known limitations#58

Open
samgutentag wants to merge 1 commit into
mainfrom
sam-gutentag/test-error-debugging
Open

Add 'Debugging from the UI' section with parallel-worker URLs and known limitations#58
samgutentag wants to merge 1 commit into
mainfrom
sam-gutentag/test-error-debugging

Conversation

@samgutentag
Copy link
Copy Markdown
Member

Summary

  • Add a "Debugging a flaky test from the UI" section to flaky-tests/dashboard.mdx covering the parallel-worker URL gap (with per-CI env-var workaround), Bundle Upload ID lookups (support-only today), and CI artifact retention vs ticket resolution lag.
  • Add a "Known limitations" subsection inside that block with <Info> callouts for RSpec::Core::MultipleExceptionError (single exception surfaced today, gem extension pending) and Go subtests reporting 0ms duration (AI analysis caveat + post-process workaround).
  • Cross-link from flaky-tests/get-started/frameworks/rspec/index.mdx to the MultipleExceptionError note and from flaky-tests/get-started/frameworks/gotestsum.mdx to the Go-subtest note.

Why

Sourced from customer feedback mining (cluster test-error-log-detail-debugging-info, verdict partial, 9 pairs across 6 customers). The dashboard page documents the failure-detail UI but doesn't address the recurring gaps: parallel-worker drill-down, Bundle Upload ID search, RSpec MultipleExceptionError visibility, Go-subtest 0ms AI caveat, artifact retention lag.

Items flagged for review

  • Parallel-worker env var workaround — confirm CIRCLE_BUILD_URL / BUILDKITE_BUILD_URL / CI_JOB_URL / the constructed GitHub Actions URL is the right surface to recommend, and that appending it to a JUnit property/system-out block is the recommended approach (vs. a CLI flag we haven't documented).
  • RSpec MultipleExceptionError fix status — copy says "is being extended". Confirm whether the gem extension has shipped; if so, update or remove the limitation.
  • Bundle Upload ID — copy says "contact support" is the only option today. Confirm there's no internal lookup URL pattern we'd rather direct users to in the interim.
  • CI artifact retention guidance — recommendation to push artifacts to S3 is generic; if the team has a preferred pattern (e.g., a Trunk-side artifact storage option on the roadmap), point me at it.
  • Page placementdashboard.mdx is already a split candidate (3 clusters target it). If you'd rather this become a standalone flaky-tests/debugging-from-the-ui.mdx page, the section is self-contained and can be lifted out without restructuring the rest of the dashboard doc.
  • Go cross-link target — pointed gotestsum.mdx (the only Go framework page) at the dashboard limitations section. If there's a more specific Go troubleshooting page planned, the link should move there.

Customer signal

@samgutentag samgutentag added the needs review PR sourced from customer-feedback-mining; needs human scrutiny for accuracy before merge label May 20, 2026
@mintlify
Copy link
Copy Markdown
Contributor

mintlify Bot commented May 20, 2026

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
trunk 🟢 Ready View Preview May 20, 2026, 11:53 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@samgutentag
Copy link
Copy Markdown
Member Author

samgutentag commented May 26, 2026

Verification status (2026-05-29): live

Verified: customers can use this. Ready to publish.

  • Flag state: no feature flag involved; LaunchDarkly not consulted (nothing to read)
  • Eng PR: none (customer feedback-sourced debugging content)
  • Flag: none
  • Signals: No engineering dependency and no feature flag. This PR documents existing UI behaviors and known limitations (parallel-worker URL gap, Bundle Upload ID lookups, RSpec MultipleExceptionError, Go-subtest 0ms duration) sourced from 9 customer pairs across 6 channels. All described behaviors and workarounds apply to the currently-shipped product.

Ready to publish after editorial and eng review of flagged items.


Generated by Claude Code

@samgutentag samgutentag added ready to merge Verify docs PR: customers can use this. Ready to publish. and removed needs review PR sourced from customer-feedback-mining; needs human scrutiny for accuracy before merge labels May 26, 2026
If a failing test raises `RSpec::Core::MultipleExceptionError`, only one of the captured exceptions appears in the failure detail view today. See [Known limitations](../../../dashboard#known-limitations) for the current status and workaround.
</Info>

### Environment Variables
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

promote this to a h2 tag

Copy link
Copy Markdown
Member Author

Verification status (May 31, 2026): live

Debugging-from-UI guide covers existing, shipped behaviors — no flag or eng dependency.

  • Flag state: not consulted
  • Eng PR: none
  • Flag: none
  • Signals: documents existing UI debugging flows already available to all customers

Next action: merge this PR.


Generated by Claude Code

Copy link
Copy Markdown
Member Author

Docs Verify Sweep · June 1, 2026 · ✅ live

Feature is live in production. Ready to merge.

Unchanged from previous sweep (May 31, 2026).


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready to merge Verify docs PR: customers can use this. Ready to publish.

Development

Successfully merging this pull request may close these issues.

1 participant