Skip to content

Fix black/frozen desktop video stream and make render failures visible - #2929

Merged
lukemarsden merged 7 commits into
mainfrom
feature/002546-fix-black-desktop-video
Aug 7, 2026
Merged

lukemarsden merged 7 commits into
mainfrom
feature/002546-fix-black-desktop-video

Conversation

@lukemarsden

Copy link
Copy Markdown
Collaborator

Summary

Luke hit a black video stream on Safari with a provably healthy server pipeline
(60 fps NVENC, pipeline_dropped=0, no backpressure, one subscribed client, proxy
socket never closed) and no error message for 7 minutes.

The reported incident was not caused by the two Jul 29 streaming commits, and not
by WebGL context loss. It was a deadlock in PlayoutScheduler, reproduced live in
the inner Helix. A second, independent silent-black mode (no WebGL context-loss
handling) was also found and fixed, along with the reason either one was invisible.

Full write-up with observed values: design/2026-08-04-black-video-stream-regression.md.

Root cause

Presentation was gated on a preroll flag cleared only by q.length > target, while
push() caps the queue at MAX_QUEUE = 30. Once targetFrames >= 30 that test is
unsatisfiable, so nothing was ever presented again and every frame was dropped at
the cap — canvas frozen, socket and decoder perfectly healthy.

targetFrames reached 152 because the depth cap was floor(MAX_DELAY_MS / median)
where median is median socket inter-arrival spacing, not frame cadence. Any
stall produces a catch-up burst that collapses arrival spacing below 1 ms, turning a
"120 ms max buffer" rule into a 152-frame (2.5 s) buffer. Peak-hold latched it
instantly; decay was 1 frame per 2 s, i.e. >4 minutes to recover.

Captured live while frozen:

{"targetFrames":152,"prerolling":true,"queueLen":30,"MAX_QUEUE":30,
 "depthMs":2508,"canEverClearPreroll":false}

Introduced in bcf7a34f9 (2026-06-17), not the Jul 29 commits. Engine-independent —
reproduced in Chromium.

Changes

playout-scheduler.ts — cadence now measured from PTS deltas (immune to bursts);
depth target capped below MAX_QUEUE so the preroll test is structurally reachable;
preroll ends at >= target (was >, also costing a needless frame of latency).

webgl-video-renderer.ts — handle webglcontextlost/webglcontextrestored and
reallocate GL state on restore; draw() returns false while lost so a non-paint is
never counted as a paint; capture the WEBGL_lose_context handle while the context is
alive (getExtension returns null once lost, which silently defeated restore);
dispose() no longer calls loseContext() — it permanently bricked the canvas.

websocket-stream.ts — setCanvas() reuses a live render target instead of
tearing it down and re-acquiring a dead context; close() no longer destroys render
targets (it also runs for bitrate/mode switches — renderer lifetime belongs to the
canvas, not the socket); deleted the broken 2D fallback on the WebKit branch
(getContext("2d") returns null on a WebGL-bound canvas, so it silently dropped every
frame); added a render watchdog; retry budget refunded on first paint, not first
decode; a deferred-while-hidden reconnect now polls document.hidden.

DesktopStreamViewer.tsx — surface renderStalled as a real message with Retry,
reusing the existing gaveUp error surface; remount the <canvas> when its context is
unrecoverable, which is the only way to obtain a working context again.

Testing

End-to-end in the inner Helix against a real spec-task desktop with a live 60 fps
stream, measured in actual paints (framesDecoded counts decode attempts and kept
climbing at 60/s while nothing was on screen — it is not a health signal).

Scenario Before After
Steady state (scheduler wedged) 0 fps, queue pinned 30, depth 2508 ms 56.4 fps, depth 50 ms
Second setCanvas(), same element 0 fps, both targets null 59.0 fps
Restorable context loss black forever auto-recovers, 59.0 fps
Unrestorable context loss black forever, silent watchdog → canvas remount → 47.3 fps
Desktop container restarted under viewer silent black real error + cause + Retry, then auto-recovers
12 forced disconnect/reconnect cycles — no gaveUp latch, video live
Visible again with no visibilitychange deferred forever reconnects via poll
Chrome 2D path (stub removed) — 51.3 → 50.0 fps across re-attach

NOT tested: real Safari — none available in this Linux sandbox. The WebKit branch
was exercised under Chromium by stubbing isAppleWebKit() to true (reverted before
commit) and driving loss with WEBGL_lose_context. That verifies the logic and
recovery, not WebKit's own context-loss timing. The primary defect is
engine-independent and was reproduced directly.

A synchronous console close() + reconnect() still yields 0 fps because the server
dedups the reused client_id (Close received after close). That is not a viewer code
path; the real Retry path is verified above.

Screenshots

Silent black before — error surfaced after fix
Recovered after context loss
Real error during desktop restart
Auto-recovered after restart


🔗 Open in Helix

📋 Spec:

🚀 Built with Helix

The scheduler could latch a preroll hold it could never release, stopping
all presentation while frames kept arriving and being dropped. The canvas
froze (or stayed black) while socket, decoder and encoder stayed healthy.

Two causes, both fixed at source:

- Buffer depth was derived from median socket ARRIVAL spacing. After any
  stall the catch-up burst collapses that to well under 1ms, so
  floor(MAX_DELAY_MS/median) inflated the depth target from ~7 frames to
  ~150 - a 2.5s buffer against a 120ms cap. Cadence now comes from PTS
  deltas, which bursts cannot distort.

- Preroll ended only when queue length EXCEEDED the target, unreachable
  once the target hit the 30-frame queue cap. It now ends at >= target,
  and the target is capped below queue capacity so the condition is
  structurally always satisfiable.

Observed before: targetFrames=152, prerolling=true, queueLen=30 pinned,
0 frames presented in 6s, framesDropped climbing ~44/s.
Observed after: 56.4fps presented, targetFrames=1, depth 50ms, queueLen=4.

Spec-Ref: helix-specs@917c35d06:002546_debug-and-fix-the
…ck video

A lost GPU context made every GL call a silent no-op: decode kept
succeeding, videoStarted had already fired, stats stayed green, and the
canvas was black indefinitely with no error. Safari hits this routinely.

- WebGLVideoRenderer now handles webglcontextlost/webglcontextrestored,
  reallocating GL objects on restore. draw() returns false while lost so a
  non-paint is never counted as a paint. The WEBGL_lose_context handle is
  captured while the context is alive (getExtension returns null once lost,
  which silently defeated restore requests).
- dispose() no longer calls loseContext(): a canvas hands out one context
  per type for its lifetime, so force-losing it bricked the element. A
  second setCanvas() on the same canvas then got the dead context back,
  failed, and fell through to getContext('2d') which returns null - leaving
  both render targets null and every frame silently dropped.
- setCanvas() reuses a live render target instead of tearing it down, and
  close() no longer destroys it (close() also runs for bitrate/mode
  switches; renderer lifetime belongs to the canvas, not the socket).
- Deleted the broken 2D fallback on the WebKit branch.
- Render watchdog: socket open + decoding but nothing painted for 5s emits
  renderStalled, and the viewer shows a real message with Retry. Nothing
  covered this window before - videoStarted fires from the decode path.
- An unrestorable context is escaped by remounting the canvas element,
  which is the only way to obtain a working context again.
- Retry budget is refunded on first PAINT rather than first decode, and a
  deferred-while-hidden reconnect self-heals if visibilitychange is missed.

Spec-Ref: helix-specs@ccaa1126f:002546_debug-and-fix-the
Spec-Ref: helix-specs@ccaa1126f:002546_debug-and-fix-the
Spec-Ref: helix-specs@ccaa1126f:002546_debug-and-fix-the
The deferral was only lifted by the visibilitychange listener, but the
heartbeat that would otherwise re-check is stopped in onClose - so if that
one event was missed the stream stayed dead with no way back. Verified:
hidden + disconnect defers, then becoming visible WITHOUT dispatching
visibilitychange now reconnects (it did not before).

Spec-Ref: helix-specs@7633544ba:002546_debug-and-fix-the
Spec-Ref: helix-specs@7633544ba:002546_debug-and-fix-the
Spec-Ref: helix-specs@a11df3244:002546_debug-and-fix-the
@cursor

cursor Bot commented Aug 4, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@lukemarsden
lukemarsden merged commit b57b1ed into main Aug 7, 2026
5 checks passed
@lukemarsden
lukemarsden deleted the feature/002546-fix-black-desktop-video branch August 7, 2026 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant