Skip to content

Opt-in rotation of portrait video onto the process canvas - #345

Open
kalwalt wants to merge 1 commit into
devfrom
fix/issue-344-portrait-video-rotation
Open

Opt-in rotation of portrait video onto the process canvas#345
kalwalt wants to merge 1 commit into
devfrom
fix/issue-344-portrait-video-rotation

Conversation

@kalwalt

@kalwalt kalwalt commented Aug 18, 2026

Copy link
Copy Markdown
Member

Summary

Closes/relates to #344.

CameraViewRenderer.prepareImage() fits the camera stream into a fixed 320x240 (4:3) process canvas. For a portrait stream this letterboxes it heavily, leaving the NFT tracker much less usable width and pushing markers below the tracking.c 3-point floor sooner than in landscape. This was measured and reported in #344 (~2.3x more tracking points when rotating the frame instead of letterboxing it).

This adds an opt-in videoSettings.rotatePortrait flag (default false, no behavior change unless set). When enabled and the stream is portrait (vh > vw):

  • CameraViewRenderer rotates the frame 90 degrees onto the process canvas instead of letterboxing it (destination footprint is identical to the non-rotated case, only the source draw is rotated).
  • NFTWorker rotates the projection matrix back so the AR content isn't rendered rotated 90 degrees relative to the (correctly oriented, untouched) visible video.
  • NFTWorker now receives the actual runtime video dimensions from CameraViewRenderer (previously it received the requested config width/height, which isn't guaranteed to match what the device actually negotiated).

The shared canvas-fitting math (previously duplicated between CameraViewRenderer.prepareImage() and ARnftUtils.getWindowSize()) is now a single getProcessGeometry() helper, so both stay in sync going forward.

Caveat — please verify on-device

The rotation-direction math for the projection matrix compensation was derived from the canvas transform used here (not copy-pasted from the reporter's own patch, which has a different pipeline), and verified with tsc/prettier, but not verified end-to-end on a real portrait camera stream with an NFT marker — I don't have a device to test with. If the AR content comes out rotated the wrong way on-device, the fix is a one-line sign flip in NFTWorker.ts/NFTWorker.simd.ts, in the if (rot) { ... } block:

- proj[4 * c] = y;
- proj[4 * c + 1] = -x;
+ proj[4 * c] = -y;
+ proj[4 * c + 1] = x;

Test plan

  • tsc --noEmit passes
  • prettier --check passes on changed files
  • Default (flag off) path verified to produce numerically identical geometry to the pre-existing code
  • On-device test with a portrait camera stream + NFT marker (rotation direction, tracking improvement)
  • Confirm landscape behavior is unaffected on a real device

🤖 Generated with Claude Code

Letterboxing a portrait video into the fixed 4:3 process canvas leaves the
NFT tracker with far less usable width, which pushes markers below the
3-point tracking floor sooner. When videoSettings.rotatePortrait is enabled
and the stream is portrait, the frame is now rotated 90 degrees onto the
process canvas instead, and the projection matrix is rotated back to
compensate so the AR content keeps the correct orientation. Landscape
streams and the default (flag off) behavior are unaffected.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@kalwalt kalwalt self-assigned this Aug 19, 2026
@kalwalt kalwalt added bug Something isn't working 🚀 enhancement New feature or request Typescript labels Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working 🚀 enhancement New feature or request Typescript

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant