Exact desktop UI → isomorphic React mirror (not screenshots, not video).
Lazy Dev walks a target application's UI through accessibility APIs, builds a structured UiNode tree, and renders it as real React controls on your phone or in a local preview. Taps, typing, and gestures on the mirror are translated back into the same Invoke / SetValue / Toggle / CDP input events the desktop app understands.
Phase 0: Tauri config + Mirror Demo (UIA → React controls, Invoke/SetValue/Toggle).
Phase 1: Native region overlay, LAN Axum host, installable PWA pairing, gesture + soft-keyboard lift.
CDP (Electron/Chromium): When the target exposes --remote-debugging-port, Lazy Dev attaches via Chrome DevTools Protocol (AX tree + DOMSnapshot styles) instead of native accessibility APIs — much better for Cursor, VS Code, Chrome, Edge. On Windows, falls back to UIA when no debug port is available. On macOS, CDP is required for mirroring in this release.
Most remote-desktop tools stream pixels. Lazy Dev streams structure:
- Walk the target window and read roles, names, bounds, patterns, and (for CDP) computed styles.
- Normalize everything into a shared
UiNodeschema (crates/schema). - Render that tree with
mirror-ui— buttons become<button>, text fields become<input>, and layout uses normalized coordinates (NormRect) so the same tree fits a phone viewport or a desktop preview. - Reverse user input: a tap on a mirrored button becomes a
ControlEvent, which the Tauri host applies through UIA or CDP on the real control.
The phone never runs the target app. It only receives tree updates and sends control events over the LAN.
+-------------------------------------------------------------------------+
| TARGET DESKTOP APP |
| (Cursor, VS Code, Chrome, Notepad, ...) |
| |
| Optional: lives on Fake Monitor (virtual display) via vdisplay |
+-------------------------------+-----------------------------------------+
|
walk / act | CDP: Accessibility.getFullAXTree
| + DOMSnapshot (styles, layout)
| UIA: IUIAutomation tree (Windows)
v
+-------------------------------------------------------------------------+
| WALK BACKEND (crates/cdp | crates/uia) |
| |
| CDP session --> ax_map --> UiNode UIA walker --> UiNode |
| live_layout --> expand region in source Invoke / SetValue / Toggle |
+-------------------------------+-----------------------------------------+
|
| UiNode + UiDiff + ControlEvent
v
+-------------------------------------------------------------------------+
| SCHEMA (crates/schema) |
| |
| UiNode | NormRect | ControlEvent | PairingSession | ServerMessage |
+---------------+-----------------------------+---------------------------+
| |
local IPC | | WebSocket broadcast
v v
+---------------------------+ +---------------------------------------+
| DESKTOP (apps/desktop) | | LAN HOST (crates/server) |
| | | |
| Config window | | Axum HTTP :8787 |
| Mirror Demo | | POST /pair --> session token |
| Phone Preview | | WS /ws --> UiTree + diffs |
| | | static/ --> built mobile PWA |
| Tauri commands.rs | | |
| state.rs (session) | | control_rx mpsc --> apply on desktop|
+-------------+-------------+ +-------------------+-------------------+
| |
| mirror-ui (packages/) |
+-----------------+-------------------+
|
v
+-------------------------------------------------------------------------+
| MIRROR SURFACES (React) |
| |
| MirrorSurface | MirrorNode | SpatialCanvas | phoneLayout stretch |
| |
| User tap / type / gesture --> ControlEvent --> back to host loop |
+-------------------------------------------------------------------------+
|
v
+-----------------------------+
| MOBILE PWA (apps/mobile) |
| PairScreen --> MirrorSession|
| gestures + keyboard lift |
+-----------------------------+
Phone PWA LAN host (Axum) Tauri backend
--------- --------------- --------------
tap mirrored button
|
| WS: ClientMessage::Control
+------------------> mpsc::UnboundedSender
| |
| | try_recv_control() every 50ms
| +------------------------------->
| | apply_control_event
| | |
| | CDP: Input.dispatch*
| | UIA: Invoke / SetValue
| | |
| | refresh_ui_tree
| WS: ServerMessage::Tree <------------------------------+
| | |
v v v
MirrorSurface re-renders broadcast to all clients target app updates
Fake Monitor keeps the target app on a real OS virtual display so your main screen stays clear while CDP mirroring continues.
Config: launch "Fake Monitor" mode
|
v
popular_apps::attach_to_fake_monitor
|
+--> vdisplay::ensure_fake_monitor
| |
| +-- macOS: CGVirtualDisplay (private API)
| | creates "Lazy Dev Fake Monitor" to the right
| |
| +-- Windows: Virtual Display Driver (named pipe)
|
+--> vdisplay::move_process_to_display
|
+-- Find windows via CGWindowList (owner = "Cursor", ...)
+-- Move via Accessibility API (AXPosition / AXSize)
+-- Verify window center is inside target display bounds
+-- Fallback: AppleScript (only if verification passes)
Phone / Phone Preview still mirror via CDP -- pixels are not streamed;
the desktop window simply lives off your primary monitor.
When you draw a mirror region, only controls inside that NormRect are included in the tree sent to the phone. For CDP targets, Lazy Dev can also reshape the source app so its layout matches the phone viewport height:
Draw region on overlay (Windows) or picker UI
|
v
set_region(NormRect) --> filter tree to region
|
+--> try_auto_live_expand (default on)
| cdp::expand_region_containers
| ties max-height to Phone Preview / mobile canvas size
|
+--> report_phone_view_size(w, h) keeps expand in sync on resize
Revert in-app sizes --> live_expand_opt_out until next region draw
lazy-dev/
|
+-- apps/
| +-- desktop/ Tauri 2 shell + React UI
| | +-- src/ ConfigApp, MirrorApp, RegionPicker, api.ts
| | +-- src-tauri/ Rust: commands, state, popular_apps
| |
| +-- mobile/ Installable PWA (Vite + React)
| +-- src/ PairScreen, MirrorSession, gestures
|
+-- packages/
| +-- mirror-ui/ Shared isomorphic UiNode renderer
|
+-- crates/
| +-- schema/ Protocol types (UiNode, events, pairing, display)
| +-- cdp/ Chrome DevTools Protocol walk + input + live expand
| +-- uia/ Windows UIA walk + macOS window enumeration
| +-- vdisplay/ Virtual display create/destroy + window placement
| +-- overlay/ Transparent region overlay (Windows HWND)
| +-- server/ Axum LAN host (pair API + WebSocket + static PWA)
| +-- input/ SendInput fallback (Windows)
|
+-- fixtures/ Sample UiNode JSON for tests
+------------------------------------------------------------------+
| PRESENTATION |
| apps/desktop (Config, Mirror Demo, Phone Preview) |
| apps/mobile (PairScreen, MirrorSession) |
| packages/mirror-ui (MirrorSurface, MirrorNode, SpatialCanvas) |
+------------------------------------------------------------------+
|
Tauri invoke / fetch + WS
|
+------------------------------------------------------------------+
| DESKTOP HOST (Tauri) |
| commands.rs -- attach, tree refresh, region, host, fake monitor|
| state.rs -- MirrorBackend (UIA | CDP), session, phone_view |
| popular_apps -- CDP launch + Fake Monitor placement |
+------------------------------------------------------------------+
| | |
v v v
+---------------+ +------------------+ +------------------+
| crates/server | | crates/cdp | | crates/uia |
| Axum routes | | CDP client | | UIA walker |
| pair + WS | | ax_map -> UiNode | | actions |
| static PWA | | live_layout | | macOS CG windows |
+---------------+ +------------------+ +------------------+
| | |
+--------------------+--------------------+
|
v
+------------------+
| crates/schema |
| UiNode protocol |
+------------------+
^
|
+------------------+
| crates/vdisplay |
| crates/overlay |
| crates/input |
+------------------+
|
v
+------------------+
| OS + target app |
+------------------+
| Backend | When used | Walk source | Actions |
|---|---|---|---|
| CDP | Electron/Chromium with --remote-debugging-port |
Accessibility.getFullAXTree + DOMSnapshot |
CDP Input, DOM focus, live layout expand |
| UIA | Windows native apps without CDP | IUIAutomation tree |
Invoke, Value, Toggle patterns |
On macOS, mirroring currently requires CDP. The uia crate still provides window enumeration (CoreGraphics) and process visibility helpers.
+---------------------------+ +---------------------------+
| Lazy Dev -- Config | | Lazy Dev -- Mirror Demo |
| | | (or Phone Preview) |
| - attach / detach | | |
| - region picker | | mirror-ui renders UiNode |
| - start LAN host | | local control loop |
| - Fake Monitor controls | | |
| - popular app launcher | +---------------------------+
+---------------------------+
| Type | Role |
|---|---|
UiNode |
Recursive tree: role, name, bounds, children, patterns, styles |
NormRect |
Region in 0-1 coordinates relative to target window |
ControlEvent |
Tap, gesture, set-value, invoke, key forward |
UiDiff |
Incremental tree update for WebSocket bandwidth |
ServerMessage / ClientMessage |
Pairing + tree broadcast + control channel |
DisplayInfo |
Monitor list for Fake Monitor / move-to-display |
- macOS 12+
- Rust
- Node.js 20+
- Electron/Chromium target launched with
--remote-debugging-port(see below)
Native Mac Accessibility mirroring and region overlay are not implemented yet — use CDP for Cursor, VS Code, Chrome, etc.
The window picker lists on-screen apps via CoreGraphics (owner name + size) without Screen Recording permission. For real window titles (e.g. document names), grant Accessibility to Lazy Dev in System Settings -> Privacy & Security -> Accessibility.
Fake Monitor window placement also requires Accessibility — Lazy Dev uses AXPosition / AXSize to move Electron windows to the virtual display and verifies placement with CGWindowList.
- Windows 10/11
- Rust (MSVC toolchain)
- Node.js 20+
- WebView2
cd apps\desktop
npm install
npm run tauri:dev- Open Notepad or Calculator.
- Refresh list -> attach.
- Open Mirror Demo -> click/type.
Cursor (auto-setup): Selecting Cursor in Lazy Dev launches a CDP-enabled instance automatically if port 9222 (or LAZY_DEV_CDP_PORT) is not already open — using an isolated profile (/tmp/lazy-dev-cursor-cdp on macOS). No manual quit/relaunch required for Cursor.
Other Chromium/Electron apps still need a debug port:
# Example: Chrome on port 9222 with an isolated profile
$ud = Join-Path $env:TEMP "lazy-dev-chrome-cdp"
& "C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222 --user-data-dir=$udOptional: force the port with LAZY_DEV_CDP_PORT=9222 before starting Lazy Dev.
Then attach that window in Lazy Dev — status backend should report cdp.
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
--remote-debugging-port=9222 \
--user-data-dir=/tmp/lazy-dev-chrome-cdpcd apps/desktop
npm install
npm run tauri:devOptional: export LAZY_DEV_CDP_PORT=9222 before starting Lazy Dev.
Launch popular apps in Fake Monitor mode to create a real OS virtual display and move the app there. Your phone / Phone Preview mirrors via CDP while the desktop window lives off your main screen.
Uses Apple's private CGVirtualDisplay API (same family as BetterDisplay / DeskPad). No driver install. The virtual display exists while Lazy Dev is running.
- Not Mac App Store compatible (private API).
- API may change in future macOS releases.
- Grant Accessibility to Lazy Dev so window move + verification work (System Events alone is unreliable for Electron apps).
Requires a one-time install of an Indirect Display Driver:
winget install --id=VirtualDrivers.Virtual-Display-Driver -eOr download from VirtualDrivers/Virtual-Display-Driver. Lazy Dev talks to the driver via its named pipe (\\.\pipe\virtualdisplaydriver).
If the driver is missing, Fake Monitor launch fails with an install link in Config.
- Detach Fake Monitor — move the app back to your primary display and remove the virtual monitor
- Reattach Fake Monitor — recreate the virtual display and move the app there again
- Move to... — pick any listed display (primary, external, or virtual)
After you draw a mirror region (CDP apps), Lazy Dev automatically applies live max-height in the source app so it tracks the Phone Preview / phone canvas size. Use Revert in-app sizes to opt out until you draw a new region or click Expand in app again.
cd apps\mobile
npm install
npm run buildThis writes apps/mobile/dist, which the host serves.
cd apps\desktop
npm run tauri:dev- Attach a target window (optional region: Draw region -> drag -> Apply / close overlay).
- Click Start host.
- Note the LAN URL (e.g.
http://192.168.x.x:8787) and 6-character code.
- Open the LAN URL in the browser.
- Enter the pairing code -> Pair.
- Optionally Add to Home Screen (PWA).
- The mirror receives the live UiTree; taps/typing drive the desktop via UIA or CDP.
HTTPS / PWA install: the host currently serves HTTP on the LAN for reliability. Control works over HTTP. Some mobile browsers only allow full "Add to Home Screen" installability on HTTPS — use mkcert or a reverse proxy if you need a trusted cert, or install from Chrome's menu where HTTP LAN is allowed.
cargo test -p schema -p uia -p cdp -p input -p overlay -p server -p vdisplay
cargo run -p uia --example smoke_walk
cd apps\desktop
npm run build
cd apps\mobile
npm run build| Command | Purpose |
|---|---|
list_windows / attach_window / detach_window |
Window picker |
list_popular_apps / launch_popular_app |
CDP launch + Fake Monitor mode |
list_displays / get_fake_monitor_status / set_fake_monitor_attached / move_attached_to_display |
Virtual display management |
get_ui_tree / refresh_ui_tree |
UIA or CDP tree |
invoke_control / set_value / toggle_control / apply_control_event |
Actions |
set_region / clear_region |
Norm-rect scope |
expand_region_in_app / revert_region_in_app / report_phone_view_size |
CDP live expand |
start_region_overlay / stop_region_overlay / poll_overlay_region |
Native overlay (Windows) |
start_host_server / stop_host_server / get_host_status / rotate_pairing_code |
LAN host |
open_mirror_demo / close_mirror_demo |
Local isomorphic demo window |
On macOS, mirroring requires CDP (--remote-debugging-port). Native AX walk and region overlay are not available yet.
UIA only works for apps that expose an accessibility tree (Windows). CDP requires --remote-debugging-port (or equivalent) on Chromium/Electron. Elevated targets may need an elevated host. No pixel streaming / no DLL injection.
MIT