herdr --remote on Windows: remote UI renders but local keyboard and mouse input never reach the session #4038

Activity

1. akbash-bot

   added

   bugSomething isn't workingSomething isn't working

   p1serious defect affecting a common platform, core workflow, or broadly used integrationserious defect affecting a common platform, core workflow, or broadly used integration

   windowsaffects Windows-specific behavioraffects Windows-specific behavior

   remoteaffects SSH, headless, or remote attachment behavioraffects SSH, headless, or remote attachment behavior

   inputkeyboard, mouse, clipboard, selection, or IME behaviorkeyboard, mouse, clipboard, selection, or IME behavior

   maintainer-neededrequires maintainer judgment or maintainer-only reproductionrequires maintainer judgment or maintainer-only reproduction

   on Sep 13, 2026
2. HUAN2022A commented on Sep 13, 2026

   

   HUAN2022A

   on Sep 13, 2026

   More actions

   Static analysis of the Windows --remote input path

   I did a full static trace of the input chain on fc44f61 (input source → client routing → endpoint gating → ssh bridge → server injection). No runtime verification: I have no Rust toolchain or Windows reproduction environment here, so everything below is code reading, including crossterm 0.29.0 sources. The herdr-internal routing checks out symmetric between local and remote Windows clients; the evidence points at the Windows console input buffer being consumed by a competing process, plus one related latent defect.

   1. Symptom recap

   Windows 11 client, herdr --remote <host> to a Linux host: UI renders and streams (SemanticFrame), server sees client connected ... surface_active=true with no disconnect, but no keyboard or mouse input ever takes effect — including client-side UI (tabs, sidebar), so events never reach the client shell at all. Local herdr on the same machine/terminal/binary works.

   2. Chain analysis

   Input source (Windows). The reader thread is spawned once (src/client/mod.rs:462-475). The backend is chosen once at startup (src/client/input.rs:324-339): the VTI console reader (windows_vti::raw_console_reader_loop) is used iff GetConsoleMode reports ENABLE_VIRTUAL_TERMINAL_INPUT, else the crossterm loop. The VTI reader waits on WaitForSingleObject(handle, 10) and then ReadConsoleInputW (src/client/input/windows_vti.rs:184-219). Console input records are destructively consumed from the single shared console input buffer — any other process attached to the same console that reads console input starves herdr completely: keys and mouse dead, view frozen, while stdout rendering keeps working. That matches the reported symptom exactly.

   Terminal setup. setup_terminal_with_capabilities (src/client/terminal_setup.rs:39-111) runs EnableMouseCapture at line 59 before enabling VT input. On Windows, crossterm 0.29 always takes the winapi path for mouse capture because EnableMouseCapture::is_ansi_code_supported() returns false unconditionally (crossterm src/event.rs:321-346; dispatch in src/command.rs:121-129), and enable_mouse_capture() replaces the console input mode with ENABLE_MOUSE_MODE (0x0098) (crossterm src/event/sys/windows.rs:13,36-41), which clears ENABLE_VIRTUAL_TERMINAL_INPUT. Herdr compensates by re-enabling VT input afterwards (terminal_setup.rs:73-76 non-SSH, 51-56 SSH), leaving mode 0x0298. This is identical for local and remote, so setup is not the divergence.

   Local vs remote process topology. A local attach runs the client in-process (src/server/autodetect.rs:295-319 → client::run_client). herdr --remote instead spawns herdr client as a child with inherited console stdio (src/remote/attach.rs:2720-2748) plus the long-lived ssh bridge as a sibling: ssh -F <cfg> -T <host> "exec ... remote-client-bridge" with stdin/stdout piped but stderr inherited into the shared console, and without -n (src/remote/attach.rs:2410-2429). The repo itself documents that Windows OpenSSH reads the console even with stdin redirected to NUL — see the comment and -n workaround for the transient prepare-phase ssh calls at src/remote/attach.rs:619-634 — yet the persistent bridge ssh gets neither -n nor a detached stderr. In the remote setup that ssh is the only extra console-attached reader relative to the working local setup.

   Routing/gating (verified symmetric, not the cause). Windows StdinEvents → shell.handle_client_events (src/client/mod.rs:988-1055) → finish_client_shell_input (src/client/shell_runtime.rs:614-734). The bridge connection registers as endpoint Local, generation 1, surface_active=true (src/client/mod.rs:510-535, src/client/endpoint/registry.rs:101-128), so active_surface_available() is true and no pending_activation exists in a fresh remote attach; the silent drop-gates at shell_runtime.rs:675-679 / 717-723 don't fire. Server-side, ClientShellPaneInput requires is_active_shell_client = shell_surface_active (src/server/headless.rs:2356-2368, src/server/clients.rs:462-464), which is true per your log. Linux/macOS remote clients read stdin bytes from a pipe (src/client/input.rs:73-218), immune to console-record theft — consistent with remote working there.

   3. Root-cause conclusion

   Primary (most probable): the bridge ssh.exe process, attached to the same console via inherited stderr, consumes console input records that herdr's VTI reader (ReadConsoleInputW) needs. Windows console reads are destructive, so a competing reader yields exactly "renders fine, zero input of any kind from the start". The in-repo comment at attach.rs:622 ("Windows OpenSSH can still read the console with stdin redirected to NUL") shows this hazard is already known for the prepare-phase calls; the bridge applies no mitigation. I could not execute a reproduction to confirm this, so treat it as the leading hypothesis rather than a proven fact.

   Secondary (verified latent defect, but not reachable with your defaults): when the server sends MouseCapture { enabled: false }, set_mouse_capture(false) runs crossterm DisableMouseCapture → winapi restores the ORIGINAL mode snapshot captured before VT input ever existed (crossterm src/event/sys/windows.rs:20-30,44-49), permanently clearing ENABLE_VIRTUAL_TERMINAL_INPUT (and mouse bits) while the already-running VTI reader keeps expecting VT sequences. Herdr re-enables VT input only when enabled == true (src/client/mod.rs:1770-1772 SSH branch, 1776-1778 non-SSH branch) — there is no re-enable on the disable path. With default ui.mouse_capture = true the server never sends enabled: false, so this does not explain this report, but it's a real bug reachable via ui.mouse_capture = false or any future disable path.

   4. Suggested fix direction

   • For the primary cause: spawn the bridge ssh detached from the console — pipe (or redirect) stderr instead of inheriting it, and/or add CREATE_NO_WINDOW/DETACHED_PROCESS creation flags to the bridge spawn in src/remote/attach.rs:2410-2429; consider -n as well, though the comment at attach.rs:622 indicates -n alone was not sufficient for the prepare-phase calls.
   • For the secondary defect: after set_mouse_capture(false, ..) on Windows with the VTI backend active, re-enable ENABLE_VIRTUAL_TERMINAL_INPUT (mirror the enabled == true handling in src/client/mod.rs:1768-1779), or make the crossterm winapi mouse-mode restore preserve the VT-input bit.

   5. Cheap discriminating tests

   1. HERDR_WINDOWS_INPUT_TRACE=1 herdr --remote <host>, then type: if no windows input trace: console key record lines appear in herdr-client.log (src/client/input/windows_vti.rs:1341-1343), the reader sees zero records → console-input theft confirmed; if they appear, it's the mapper/mode instead.
   2. HERDR_WINDOWS_INPUT_BACKEND=crossterm herdr --remote <host> (src/client/terminal_setup.rs:228-232): both backends dead → theft; crossterm works → VT-input mode lost.
   3. While attached, probe the client's console input mode (small GetConsoleMode helper) for bit 0x0200 and watch for foreign modes (0x0098).
   4. Reproduce the bridge shape alone: ssh -F <cfg> -T <host> "exec cat" with piped stdin/stdout + inherited stderr alongside any console reader; if the reader starves, the bridge ssh is confirmed as the culprit.
3. JJLiebig commented on Sep 13, 2026

   

   JJLiebig

   on Sep 13, 2026

   Collaborator

   More actions

   @akbash-bot can you check if this would already be tackled in #3687 or #3713, #3714, #3715 our remaining windows server work?

   If no, see if the comment above is enough to start with a preliminary PR that I can verify locally on my Windows machines. Keep it draft and ping me in the PR comment to test + what to send my agent.
4. JJLiebig

   self-assigned this

   on Sep 13, 2026
5. akbash-bot commented on Sep 13, 2026

   

   akbash-bot

   on Sep 13, 2026

   Collaborator

   More actions

   @Pimpmuckl I checked the current diffs. None directly addresses the proposed Windows client-side input failure:

   • feat: install windows remote host packages #3687 installs packages on Windows targets.
   • feat: host remote servers in the signed-in windows desktop #3713 starts servers in the Windows desktop. Its console-detachment fix is for that server bootstrap, not the local SSH bridge.
   • feat: add transferable windows pty foundation #3714 transfers Windows pane PTYs.
   • feat: support live server handoff on windows #3715 adds server handoff. Its broader diff includes the named-pipe polling fix already merged through fix: support preinstalled windows remote hosts #3661, but leaves the local bridge console/input setup unchanged.

   The analysis above is a useful lead, not enough for a draft fix yet: it explicitly has no runtime verification. Redirecting stderr does not itself detach a process from its console, and missing input traces cannot establish another process consumed the events. Also, do not add ssh -n to the bridge: its stdin carries protocol traffic. The mouse-disable observation is separate from the reported default configuration.

   Here is a bounded prompt for your Windows agent:

   Read AGENTS.md. Investigate #4038 without changing production behavior. From native Windows Terminal/PowerShell, use a disposable named Linux/WSL SSH session and test the reported preview client 62431dbd033b, then current master in a fresh session. Compare local attach using the same binary. Record SHAs/versions, typing/click results, and whether independent remote output keeps updating.
   
   If reproduced, trace the selected Windows input backend, console read/wait errors, and input delivery to the server. Temporary diagnostic logging is fine; do not assume SSH consumes console events. Return the smallest sanitized trace showing where input stops, plus exact steps. Preserve existing sessions/config and clean up only disposable resources.
   

   I have not opened a speculative PR. A captured failure on the unmodified path is the missing evidence before a draft fix and Windows verification request.
6. HUAN2022A commented on Sep 13, 2026

   

   HUAN2022A

   on Sep 13, 2026

   More actions

   Thanks @Pimpmuckl. Agreed with @akbash-bot's read — from static analysis on fc44f61 the four PRs look orthogonal to this failure mode. They are all server-side Windows work (remote host package install, hosting servers in the signed-in desktop, transferable PTY foundation, live server handoff), while the suspects in my earlier comment sit in the client input path on the attaching Windows machine:

   • The interactive ssh bridge spawn (src/remote/attach.rs:2415-2425) passes -T but no -n, and uses Stdio::inherit() for stderr in the interactive path — so ssh.exe keeps handles on the shared console and can consume input records that the VTI reader reads destructively via ReadConsoleInputW (src/client/input/windows_vti.rs:201). The code's own comment at src/remote/attach.rs:622 ("Windows OpenSSH can still read the console with stdin redirected to NUL") is why the framed path adds -n; the interactive bridge doesn't.
   • The MouseCapture { enabled: false } path calls into set_mouse_capture(false, ..) (crossterm DisableMouseCapture, src/client/terminal_setup.rs:285-292), and the VTI re-enable at src/client/mod.rs:1770-1778 only runs when enabled is true — so ENABLE_VIRTUAL_TERMINAL_INPUT can stay cleared after a disable.

   The only tangential overlap I can see is the second one: server-side handoff/hosting changes could alter when MouseCapture { enabled: false } messages arrive, but the mode-clearing defect itself lives client-side and triggers on any disable, so it wouldn't be fixed by those PRs either way.

   If a preliminary PR would help, I'm happy to prepare one per your process (draft, with a note on what to test locally) — otherwise deferring to @akbash-bot. Still static-analysis-only on my side; no runtime repro yet.
7. JJLiebig commented on Sep 13, 2026

   

   JJLiebig

   on Sep 13, 2026

   Last edited by JJLiebig

   Collaborator

   More actions

   Reproduced and narrowed on native Windows 11 (26200), Windows OpenSSH 9.5p2, WSL2 Linux target, client running natively on Windows.

   Versions: reported client/server 0.9.0-preview.2026-09-08-62431dbd033b; master client built at 540b20fd.

   Reported build — fails. Server logs client connected client_id=2 ... surface_active=true render_encoding=SemanticFrame; client log stops right after endpoint handshake succeeded. With HERDR_WINDOWS_INPUT_TRACE=1, typing produces 1588 console key record lines and 56 parsed client input events (incl. Enter) — console events do reach the input reader — but the remote pane never receives them. HERDR_WINDOWS_INPUT_BACKEND=crossterm fails the same way. The same binary attached locally accepts input normally.

   This rules out the "bridge ssh steals the console input buffer" theory: a stolen buffer would show zero records, while the reader sees all of them. The loss is on the client→server write path, not the Windows console reader.

   Current master (540b20fd) — passes. With the same setup, typing, a sidebar click (remote workspace focus changed), and independent remote output updates all work.

   The Windows endpoint-write fixes already on master line up with this: c17dc3fd (#3721), d184b41f (#4004), plus the windows_vti work in 7265a2b1/9ad65d90. No source change appears needed.

   Caveat: I ran the client in a native Windows ConPTY console (Herdr pane), not a literal Windows Terminal window, so I reproduced the input failure but not the exact "renders then freezes" frame.

   Closing as fixed on current master but please tag me if you can reproduce yourself on a current master build or if next preview release doesn't fix it.
8. JJLiebig

   closed this as completedon Sep 13, 2026