Skip to content

fix(vsock): preserve fragmented stream uploads - #138

Open
sazlin wants to merge 7 commits into
superradcompany:krunfrom
sazlin:fix/vsock-fragmented-tx
Open

sazlin wants to merge 7 commits into
superradcompany:krunfrom
sazlin:fix/vsock-fragmented-tx

Conversation

@sazlin

@sazlin sazlin commented Sep 15, 2026 •

Copy link
Copy Markdown

Context

I am employing microsandbox to sandbox an OMP agent with an OpenAI Codex subscription. I don't want the subscription credentials visible inside the vm, so I am using a vsock channel to broker/proxy OMP --> OpenAI communication through the host. So: OMP (sandboxed guest) --vsock--> Broker/Proxy (host) --internet--> OpenAI. This patch fixes libkrun’s vsock handling so complete OMP requests can reach the host-side broker without being truncated or stalled.

Problem

Large guest-to-host stream writes can arrive as a virtqueue descriptor chain. VsockPacket::from_tx_virtq_head currently requires the entire declared payload in the first descriptor, logs BufDescTooSmall, and drops the remaining HTTP request. Separately, the custom stream advertises an 8 MiB receive window but returns credit only after 4 MiB; ordinary guest socket transmit buffers can stall before reaching that threshold.

This was reproduced through microsandbox 0.6.17 and 0.6.18 with an HTTP POST: 32 KiB succeeds, 65,535 bytes triggers BufDescTooSmall, and larger uploads stall.

Change

  • Preserve the zero-copy path for a single payload descriptor.
  • Gather fragmented TX payloads into a bounded (64 KiB maximum) allocation.
  • Reject writable or truncated descriptor chains.
  • Use a 64 KiB custom-stream receive window, returning credit after 32 KiB.

Verification

  • New regressions fail against the previous parser/credit window and pass with this change.
  • 11 scoped virtio::vsock tests pass.
  • A patched microsandbox CLI delivered byte-identical HTTP requests of 32 KiB, 65,535 B, 64 KiB, 256 KiB, 1 MiB, and 16 MiB both directly and through a guest TCP-to-vsock relay.
  • Chunked 1 MiB and four concurrent 1 MiB relay requests pass.

RetriggerConfidence Score: 5/5

The PR appears safe to merge based on the changes since the previous review.

Reviews (6) · Last reviewed commit: "fix(vsock): retry credit safely after RX..."

Large guest HTTP requests were truncated because TX payloads spanning virtqueue descriptors were rejected, and credit updates arrived only after ordinary guest send buffers stalled.
@sazlin

sazlin commented Sep 15, 2026

Copy link
Copy Markdown
Author

Cloud Hypervisor had the same issue: cloud-hypervisor/cloud-hypervisor#7672

@sazlin sazlin closed this Sep 15, 2026
@sazlin sazlin reopened this Sep 15, 2026
@toksdotdev

Copy link
Copy Markdown
Member

hey @sazlin, thanks for flagging and for the fix.

could we use a PacketBuffer enum with Borrowed { ptr, len } and Owned(Vec<u8>) variants, similar to Cloud Hypervisor’s fix? that would replace buf, buf_size, and owned_buf with one field, avoiding a separate pointer and length for the owned buffer. behavior would stay the same, but ownership would be clearer.

@toksdotdev

Copy link
Copy Markdown
Member

could we also add a CI step for the vsock tests? the default cargo test doesn't select the devices crate, and the existing devices step filters for virtio::net, so these new regressions won't run.

cargo test --locked --manifest-path src/devices/Cargo.toml virtio::vsock

Represent borrowed guest memory and gathered TX data with PacketBuffer variants, preserving the existing payload behavior. Cover receive writes for both descriptor layouts and run the vsock tests in CI.

@toksdotdev toksdotdev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the fix. made a tiny adjustment to use concrete enum types. asides that, lgtm!

Comment thread src/devices/src/virtio/vsock/packet.rs
Comment thread src/devices/src/virtio/vsock/mod.rs Outdated
Reject truncated TSI request payloads and socket addresses before parsing. Preserve the 8 MiB window for Unix and TSI streams while carrying the custom stream 64 KiB window through queued control packets. Add malformed request regressions and verify advertised custom stream credit.
Decouple the 32 KiB credit update threshold from the 8 MiB receive window for Unix and TSI streams. Waiting for 4 MiB prevents 1 MiB guest uploads from completing.

Extend existing VM integration tests with 1 MiB uploads and keep the TCP test connection alive until VM exit to avoid racing its final response with host close.
Comment thread src/devices/src/virtio/vsock/unix.rs Outdated
Return forwarding credit only after the Unix socket accepts queued bytes, including writable-event flushes. Reject guest writes that exceed the advertised 8 MiB window and copy only the declared packet payload.

Add real-socket regressions for blocked peers, credit recovery after draining, and guests that exceed their window. Both tests fail before the fix and pass on macOS and Linux.
Comment thread src/devices/src/virtio/vsock/unix.rs Outdated
Keep upload credit and writable polling active while downloads wait for guest receive space. Disable read polling independently, and stop writable polling once pending bytes drain.

Extend the real socket backpressure regression to both connected and receive-credit-waiting states; it fails before the fix and passes on macOS and Linux.
Keep custom uploads progressing while downloads wait for guest credit. Retain undelivered credit updates and requests, and retry when RX capacity is freed without requiring socket readiness or extra backend notifications.

Share control delivery across Unix and Windows, clear stale control headers, return invalid RX descriptors, and restrict TSI sends to declared payloads. Add five regressions that fail on the prior commit and pass after repair.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants