Conversation
The next commit needs the same per-thread /proc/<pid>/stack dump for the vhost-user backend, which is not a Microvm. Signed-off-by: Riccardo Mancini <mancio@amazon.com>
When a command in the guest never returns, the failure dump covers the host side only: Firecracker's log and its threads, which are simply waiting. Nothing says which guest task is stuck, or where. Ask the guest instead. sysrq-w reports every task in uninterruptible sleep along with the stack it is stuck in. The hung-task detector reports the same stacks, but 120s after the fact, which outlives the command timeout that brings us here. The dump runs from the SSH error hook and now issues SSH of its own, hence the reentrancy guard. Signed-off-by: Riccardo Mancini <mancio@amazon.com>
A vhost-user drive is served by a separate process, so nothing in the failure dump covers the host side of its queues. When the guest waits on a request that never completes, that is exactly what is missing. This has been seen four times in CI without ever establishing where the request was lost. Add the backend's own log and the backtraces of its process tree. crosvm runs each device in a child process, hence the tree rather than the spawned process alone, and it reports a request it failed to execute at DEBUG, hence the level change from off. It logs nothing per successful request, so the volume stays flat under load. Signed-off-by: Riccardo Mancini <mancio@amazon.com>
kill() removed the socket only on the branch where it terminated a live backend, then asserted the socket was gone. A backend that had exited on its own therefore failed teardown with "socket still exists", which says nothing about the exit that caused it. Unlink either way and report the exit code with the log. Signed-off-by: Riccardo Mancini <mancio@amazon.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6231 +/- ##
=======================================
Coverage 83.05% 83.05%
=======================================
Files 277 277
Lines 31481 31481
=======================================
Hits 26147 26147
Misses 5334 5334
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Manciukic
marked this pull request as ready for review
September 18, 2026 15:35
Manciukic
added
the
Status: Awaiting review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Test framework only, in four commits:
Microvm.thread_backtracesintoutils.thread_backtraces(pid).process tree. crosvm runs at
--log-level debugrather thanoff.VhostUserBlkBackend.kill()unlinks the socket even when the backend hadalready exited.
Reason
When a command in the guest does not return, the failure dump covers the host
side only, and only the part of it Firecracker owns: its log, and its threads,
which are parked waiting for something. Nothing says which guest task is stuck
or where, and for a vhost-user drive nothing covers the process actually
serving the queues.
That leaves a specific failure undiagnosable. A vhost-user block test has
timed out four times in CI on a guest waiting for a request that never
completed, and the artifacts have never been enough to say where the request
was lost. The guest sat in WFI, the backend was invisible:
spawn()calledPopenwithout capturing stdout or stderr, and crosvm ran with--log-level off, so the backend left no record at all.The three additions each answer a question the dump could not:
DEBUG, sothat is the lowest level at which the log says anything about a request that
did not complete.
warnproduces zero bytes.thread serving the queue is not in the process we spawned, and backtracing
self.proc.pidalone would miss it.with the stack it is stuck in. The hung-task detector reports the same
stacks, but only 120s after the fact, which outlives the command timeout that
triggers the dump.
Testing
I reproduced the failure mode by sending
SIGSTOPto the crosvm device processduring
mkfs.ext4on a vhost-user drive, which stalls the queue the same way.Before these commits the dump showed Firecracker's vCPUs in
kvm_vcpu_blockand nothing else. After, it shows:
crosvm,blockingPool0and theblock-0childtask:mkfs.ext4 state:D, inio_schedule→wait_on_page_bit_common→blkdev_read_iterTwo things worth flagging about the log level, since
debugon a devicebackend invites the question:
roughly 1.5 GB of mixed direct I/O, because crosvm logs nothing per
successful request. The same backend serves the
vhost-user-blockperformance tests, so a per-request log would have been a problem.
traceadds nothing overdebughere, so there is no reason to go further.Raising the guest's hung-task threshold instead was the obvious alternative and
does not work:
khungtaskdis already asleep on the old timeout when thesysctl changes, so the current sleep is not shortened. With the threshold at
10s the guest sat in
Dstate for 40s with nothing indmesg. sysrq-w returnsthe same information in about a second, so no command timeout needs raising.
Because the dump runs from the SSH error hook and now issues SSH itself, it
takes a reentrancy guard, and a guest that cannot answer is reported as such
rather than replacing the original failure.
tools/devtool checkbuild --allis unchecked below because no Rust changed;the diff is three Python files under
tests/.What this does not cover
The queue state itself. Knowing
avail.idx,used.idx,used_eventandavail_eventwould say directly whether the backend published a completion andwhether a notification was suppressed, but the rings live in guest memory,
crosvm exposes no control-socket command for them, and vhost-user-blk has no
snapshot support, so there is no memory image to read them from. That needs a
change outside the test framework.
License Acceptance
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.PR Checklist
tools/devtool checkbuild --allto verify that the PR passesbuild checks on all supported architectures.
tools/devtool checkstyle --no-clippyto verify that the PRpasses the automated style checks.
how they are solving the problem in a clear and encompassing way.
in the PR.
CHANGELOG.md.Runbook for Firecracker API changes.
integration tests.
TODO.rust-vmm.