linux: ship DWARF5 symbols as a separate -debug artifact#65
Open
benhillis wants to merge 2 commits into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
This PR updates the Linux test-kernel build/publish pipeline to ship DWARF5 debug symbols (compressed) as a separate -debug release artifact, keeping the default boot kernel artifact small while enabling source-level debugging for crash dump analysis.
Changes:
- Enable DWARF5 debug info (compressed) in 6.1 and 6.18 kernel configs for both x86_64 and aarch64.
- Split
vmlinuxintovmlinux(stripped of.debug_*) andvmlinux.debug(symbols) viaobjcopy, and publish the symbols as a newopenvmm-test-linux-<ver>-debug...artifact. - Extend Docker build stages and the GitHub Actions release packaging to produce and upload the new
-debugtarballs; update documentation to reflect the new layout.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| README.md | Documents the new output directory layout and new -debug release artifacts. |
| pkg/linux/README.md | Explains symbol splitting, .gnu_debuglink behavior, and compression/toolchain constraints; updates “add kernel version” instructions. |
| pkg/linux/build.sh | Implements vmlinux/vmlinux.debug splitting and outputs boot vs debug payloads into separate sysroot directories. |
| pkg/linux/6.18/x86_64.config | Enables CONFIG_DEBUG_INFO_DWARF5 and zlib-compressed debug sections for 6.18 x86_64. |
| pkg/linux/6.18/aarch64.config | Enables CONFIG_DEBUG_INFO_DWARF5 and zlib-compressed debug sections for 6.18 aarch64. |
| pkg/linux/6.1/x86_64.config | Enables CONFIG_DEBUG_INFO_DWARF5 and compressed debug info for 6.1 x86_64. |
| pkg/linux/6.1/aarch64.config | Enables CONFIG_DEBUG_INFO_DWARF5 and compressed debug info for 6.1 aarch64. |
| Dockerfile | Adds result-linux-<ver>-debug stages and exports linux-<ver>-debug/ outputs alongside existing kernel outputs. |
| .github/workflows/build.yml | Packages and publishes new openvmm-test-linux-<ver>-debug... tarballs in the release job. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Consumers that don't need symbols don't have to download them. Tools | ||
| like `gdb`, `crash`, and `drgn` will automatically pick up | ||
| `vmlinux.debug` when both files are extracted into the same directory | ||
| (the link is recorded via `objcopy --add-gnu-debuglink` in `build.sh`). |
Member
Author
There was a problem hiding this comment.
Good point, thanks. Added a paragraph right after this one in eb68f63 documenting the sibling-directory gotcha for the local docker build --output layout and suggesting both cp/ln and set debug-file-directory as workarounds.
Enable CONFIG_DEBUG_INFO_DWARF5 (with zlib-compressed debug sections) for both the 6.1 and 6.18 test kernels on x86_64 and aarch64, so crash dumps can be analysed with gdb/crash/drgn against the matching release. To keep the existing kernel tarball small, the build now splits vmlinux into a stripped image and a separate vmlinux.debug symbol file via objcopy --only-keep-debug / --strip-debug / --add-gnu-debuglink. The stripped vmlinux ships in the existing openvmm-test-linux-<ver> tarball (now ~20 MB); the symbol file ships in a new openvmm-test-linux-<ver>-debug.<arch>.<ver>.tar.gz artifact so consumers only download symbols when they need them. Tools that follow .gnu_debuglink (gdb, crash, drgn, ...) auto-load vmlinux.debug when both files are extracted into the same directory; the bzImage/Image bootable images are unaffected. Verified end-to-end by building all four (arch x version) pairs locally and confirming BuildID match between each vmlinux and vmlinux.debug, that .gnu_debuglink points at vmlinux.debug, and that the stripped vmlinux retains kallsyms (only .debug_* sections were removed). Note: the bundled cross toolchain (binutils 2.33.1 / GCC 11.5.0) does not support zstd debug-section compression, so we use zlib; documented in pkg/linux/README.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address PR feedback: `docker build --output` lands vmlinux and vmlinux.debug in sibling directories, so .gnu_debuglink auto-load doesn't fire out of the box. Document the workaround (copy/symlink or set debug-file-directory). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
benhillis
force-pushed
the
linux-debug-symbols
branch
from
eb68f63 to
006cbd3
Compare
Comment on lines
+32
to
+33
| directory explicitly (for `gdb`: `set debug-file-directory | ||
| out/linux-<ver>-debug`). The release tarballs are designed to be |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The 6.1 and 6.18 test kernels currently ship without DWARF debug info, which makes crash dumps hard to debug (only
kallsymssymbol names are available — no source-level info forgdb/crash/drgn).This PR enables
CONFIG_DEBUG_INFO_DWARF5(with zlib-compressed debug sections) for both kernels on both architectures, and splits the debug info into its own release artifact so consumers only download symbols when they actually need them.Layout change
pkg/linux/build.shnow usesobjcopy --only-keep-debug/--strip-debug/--add-gnu-debuglinkto produce:vmlinux— stripped of.debug_*sections, but retains.symtab/kallsyms; ships in the existingopenvmm-test-linux-<ver>.<arch>.<ver>.tar.gztarball.vmlinux.debug— the DWARF5 symbol file (zlib-compressed); ships in a newopenvmm-test-linux-<ver>-debug.<arch>.<ver>.tar.gzartifact.gdb,crash,drgn, etc. auto-loadvmlinux.debugvia the.gnu_debuglinkrecorded insidevmlinuxwhen both files are extracted into the same directory. The bootable image (bzImage/Image) is unaffected.Artifact size impact (final
.tar.gz)openvmm-test-linux-6.1openvmm-test-linux-6.1-debugopenvmm-test-linux-6.18openvmm-test-linux-6.18-debugSo the boot-only kernel artifact actually got smaller than before (was ~125-200 MB with embedded debug info; previously had no debug at all before this stack of work), and people who don't debug never download the symbol file.
Verification
Built all four (arch × version) pairs locally via the
pkg/linux/README.mdbootstrap procedure and confirmed for each:vmlinuxandvmlinux.debug(e.g. 6.1 x86_64: both reportc6e4ac6f...).readelf -p .gnu_debuglink vmlinux→vmlinux.debug.file vmlinux.debug→with debug_info, not stripped.file vmlinux→not stripped(kallsyms preserved; only.debug_*removed).The four kernel configs were regenerated from in-build
olddefconfigoutput per the README, so the committed diffs are minimal and toolchain-correct.Notes
zlib(notzstd) because the bundled cross toolchain (binutils 2.33.1 / GCC 11.5.0) predateszstddebug-section support;CONFIG_DEBUG_INFO_COMPRESSED_ZSTDwould be silently dropped byolddefconfig. Documented inpkg/linux/README.md.pkg/linux/README.mdis updated to include the newresult-linux-<ver>-debugDockerfile stage.