Repository navigation
Bound PDF stream decoding and content parsing - #3
Merged
Merged
Conversation
The XMP reader, the unreferenced-object check and the content-stream scan decoded streams in full before looking at their size. A FlateDecode stream of about 1 MB that expands 1024:1 made a single analysis peak at about 3.3 GB of memory, and the 200 MB input cap allows far more. - pdfutil.read_stream() decodes unfiltered and Flate-only streams (including nested Flate) incrementally and stops at a byte limit. Other filter chains are handed to qpdf only when their worst-case expansion fits; otherwise StreamTooLarge is raised. pdfutil.stream_fits() answers the same question without keeping the decoded data. - XMP is read through the bounded reader (2 MB, as before, but checked before decoding). Unreferenced streams are sampled up to 256 KB, as before, without decoding the rest. Page and form content streams are size-checked (64 MB) before pikepdf parses them. - Each case that hits a limit is reported as analysis.stream-too-large (LOW) in its own category, so the summary no longer rules that area out. On the 1 GiB probe files peak memory drops from about 3.3 GB to under 100 MB for XMP and unreferenced streams, and to about 210 MB for a page content stream, where pypdf's own 75 MB per-stream cap still applies to text extraction. Results for the bank-statement examples are unchanged.
…filters working Follow-up to the bounded-decoding change, after review: - Content streams are scanned with a pikepdf StreamParser that hands each instruction to a callback. parse_content_stream built Python objects for every token, about a hundred times the decoded size, so a stream under the size limit could still need gigabytes. - The decoded-size limit is charged per page across all content streams and form XObjects (64 MB) and per document (256 MB), with sizes memoised per object. A /Contents array repeating one stream, or a form shared by many pages, can no longer multiply the work. - Separate the sample size from the decoding budget. Filter chains other than Flate (LZW, ASCIIHex + Flate, predictors) are decoded by qpdf whenever their worst case fits a 256 MB ceiling, and the real size is then checked. Legacy LZW content and hex-wrapped streams are analysed as before instead of being reported as too large. - Corrupt Flate data keeps the output decoded before the error, as qpdf does, and an unreadable zlib header raises like qpdf. - Filter chains longer than eight entries are not decoded. - Limit findings get their own ids (metadata.xmp-too-large, structure.stream-too-large, content.stream-too-large) and the summary lists the limit under limitations. - An operator token that is not valid UTF-8 no longer aborts the scan of the rest of the page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Several analysers decoded PDF streams in full before looking at their size. A file of about 1 MB with a FlateDecode stream that expands to 1 GiB made a single analysis peak at about 3.3 GB of memory, and the 200 MB input cap allows much more. Content streams had a second problem:
pikepdf.parse_content_streambuilds a Python object for every token, which costs roughly a hundred times the decoded size.This PR bounds stream decoding in the metadata, structure and content analysers and reports every case that hits a limit.
Changes
pdfutil.read_stream()andpdfutil.decoded_size(): unfiltered and Flate-only streams (including nested Flate) are decoded incrementally and never beyond the caller's limit. Corrupt Flate data keeps what was decoded before the error, as qpdf does. Other filter chains (LZW, ASCIIHex, ASCII85, Flate with predictors) are decoded by qpdf when their worst-case expansion fits a 256 MB ceiling, and their real size is then checked. Chains of more than eight filters are not decoded.pikepdf.StreamParsercallback instead ofparse_content_stream, so no instruction list is built. Decoded content is charged against a budget of 64 MB per page (all content streams and forms together) and 256 MB per document, with sizes memoised per object. A damaged operator no longer stops the scan of the rest of the page.metadata.xmp-too-large,structure.stream-too-large,content.stream-too-large. The summary then no longer lists that area as checked, and names the limit under limitations.Measurements
Peak memory for
analyze_file, before and after:/ContentsThe remaining 212 MB for the page-content case comes from pypdf's text extraction in the revisions analyser, which has its own 75 MB per-stream cap. That path, and qpdf's decoding of object streams when a file is opened, are not changed here.
Testing
pytest: all tests pass, including new tests for bounded Flate and nested Flate, qpdf parity on corrupt Flate, legacy LZW and ASCIIHex + Flate streams, repeated content streams, the three kinds of oversized stream, and parsing without an instruction list.