Skip to content

[BUG] Stream import files in interpreters to reduce memory usage - #686

Open
Jonathon-Meney-Torq wants to merge 3 commits into
pimcore:2026.2from
TorqIT:feat/streaming-interpreters
Open

[BUG] Stream import files in interpreters to reduce memory usage#686
Jonathon-Meney-Torq wants to merge 3 commits into
pimcore:2026.2from
TorqIT:feat/streaming-interpreters

Conversation

@Jonathon-Meney-Torq

Copy link
Copy Markdown
Contributor

Closes pimcore/platform-version#425

Large import files could exhaust PHP memory during queue preparation. Interpreters now stream instead of loading whole files:

  • JSON/SQL: streamed via json-machine when the JMESPath is a simple field path (full-load fallback for complex expressions)
  • XML: streamed via XMLReader for simple absolute element paths, XSD validated incrementally (full-DOM fallback for complex XPath)
  • XLSX: chunked reads (1000 rows) instead of loading the whole workbook
  • Archive-import-file option now streams into application_log storage instead of file_get_contents

Adds halaxa/json-machine ^1.2. Unit tests included for all streaming paths. 53MB JSON verified at 16.7MB peak memory.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Introduces memory-conscious streaming and chunked processing for large import files.

Changes:

  • Streams JSON/SQL and simple-path XML records.
  • Processes XLSX files in chunks.
  • Streams archived imports into application-log storage and adds tests/dependency support.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
composer.json Adds JSON Machine dependency.
src/DataSource/Interpreter/AbstractInterpreter.php Streams archived import files to storage.
src/DataSource/Interpreter/ChunkedRowsReadFilter.php Filters spreadsheet rows by chunk.
src/DataSource/Interpreter/JsonFileInterpreter.php Adds JSON streaming and validation.
src/DataSource/Interpreter/XlsxFileInterpreter.php Adds chunked XLSX importing.
src/DataSource/Interpreter/XmlFileInterpreter.php Adds XMLReader-based streaming.
tests/unit/JsonStreamingInterpreterTest.php Tests JSON streaming behavior.
tests/unit/XlsxChunkedReadTest.php Tests XLSX chunk filtering and reading.
tests/unit/XmlStreamingInterpreterTest.php Tests XML streaming and validation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/DataSource/Interpreter/XlsxFileInterpreter.php Outdated
Comment thread src/DataSource/Interpreter/XlsxFileInterpreter.php Outdated
Comment thread src/DataSource/Interpreter/JsonFileInterpreter.php Outdated
@Jonathon-Meney-Torq
Jonathon-Meney-Torq marked this pull request as ready for review August 28, 2026 17:49
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants