Skip to content

fix(sync): send at most one page of new messages when the client knows none - #13823

Open
dillardblom wants to merge 1 commit into
nextcloud:mainfrom
dillardblom:fix/sync-without-known-ids
Open

dillardblom wants to merge 1 commit into
nextcloud:mainfrom
dillardblom:fix/sync-without-known-ids

Conversation

@dillardblom

Copy link
Copy Markdown

Summary

POST /api/mailboxes/{id}/sync with an empty ids list treated every message of the mailbox as new. getDatabaseSyncChanges() called findAllIds(), applied the filter without a limit, and passed every result through the PreviewEnhancer in that single request.

The client sends an empty list whenever it shows none of the messages of a list. In the priority inbox this happens all the time: the client syncs every account's INBOX once per section (is:pi-important, is:pi-other), and an account whose messages are all pushed off the combined page by other accounts has no known ids.

On an INBOX with 33,000 messages (33,000 of them never structure-analyzed), every one of these sync requests ran out of PHP memory (1 GB) in the MIME parser and charset converter. The client then retried it on every sync round, so the log filled with Allowed memory size … exhausted errors for as long as the priority inbox was open. The same mailbox syncs fine from occ mail:account:sync (peak 79 MB), because that path does not run the database diff.

Without known ids, the sync now returns at most one page (20, the frontend's PAGE_SIZE) of new messages in the requested sort order, like the first page the client would load. Requests with known ids are unchanged.

While here: the sort order was compared with 'oldest', but MailboxesController::sync() passes IMailSearch::ORDER_* ('ASC'/'DESC'), so the comparison never matched. It now compares with IMailSearch::ORDER_OLDEST_FIRST. That matters for the limited query: with "oldest first", the page must be the oldest messages.

Related to #13778: this removes one way a sync request can run out of memory. The locking and retry behaviour described there is unchanged.

Test plan

  • Unit test: a sync without known ids queries one page (findIdsByQuery with limit 20) and never calls findAllIds/findNewIds. It fails without the change.
  • tests/Unit/Service/Sync green, php-cs-fixer clean on the changed files.
  • Running on a production instance (NC 34, PostgreSQL, 5.12.3 with this patch): the priority inbox loads, the important and other sections show the expected messages, and the out-of-memory errors stopped.
  • CI

…s none

When the client syncs a list of which it shows no messages, it sends an
empty list of known ids. The sync then treated every message of the
mailbox as new and passed all of them through the preview enhancer in
one request. This happens in the priority inbox: the client syncs each
account's INBOX per section, and an account whose messages are all
pushed off the combined page by other accounts sends no ids. With an
INBOX of 33,000 messages, 33,000 of them not structure-analyzed yet,
every sync request ran out of memory (1 GB) in the MIME parser, and
the client retried on every sync round.

Without known ids, return the first page (20, the page size of the
frontend) in the requested sort order, like the first page the client
would load. Also compare the sort order with IMailSearch::ORDER_OLDEST_FIRST:
the controller passes 'ASC'/'DESC', so the comparison with 'oldest'
never matched.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Dillard Blom <dillard.blom@ensembia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant