Skip to content

Files Are Uploaded Multiple Times by 'rclone sync' #277

Description

@sourceturner

After observing that "rclone sync" uploaded my files multiple times (and as a consequence needed ages to finish), I created the following minimal example to reproduce the issue:

The local folder "internxt-test" contains only one file named "testfile", with just a size of 12 bytes.

After enabling webdav, I ran rclone sync --no-update-dir-modtime --no-update-modtime ~/tmp/internxt-test/ internxt:internxt-test, which produced the following output:

2025/06/07 20:14:11 ERROR : testfile: Failed to copy: object not found
2025/06/07 20:14:11 ERROR : webdav root 'internxt-test': not deleting files as there were IO errors
2025/06/07 20:14:11 ERROR : webdav root 'internxt-test': not deleting directories as there were IO errors
2025/06/07 20:14:11 ERROR : Attempt 1/3 failed with 1 errors and: object not found
2025/06/07 20:14:13 ERROR : testfile: Failed to copy: object not found
2025/06/07 20:14:13 ERROR : webdav root 'internxt-test': not deleting files as there were IO errors
2025/06/07 20:14:13 ERROR : webdav root 'internxt-test': not deleting directories as there were IO errors
2025/06/07 20:14:13 ERROR : Attempt 2/3 failed with 1 errors and: object not found
2025/06/07 20:14:16 ERROR : testfile: Failed to copy: object not found
2025/06/07 20:14:16 ERROR : webdav root 'internxt-test': not deleting files as there were IO errors
2025/06/07 20:14:16 ERROR : webdav root 'internxt-test': not deleting directories as there were IO errors
2025/06/07 20:14:16 ERROR : Attempt 3/3 failed with 1 errors and: object not found
2025/06/07 20:14:16 NOTICE: Failed to sync: object not found

Afterwards, the web interface on https://drive.internxt.com showed me that the file was present in the folder (the folder had already been created previously). Additionally, the file had been added twice to the Trash.

The file ~/.internxt-cli/logs/internxt-webdav-combined.log contained the following log lines:

{"level":"info","message":"WebDav request received\nMethod: PROPFIND\nURL: /internxt-test\nBody: undefined\nHeaders: {\"host\":\"webdav.local.internxt.com:3005\",\"user-agent\":\"rclone/v1.69.3\",\"authorization\":\"Basic <deleted>\",\"depth\":\"1\",\"referer\":\"https://webdav.local.internxt.com:3005/\",\"accept-encoding\":\"gzip\"}","service":"internxt-webdav","timestamp":"2025-06-07T18:14:10.203Z"}
{"level":"info","message":"[PROPFIND] Request received for file at /internxt-test","service":"internxt-webdav","timestamp":"2025-06-07T18:14:10.203Z"}
{"level":"info","message":"WebDav request received\nMethod: PROPFIND\nURL: /internxt-test/\nBody: undefined\nHeaders: {\"host\":\"webdav.local.internxt.com:3005\",\"user-agent\":\"rclone/v1.69.3\",\"authorization\":\"Basic <deleted>\",\"depth\":\"1\",\"referer\":\"https://webdav.local.internxt.com:3005/\",\"accept-encoding\":\"gzip\"}","service":"internxt-webdav","timestamp":"2025-06-07T18:14:10.384Z"}
{"level":"info","message":"[PROPFIND] Request received for folder at /internxt-test/","service":"internxt-webdav","timestamp":"2025-06-07T18:14:10.384Z"}
{"level":"info","message":"WebDav request received\nMethod: MKCOL\nURL: /internxt-test/\nBody: undefined\nHeaders: {\"host\":\"webdav.local.internxt.com:3005\",\"user-agent\":\"rclone/v1.69.3\",\"authorization\":\"Basic <deleted>\",\"referer\":\"https://webdav.local.internxt.com:3005/\",\"accept-encoding\":\"gzip\"}","service":"internxt-webdav","timestamp":"2025-06-07T18:14:10.701Z"}
{"level":"info","message":"[MKCOL] Request received for folder at /internxt-test/","service":"internxt-webdav","timestamp":"2025-06-07T18:14:10.701Z"}
{"level":"info","message":"[MKCOL] ❌ Folder '/internxt-test/' already exists","service":"internxt-webdav","timestamp":"2025-06-07T18:14:10.879Z"}
{"level":"error","message":"[ERROR MIDDLEWARE] [MKCOL - /internxt-test/] Folder already exists","name":"MethodNotAllowed","service":"internxt-webdav","stack":"MethodNotAllowed: Folder already exists\n    at handle (/usr/lib/node_modules/@internxt/cli/dist/webdav/handlers/MKCOL.handler.js:32:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)","statusCode":405,"timestamp":"2025-06-07T18:14:10.879Z"}
{"level":"info","message":"WebDav request received\nMethod: PUT\nURL: /internxt-test/testfile\nBody: undefined\nHeaders: {\"host\":\"webdav.local.internxt.com:3005\",\"user-agent\":\"rclone/v1.69.3\",\"content-length\":\"12\",\"authorization\":\"Basic <deleted>\",\"content-type\":\"application/octet-stream\",\"referer\":\"https://webdav.local.internxt.com:3005/\",\"x-oc-mtime\":\"1749320026\",\"accept-encoding\":\"gzip\"}","service":"internxt-webdav","timestamp":"2025-06-07T18:14:10.881Z"}
{"level":"info","message":"[PUT] Request received for file at /internxt-test/testfile","service":"internxt-webdav","timestamp":"2025-06-07T18:14:10.881Z"}
{"level":"info","message":"[PUT] Uploading 'testfile' to '/internxt-test/'","service":"internxt-webdav","timestamp":"2025-06-07T18:14:10.881Z"}
{"level":"info","message":"[PUT] ✅ File uploaded to network","service":"internxt-webdav","timestamp":"2025-06-07T18:14:11.788Z"}
{"level":"info","message":"[PUT] ✅ File uploaded in 831ms to Internxt Drive","service":"internxt-webdav","timestamp":"2025-06-07T18:14:11.901Z"}
{"level":"info","message":"WebDav request received\nMethod: PROPFIND\nURL: /internxt-test/testfile\nBody: undefined\nHeaders: {\"host\":\"webdav.local.internxt.com:3005\",\"user-agent\":\"rclone/v1.69.3\",\"authorization\":\"Basic <deleted>\",\"depth\":\"1\",\"referer\":\"https://webdav.local.internxt.com:3005/\",\"accept-encoding\":\"gzip\"}","service":"internxt-webdav","timestamp":"2025-06-07T18:14:11.903Z"}
{"level":"info","message":"[PROPFIND] Request received for file at /internxt-test/testfile","service":"internxt-webdav","timestamp":"2025-06-07T18:14:11.903Z"}
{"level":"info","message":"WebDav request received\nMethod: PROPFIND\nURL: /internxt-test/\nBody: undefined\nHeaders: {\"host\":\"webdav.local.internxt.com:3005\",\"user-agent\":\"rclone/v1.69.3\",\"authorization\":\"Basic <deleted>\",\"depth\":\"1\",\"referer\":\"https://webdav.local.internxt.com:3005/\",\"accept-encoding\":\"gzip\"}","service":"internxt-webdav","timestamp":"2025-06-07T18:14:12.003Z"}
{"level":"info","message":"[PROPFIND] Request received for folder at /internxt-test/","service":"internxt-webdav","timestamp":"2025-06-07T18:14:12.003Z"}
{"level":"info","message":"WebDav request received\nMethod: MKCOL\nURL: /internxt-test/\nBody: undefined\nHeaders: {\"host\":\"webdav.local.internxt.com:3005\",\"user-agent\":\"rclone/v1.69.3\",\"authorization\":\"Basic <deleted>\",\"referer\":\"https://webdav.local.internxt.com:3005/\",\"accept-encoding\":\"gzip\"}","service":"internxt-webdav","timestamp":"2025-06-07T18:14:12.528Z"}
{"level":"info","message":"[MKCOL] Request received for folder at /internxt-test/","service":"internxt-webdav","timestamp":"2025-06-07T18:14:12.528Z"}
{"level":"info","message":"[MKCOL] ❌ Folder '/internxt-test/' already exists","service":"internxt-webdav","timestamp":"2025-06-07T18:14:12.703Z"}
{"level":"error","message":"[ERROR MIDDLEWARE] [MKCOL - /internxt-test/] Folder already exists","name":"MethodNotAllowed","service":"internxt-webdav","stack":"MethodNotAllowed: Folder already exists\n    at handle (/usr/lib/node_modules/@internxt/cli/dist/webdav/handlers/MKCOL.handler.js:32:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)","statusCode":405,"timestamp":"2025-06-07T18:14:12.704Z"}
{"level":"info","message":"WebDav request received\nMethod: PUT\nURL: /internxt-test/testfile\nBody: undefined\nHeaders: {\"host\":\"webdav.local.internxt.com:3005\",\"user-agent\":\"rclone/v1.69.3\",\"content-length\":\"12\",\"authorization\":\"Basic <deleted>\",\"content-type\":\"application/octet-stream\",\"referer\":\"https://webdav.local.internxt.com:3005/\",\"x-oc-mtime\":\"1749320026\",\"accept-encoding\":\"gzip\"}","service":"internxt-webdav","timestamp":"2025-06-07T18:14:12.707Z"}
{"level":"info","message":"[PUT] Request received for file at /internxt-test/testfile","service":"internxt-webdav","timestamp":"2025-06-07T18:14:12.707Z"}
{"level":"info","message":"[PUT] Uploading 'testfile' to '/internxt-test/'","service":"internxt-webdav","timestamp":"2025-06-07T18:14:12.707Z"}
{"level":"info","message":"[PUT] File 'testfile' already exists in '/internxt-test', trashing it...","service":"internxt-webdav","timestamp":"2025-06-07T18:14:12.893Z"}
{"level":"info","message":"[PUT] ✅ File uploaded to network","service":"internxt-webdav","timestamp":"2025-06-07T18:14:13.495Z"}
{"level":"info","message":"[PUT] ✅ File uploaded in 617ms to Internxt Drive","service":"internxt-webdav","timestamp":"2025-06-07T18:14:13.612Z"}
{"level":"info","message":"WebDav request received\nMethod: PROPFIND\nURL: /internxt-test/testfile\nBody: undefined\nHeaders: {\"host\":\"webdav.local.internxt.com:3005\",\"user-agent\":\"rclone/v1.69.3\",\"authorization\":\"Basic <deleted>\",\"depth\":\"1\",\"referer\":\"https://webdav.local.internxt.com:3005/\",\"accept-encoding\":\"gzip\"}","service":"internxt-webdav","timestamp":"2025-06-07T18:14:13.613Z"}
{"level":"info","message":"[PROPFIND] Request received for file at /internxt-test/testfile","service":"internxt-webdav","timestamp":"2025-06-07T18:14:13.613Z"}
{"level":"info","message":"WebDav request received\nMethod: PROPFIND\nURL: /internxt-test/\nBody: undefined\nHeaders: {\"host\":\"webdav.local.internxt.com:3005\",\"user-agent\":\"rclone/v1.69.3\",\"authorization\":\"Basic <deleted>\",\"depth\":\"1\",\"referer\":\"https://webdav.local.internxt.com:3005/\",\"accept-encoding\":\"gzip\"}","service":"internxt-webdav","timestamp":"2025-06-07T18:14:13.716Z"}
{"level":"info","message":"[PROPFIND] Request received for folder at /internxt-test/","service":"internxt-webdav","timestamp":"2025-06-07T18:14:13.717Z"}
{"level":"info","message":"WebDav request received\nMethod: MKCOL\nURL: /internxt-test/\nBody: undefined\nHeaders: {\"host\":\"webdav.local.internxt.com:3005\",\"user-agent\":\"rclone/v1.69.3\",\"authorization\":\"Basic <deleted>\",\"referer\":\"https://webdav.local.internxt.com:3005/\",\"accept-encoding\":\"gzip\"}","service":"internxt-webdav","timestamp":"2025-06-07T18:14:14.244Z"}
{"level":"info","message":"[MKCOL] Request received for folder at /internxt-test/","service":"internxt-webdav","timestamp":"2025-06-07T18:14:14.244Z"}
{"level":"info","message":"[MKCOL] ❌ Folder '/internxt-test/' already exists","service":"internxt-webdav","timestamp":"2025-06-07T18:14:14.413Z"}
{"level":"error","message":"[ERROR MIDDLEWARE] [MKCOL - /internxt-test/] Folder already exists","name":"MethodNotAllowed","service":"internxt-webdav","stack":"MethodNotAllowed: Folder already exists\n    at handle (/usr/lib/node_modules/@internxt/cli/dist/webdav/handlers/MKCOL.handler.js:32:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)","statusCode":405,"timestamp":"2025-06-07T18:14:14.413Z"}
{"level":"info","message":"WebDav request received\nMethod: PUT\nURL: /internxt-test/testfile\nBody: undefined\nHeaders: {\"host\":\"webdav.local.internxt.com:3005\",\"user-agent\":\"rclone/v1.69.3\",\"content-length\":\"12\",\"authorization\":\"Basic <deleted>\",\"content-type\":\"application/octet-stream\",\"referer\":\"https://webdav.local.internxt.com:3005/\",\"x-oc-mtime\":\"1749320026\",\"accept-encoding\":\"gzip\"}","service":"internxt-webdav","timestamp":"2025-06-07T18:14:14.414Z"}
{"level":"info","message":"[PUT] Request received for file at /internxt-test/testfile","service":"internxt-webdav","timestamp":"2025-06-07T18:14:14.415Z"}
{"level":"info","message":"[PUT] Uploading 'testfile' to '/internxt-test/'","service":"internxt-webdav","timestamp":"2025-06-07T18:14:14.415Z"}
{"level":"info","message":"[PUT] File 'testfile' already exists in '/internxt-test', trashing it...","service":"internxt-webdav","timestamp":"2025-06-07T18:14:14.594Z"}
{"level":"info","message":"[PUT] ✅ File uploaded to network","service":"internxt-webdav","timestamp":"2025-06-07T18:14:16.367Z"}
{"level":"info","message":"[PUT] ✅ File uploaded in 1778ms to Internxt Drive","service":"internxt-webdav","timestamp":"2025-06-07T18:14:16.475Z"}
{"level":"info","message":"WebDav request received\nMethod: PROPFIND\nURL: /internxt-test/testfile\nBody: undefined\nHeaders: {\"host\":\"webdav.local.internxt.com:3005\",\"user-agent\":\"rclone/v1.69.3\",\"authorization\":\"Basic <deleted>\",\"depth\":\"1\",\"referer\":\"https://webdav.local.internxt.com:3005/\",\"accept-encoding\":\"gzip\"}","service":"internxt-webdav","timestamp":"2025-06-07T18:14:16.478Z"}
{"level":"info","message":"[PROPFIND] Request received for file at /internxt-test/testfile","service":"internxt-webdav","timestamp":"2025-06-07T18:14:16.478Z"}

This makes using rclone/webdav extremely cumbersome, because a sync of the folder that I usually want to backup (a folder of ~ 240 GB) needed more than 4 days (!) the last time that I tried, and the Trash was full of duplicate files afterwards.

Please fix this very soon!

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions