Repository navigation
[CI] Windows remote MSB3021: Source array was not long enough copying iOS outputs #26562
Description
Failure signature
Windows Integration remote iOS builds intermittently fail in Xamarin.iOS.Common.After.targets with:
error MSB3021: Unable to copy file "...".
Source array was not long enough. Check the source index, length, and the array's lower bounds.
The failing source file varies, but the stable root is the MSBuild copy operation and identical Source array was not long enough exception.
Confirmed occurrences
| PR / head | Azure occurrence | Evidence |
|---|---|---|
#26495 / 0a76122a3e88077de5c52fc48987562aff5e1aff |
Build 15180799 attempt 1, Windows Integration Dotnet tests, Run .NET tests remotely |
BuildEmbeddedFrameworkInBindingProjectApp(iOS, "iossimulator-arm64") failed while copying Microsoft.iOS.pdb. |
#26554 / 7e8c4edb02eb98aa7ad72f1da2e6755a61af1517 |
Build 15210709 attempt 1, Windows Integration Dotnet tests, Run .NET tests remotely |
BuildIpaAndArchiveOnRemoteWindowsTest(iOS, "ios-arm64") failed while copying R2R/_Microsoft.iOS.TypeMaps.dll and R2R/_MonoTouch.Dialog.TypeMap.dll. |
Count: 2 confirmed occurrences across 2 distinct monitored PRs.
First observed: 2026-09-02T09:07:59Z
Latest observed: 2026-09-04T06:33:00Z
Classification and retry evidence
Genuinely flaky Windows remote MSBuild copy failure.
- The same exception occurred in different PRs, tests, and source files.
- The [bgen] Fix invalid code generation for delegate parameters without [BlockCallback]/[CCallback]. Fixes #10015. #26495 full Windows Integration Tests stage retry succeeded.
- In [main] Update dependencies from dotnet/dotnet #26554, the test failure was recorded before the job reached its 120-minute limit. The timeout subsequently canceled only later artifact publication and is not the root cause.
- Other local and remote Windows tests in the same [main] Update dependencies from dotnet/dotnet #26554 job passed.
Impact
The failure makes the entire simulator validation check red after a lengthy Windows Integration run, even though unrelated package, API-diff, simulator, and AppSize checks pass.
Investigation
- Inspected Azure build metadata, full stage/job/task timeline, remote-test output, failure-gate output, and the published TestSummary for both occurrences.
- Correlated the normalized MSBuild exception, target, and retry outcome.
- Searched all open and closed
dotnet/maciosissues separately by the exact exception,MSB3021, target file, tests, and copied outputs; no matching issue existed.
Workaround
Rerun the entire Windows Integration Tests stage. Do not rerun only the failed job. Azure logs and artifacts are retention-limited.
Activity
dalexsoto commentedon Sep 4, 2026
Full-stage retry started for the new #26554 occurrence:
7e8c4edb02eb98aa7ad72f1da2e6755a61af15172026-09-04T07:42:53Z.windows_integrationstage, as required for Windows Integration failures.MSB3021: Source array was not long enoughsignature matches [bgen] Fix invalid code generation for delegate parameters without [BlockCallback]/[CCallback]. Fixes #10015. #26495, whose full-stage retry succeeded.No equivalent retry was already queued or running, and the PR remained conflict-free at the same head immediately before submission. The retry was submitted through the Azure DevOps CLI, not a PR comment.
Issue-specific checklist: none; the issue body has no Additional info to collect section.
dalexsoto commentedon Sep 4, 2026
The full-stage retry for the #26554 recurrence succeeded:
7e8c4edb02eb98aa7ad72f1da2e6755a61af1517Dotnet testscompleted successfully at2026-09-04T09:50:37Z.windows_integrationstage completed successfully at2026-09-04T09:53:20Z.succeededWithIssuesrecords.This confirms the
MSB3021: Source array was not long enoughremote-copy signature is genuinely flaky. The canonical count remains 2 confirmed occurrences across 2 distinct PRs (#26495 and #26554), and both full-stage retries succeeded.Issue-specific checklist: none; the issue body has no Additional info to collect section.
dalexsoto commentedon Oct 14, 2026
New verified remote-copy occurrence and full Windows-stage retry
#26856, head
95f093d687120d0db81cb53784b6f80004fde5f4, run 15671587 / Windows Integration Tests / Dotnet tests, pipeline 21922, stage/job/task-record attempt 1/1/1. The remote task is7eef1eb5-91f9-5641-dbf4-d80937931e01, log 2767.The raw
windows-remote-dotnet-tests.trxinHtmlReport-windows_integrationwindows-1identifiesXamarin.Tests.WindowsTest.PluralRuntimeIdentifiersWithRemoteMac_Mono(iOS,"iossimulator-arm64;iossimulator-x64","Debug"), 2026-10-13 20:22:52-20:32:53 UTC. It records MSB3021 inXamarin.iOS.Common.After.targets:285copyingnunitlite.dll: "Source array was not long enough", followed by the 600-seconddotnet buildtimeout. The copy exception, not merely the timeout or check title, matches this issue.Classification/counts: genuinely flaky Windows remote MSBuild copy failure. This is 1 newly independently verified occurrence / 1 current monitored PR; together with the 2 previously confirmed canonical occurrences on 2 other PRs retained in this issue, the documented signature now has 3 confirmed case-failure occurrences / 3 distinct PRs. Previous full-stage recovery evidence is retained; no new occurrence is inferred from a retry being queued.
Historical-data availability: Azure DevOps now reports builds 15180799 and 15210709 as not found. Their original diagnostics and retry outcomes cannot be re-downloaded; the two previously confirmed records above are attributed to the retained issue body/comments, not claimed as newly reverified raw build data.
Action: Azure DevOps accepted a retry of the entire
windows_integrationstage withforceRetryAllJobs=true, not only the failed Dotnet-tests job. Stage and all three Windows phases now have attempt 2, including fresh bot reservation and re-enable phases; the outcome is still pending. No equivalent retry was active and the unchanged PR head was conflict-free immediately before submission. A separate simulator failed-job retry for the known iOS signature makes #26856's accepted request counters 2/25 lifetime / 2/10 head; each request counts once regardless of its number of jobs. The APIScan child was verified as an initial automaticPipelineCompletion, not a manual retry. No PR-comment trigger or source edit was used.Issue-specific checklist: the freshly read issue body has no
Additional info to collectrequests. For this occurrence, TestSummary, TRX, full-stage timeline, test identity, timestamps, exception/target/source-file facts, and accepted retry scope were collected. No Helix item applies to this direct Azure Windows job. Machine addresses, worker identifiers, credentials, full build commands, and environment blocks are withheld. Azure diagnostics are retention-limited.Note
This update was generated by GitHub Copilot.
dalexsoto commentedon Oct 14, 2026
Full Windows-stage retry passed the exact original remote-copy case
#26856, unchanged head
95f093d687120d0db81cb53784b6f80004fde5f4, run 15671587 / Windows Integration Tests / Dotnet tests attempt 2, definition 21922. Job9e1d9d57-f452-5f66-496f-9f10db70de3d, remote-test task2c190964-9d1d-5c21-d7b6-f2cb070386be, task-record attempt 1, log 3158.Unmodified
HtmlReport-windows_integrationwindows-2/windows-remote-dotnet-tests.trxrecords the identical originalWindowsTest.PluralRuntimeIdentifiersWithRemoteMac_Mono(iOS,"iossimulator-arm64;iossimulator-x64","Debug")as Passed, 2026-10-14 02:35:01-02:37:17 UTC. All 34 remote TRX cases Passed; Windows TestSummary reports all 3 tasks passed.The entire Windows Integration Tests stage attempt 2 succeeded at 02:51:19 UTC, including fresh reservation, tests and re-enable phases. This is the outcome of the previously accepted full-stage request (
forceRetryAllJobs=true), not a failed-job-only Windows retry. The unrelated iOS job subsequently failed from a different SpringBoard launch signature; that does not invalidate the exact Windows recovery or establish a green PR.Signature/classification:
windows-remote-msb3021-source-array-short, genuinely flaky remote MSBuild copy failure. Counts remain 3 confirmed failures across 3 distinct historically monitored PRs: 1 current independently verified occurrence plus the 2 retained canonical historical occurrences. This passing retry adds no failure and does not erase the originalMSB3021 / Source array was not long enoughrecord. Historical runs 15180799 / 15210709 remain unavailable from Azure; their earlier diagnostics/recoveries are attributed to retained issue history, not newly raw-reverified.Accounting: #26856 now has 3 of 25 lifetime / 3 of 10 head accepted requests: the original full Windows-stage request and two separate simulator failed-job requests. Current iOS attempt 3 is verified running; the successful Windows stage remains at attempt 2 and was not repeated. Dependent publisher attempt 4 is not another accepted retry request. No source fix or permanent transport/copy repair is inferred from a passing stage.
Issue-specific diagnostics: refreshed issue body has no Additional info to collect section; none is outstanding for this outcome. Correct-attempt TestSummary/TRX, exact parameterized case/PASS/timestamps, remote task identity/log and successful full-stage timeline were collected. No Helix item applies to this direct Azure Windows job; artifacts are retention-limited. Worker identifiers, addresses, credentials, complete build commands and environment blocks are withheld. Issue remains OPEN; no source edit, reopening or merge occurred.
Note
This update was generated by GitHub Copilot.
dalexsoto commentedon Oct 14, 2026
New exact remote-copy failure on #26842; full Windows-stage retry accepted
#26842, unchanged head
734c30728ae63354b58735a0a9ddac8494481478, definition 21922, run 15681482, original Windows Dotnet tests job. Stage/job/task 1/1/1, remote-test task7eef1eb5-91f9-5641-dbf4-d80937931e01, log 2730.WindowsTest.RemoteTest(iOS,"ios-arm64","Release")failed 2026-10-14 09:48:27-09:50:21 UTC while copyingTouch.Client.dllatXamarin.iOS.Common.After.targets:285:3: MSB3021 / Source array was not long enough. This is the same copy-operation/exception root, despite a different test and source file. The actual build exited 1; no 600-second timeout is claimed for this occurrence.Primary
HtmlReport-windows_integrationwindows-1/windows-remote-dotnet-tests.trxcontains 28 Passed / 1 Failed remote cases; local TRX has 68 Passed, bgen 419 Passed / 1 NotExecuted. TestSummary reports 1 failed / 2 passed task groups. The remote-result member SHA-256 is0c53837458360b8be02ea6e99f40b2ae784f1ebb3b04d9352e0d967954ada7d9. Failure-gate task67289194-b400-5a1f-9ba1-41bc8c2f8f90, log 2733, failed 09:53:39 UTC; the complete Windows stage ended failed at 09:57:53 UTC, after successful reservation and re-enable phases.Signature/classification/counts:
windows-remote-msb3021-source-array-short, genuinely flaky remote MSBuild copy failure. This monitor now has 2 independently raw-verified failures across 2 distinct PRs (#26856 and #26842), plus the 2 retained canonical historical occurrences on #26495/#26554: 4 recorded failures across 4 historically monitored PRs. Older runs 15180799 / 15210709 remain unavailable and are not claimed newly raw-reverified. Passing retries do not erase failures.Action: after fresh exact-head/merge/scope gates, complete all-actor history reconciliation and atomic reservation, Azure accepted one entire Windows Integration Tests stage retry,
forceRetryAllJobs=true. Stage attempt 2 is independently verified running, with a fresh reservation phase; all reservation/test/re-enable phases are requested, not a failed-job-only retry. Unrelated simulator/macOS stages remain attempt 1. No recovery is claimed.Accepted counters are 3 of 25 lifetime / 1 of 10 on this head; the two earlier macOS requests on the prior head remain counted. Six manual-reason APIScan runs are verified automatic
PipelineCompletionchildren, not retry requests. An empty stage-PATCH response was reconciled through the actual created attempt, never blindly resubmitted.Issue-specific diagnostics: refreshed issue body has no Additional info to collect section; none is outstanding for this occurrence. Exact identities, primary summary/TRX/copy target/file/error/times, failure gate, cleanup and accepted full-stage scope were collected. The exclusive copy/transport concurrency mechanism and decoded binlog details remain unestablished; no global binlog decoder is available. Direct Azure jobs have no Helix item; artifacts are retention-limited. Worker identities, addresses, credentials, full build commands and environment blocks are withheld. No source edit, merge or reopening occurred.
Note
This update was generated by GitHub Copilot.
dalexsoto commentedon Oct 14, 2026
#26842's original case recovered; new exact copy failure on #26866
Signature/classification:
windows-remote-msb3021-source-array-short, genuinely flaky remote MSBuild copy failure. There are now 3 independently raw-verified monitor failures across 3 distinct PRs (#26856/#26842/#26866), plus 2 retained canonical historical records on #26495/#26554: 5 recorded failures across 5 historically monitored PRs. Historical runs 15180799/15210709 remain unavailable, not newly raw-reverified.Actual #26842 recovery: unchanged head
734c30728ae63354b58735a0a9ddac8494481478, definition 21922, run 15681482, Windows tests attempt 2. Stage/job/task-record 2/2/1, remote task2c190964-9d1d-5c21-d7b6-f2cb070386be, log 3051. Correct-attemptwindows-remote-dotnet-tests.trxcontains the identical originalRemoteTest(iOS,"ios-arm64","Release")as Passed at 2026-10-14 11:11:32-11:15:35 UTC; all 29 remote cases Passed. The complete Windows stage, including fresh reservation/tests/re-enable attempt 2, succeeded 11:24:20 UTC. Its parent nevertheless failed 11:28:32 from unrelated simulator credential stalls and an unresolved macOS app timeout; no whole-PR recovery is claimed. Budget remains 3 lifetime / 1 on this head.New #26866 occurrence: exact head
1a7f3813e7dd985be9380bbd1fd6bdb2d910d17e, definition 21922, run 15681906, original Windows tests job. Stage/job/task 1/1/1, remote task7eef1eb5-91f9-5641-dbf4-d80937931e01, log 2783.BundleStructureWithRemoteMac_CoreCLR(iOS,"ios-arm64",All,"Debug")failed 11:05:15-11:08:08 UTC, copying_Microsoft.iOS.TypeMaps.dllatXamarin.iOS.Common.After.targets:310:3, with exact MSB3021 / Source array was not long enough. Source file/line vary, but the copy-operation exception root matches. Correct original remote TRX has 34 Passed / 1 Failed; no 600-second timeout is claimed. Windows stage 1 failed 11:38:56 UTC after cleanup succeeded.Action: after exact-head/conflict/scope gates, complete all-actor histories on both initial head cohorts and atomic reservation, Azure accepted one whole Windows-stage retry,
forceRetryAllJobs=true, verified as stage attempt 2 running. All reservation/test/re-enable phases are requested; unrelated stages remain attempt 1. An earlier attempt was conclusively rejected while the previous publisher was running, with no stage 2 created and no budget charge; the accepted request followed a fresh gate after publisher completion. #26866's counters are 1 of 25 lifetime / 1 of 10 head. No retry result is invented.Issue-specific diagnostics: refreshed issue body has no Additional info to collect section for the failure or outcome. Exact identities, correct-attempt primary summaries/TRX, normalized target/file/error, original-case PASS/times, cleanup/stage results and accepted full scope were collected. Exclusive copy/transport concurrency causes and decoded binlog details remain unavailable/unestablished; no global decoder is available. Direct Azure jobs have no Helix item; artifacts are retention-limited. Worker identities, addresses, credentials, full commands and environment blocks are withheld. No source edit, merge or permanent copy repair is claimed.
Note
This update was generated by GitHub Copilot.
dalexsoto commentedon Oct 14, 2026
#26866's exact original copy-failing case passed on the full Windows retry
This is an outcome of the previously recorded
windows-remote-msb3021-source-array-shortfailure, not another failure. Counts remain 3 raw-verified monitor failures across 3 PRs, plus 2 retained historical canonical records: 5 failures across 5 historically monitored PRs. Older unavailable runs are not newly raw-reverified.#26866, unchanged head
1a7f3813e7dd985be9380bbd1fd6bdb2d910d17e, definition 21922, run 15681906, Windows test job attempt 2. Stage/job/task-record 2/2/1, remote task2c190964-9d1d-5c21-d7b6-f2cb070386be, log 3095.Correct-attempt
HtmlReport-windows_integrationwindows-2/windows-remote-dotnet-tests.trxrecords the identical originalBundleStructureWithRemoteMac_CoreCLR(iOS,"ios-arm64",All,"Debug")as Passed at 2026-10-14 14:14:51-14:18:44 UTC; all 35 remote cases Passed. Member SHA-256:1d3af6b1161cec13eb4b3d63bacb6d0db7ad92cea0c30c9f2625dfe87c5a7ed2. TestSummary reports all three task groups passed.The entire Windows stage attempt 2 succeeded at 14:49:12 UTC, including fresh reservation, tests and re-enable phases. The parent still failed 14:53:13 from the separate original iOS credential stall under #26221; this is not whole-PR recovery.
After fresh exact-head/conflict/scope gates, full all-actor accounting and atomic reservation, the sole failed iOS job received a separate ordinary failed-job request, creating simulator stage attempt 2 queued. Successful Windows attempt 2 was not repeated. Accepted counters are now 2 of 25 lifetime / 2 of 10 head: one full Windows request and one iOS request. The earlier conclusively rejected publisher-blocked request counts zero; success does not reset counters. No iOS retry outcome or permanent copy repair is claimed.
Issue-specific diagnostics: refreshed issue body has no Additional info to collect section. Exact original/retry identities, raw case PASS/time/hash, complete remote counts, task/job/full-stage results and separate failed-parent cause were collected. Original exclusive copy/transport concurrency cause remains unestablished; decoded binlogs are unavailable without a global decoder. Direct Azure jobs have no Helix item; artifacts are retention-limited. Worker identifiers, addresses, credentials, full build commands and environment blocks are withheld. No source edit, merge or reopening occurred.
Note
This update was generated by GitHub Copilot.