Repository navigation
[CI] BuildSdkVersions races on shared SdkVersions.cs.tmp during multi-target builds #26685
Description
Summary
assembly-preparer's BuildSdkVersions target can run concurrently for its target frameworks, with both invocations generating and comparing the same tools/common/SdkVersions.cs.tmp. The declared MSBuild output path also points to common/SdkVersions.cs, not the actual tools/common/SdkVersions.cs.
Three independently confirmed package-build failures across two monitored PRs show the same overlapping generation and regeneration-check error. Classification: source-related build-system generation/concurrency defect. The evidence supports a shared-output race; the exact transient bytes observed by diff were not captured.
Confirmed occurrences
| PR / head | Run / pipeline | Failure time (UTC) | Failed framework |
|---|---|---|---|
#26669 / 46f4ac0ffb23ed460aa44a59dec699c483418170 |
15414974 / simulator pipeline | 2026-09-22 15:01:19 | net11.0 |
#26669 / 46f4ac0ffb23ed460aa44a59dec699c483418170 |
15414971 / package pipeline | 2026-09-22 15:08:51 | netstandard2.0 |
#26574 / 306b9621faa1e9c5b6da750f2d701ceef2c2d378 |
15425064 / package pipeline | 2026-09-23 02:24:34 | net11.0 |
All three are stage/job/task attempt1, Build packages -> Build. Task log IDs are 98,145,145, respectively. Pipeline definitions are21922 (xamarin-macios-sim-pr-tests) and16533 (xamarin-macios-pr).
Counts: 3 confirmed job/task occurrences across2 distinct PRs. First confirmed failure: 2026-09-22 15:01:19 UTC; latest: 2026-09-23 02:24:34 UTC. The first two records were retained from the preceding monitoring session; they are not counted again as new failures.
Original diagnostic
The file ../../tools/common/SdkVersions.cs has been automatically re-generated; please commit the changes.
make[2]: *** [.../tools/common/SdkVersions.cs] Error 1
assembly-preparer.csproj(387,3): error MSB3073:
The command "make generated-files -C .../tools/assembly-preparer" exited with code 2.
No actual SdkVersions.cs git diff is printed by the failing recipe in these task logs. The message alone is not evidence that SDK-version constants need changing.
Evidence and mechanism
The published macios/msbuild/.build-stamp.binlog in each all-binlogs-<run>-1-1 artifact shows two overlapping Exec calls in BuildSdkVersions for net11.0 and netstandard2.0, both running make generated-files in the same directory:
- 15414974: starts0ms/95.988ms, finishes1641.165ms(success)/1682.791ms(failure).
- 15414971: starts0ms/65.214ms, finishes1469.988ms(success)/1516.860ms(failure).
- 15425064: starts0ms/44.241ms, finishes1529.531ms(success)/1575.443ms(failure).
The losing framework reverses between runs. These are relative intervals, not host-rendered absolute binlog times.
The exact CI merge sources (eb80579eee5afe240317be3dd82f2efda42d3287 and 1ac0a9cb160db018228059387b34da2179dfd850) contain identical relevant rules:
tools/assembly-preparer/assembly-preparer.csproj:384-388:BuildSdkVersionsruns beforeCoreCompile, with output../../common/SdkVersions.cs. From that project directory this resolves to repositorycommon/SdkVersions.cs, while the actual generated file is intools/common. Both inner builds explicitly log that the declared output does not exist.tools/common/Make.common:7-51:sedwrites directly to fixed$@.tmp; a subsequentdiff/copy/git-diff/error sequence uses the same paths, with no cross-process serialization or invocation-specific temporary path.
The PRs change dependencies/expected sizes or XHarness, not these generation rules; this is not attributed to a source edit introduced by either PR.
Impact and direction
Package build aborts. Dependent test stages can then be skipped; those skips are not separate test failures. A successful API-diff pipeline does not clear this independent build gate.
The build-system owner should make shared-file generation single-writer/concurrency-safe, correct the incremental input/output paths, and avoid shared temporary output. Blindly committing regenerated constants, suppressing diagnostics, or repeatedly rerunning CI is not a verified fix.
No monitor retry, PR-source edit, commit or push was performed. No validated fix/recovery is claimed for these three occurrences.
Investigation and limits
Original task logs, published binlogs and exact-head/merge source were compared. The local diagnostic reader skipped nine unknown kind35 events per binlog; the conclusion uses supported project/target/task/error/message events, independently checked against task logs and source. Exact transient intermediate-file bytes and the filesystem interleaving remain unavailable.
Separate fully paginated open/closed searches covered SdkVersions.cs, BuildSdkVersions, automatically re-generated, and assembly-preparer + generated-files. #26052 concerns a different stale linked-source generator and CS0117; #8487 concerns missing xharness prerequisites/CS2001. Neither is this mechanism, and neither points to a matching canonical issue.
Worker identifiers/IPs and raw environment details are withheld. Binlogs remain linked through the Azure artifact pages rather than uploaded unreviewed; artifact retention limits apply.
Stable signature: source-related|assembly-preparer-BuildSdkVersions|multi-TFM-shared-SdkVersions.cs.tmp|generated-file-verification-MSB3073
Activity
dalexsoto commentedon Sep 23, 2026
Verified external package-build recoveries
Both previously failed package pipelines succeeded on external retry2 with their original PR heads and merge-source revisions unchanged:
306b9621faa1e9c5b6da750f2d701ceef2c2d378: run15425064 / Build packages / Build. Job finished 2026-09-23 11:50:52 UTC; pipeline succeeded 11:51:03 UTC.46f4ac0ffb23ed460aa44a59dec699c483418170: run15414971 / Build packages / Build. Job finished 2026-09-23 11:55:06 UTC; pipeline succeeded 11:55:12 UTC.Both are
xamarin-macios-pr(definition16533), stage/job attempt2, task-record attempt1, task log236. Their retry-specificall-binlogs-<run>-2-2artifacts showBuildSdkVersionsexecuting for netstandard2.0 and net11.0, with bothExectasks succeeding and zero MSBuild errors. Each retry published its ownnot-signed-packageartifact.Not a source fix: the incorrect declared output path is still logged. In these successful executions one invocation generated
SdkVersions.cs, and the other reported no work forgenerated-files. This is consistent with the existing timing-sensitive shared-file defect; it does not remove the race or justify committing regenerated constants.Counts/classification remain unchanged: 3 confirmed original failures across2 distinct monitored PRs, classified as source-related. These are two recovery outcomes, not new failure occurrences. PR #26669's separate simulator run15414974 still has its original failure; PR #26574's simulator retry is independent and still being monitored.
Diagnostics/checklist: the refreshed body has no Additional info to collect section. Per-run head, source revision, task/job/stage timing, generator execution and package publication were collected. The offline reader skipped nine unknown kind35 events per binlog; supported events, Azure task outcomes and retry-specific artifacts provide the recovery evidence. Exact original transient file bytes, a permanent fix and complete all-actor retry-request grouping remain unavailable. Worker identifiers/IPs are withheld.
No retry, source edit or branch change was performed by this monitor. Retry history and budget reservations are not reset by success. Artifact retention limits apply.
dalexsoto commentedon Sep 24, 2026
External retry blocked before BuildSdkVersions could be retested
PR #26669, unchanged head
46f4ac0ffb23ed460aa44a59dec699c483418170: simulator run 15414974 / Build packages / Build, stage/job attempt 2, task-record attempt 1, pipeline 21922, task log 231. At 2026-09-24 07:05:15 UTC, NuGet restore failed withNU1301: Azure Artifacts returned HTTP 500 rate-limiting responses (ATCPU) while fetching multiple ordinary packages. The build stage failed at 07:05:59 UTC, and the parent pipeline completed failed at 07:09:14 UTC.This is a distinct CI infrastructure failure in the retry, not another
BuildSdkVersionsrace. The generator did not run to its prior shared-file failure point; the attempt cannot validate a source fix or recovery. Canonical source-race totals remain 3 confirmed failures across 2 distinct monitored PRs. Linked fix PR #26687 is still open/unmerged; this run does not validate it.Retries: one or more external accepted requests are observable from stage attempt 2, but the actor, exact request payload/grouping and all-actor lifetime/head budgets remain uncertified. This monitor requested no retry and made no bot-branch edit. Issue-specific checklist: no Additional info to collect section in the refreshed issue body; no requested diagnostic items. The Azure task recorded 312
NU1301lines, 156 of them explicitly reporting HTTP 500/ATCPU throttling. Exact request grouping is unavailable; worker identifiers and IPs are withheld. Original binlogs and retry task logs are subject to Azure retention limits.dalexsoto commentedon Sep 24, 2026
Linked fix PR #26687: build target executed and packages passed
#26687, head
6df0ea155215603133a01b583c94440eeb32ca07, is the proposed fix for thesource-related|assembly-preparer-BuildSdkVersions|multi-TFM-shared-SdkVersions.cs.tmp|generated-file-verification-MSB3073race. Both Azure PR run 15431337 /Build packagesand simulator run 15431340 /Build packagesjob600d49c6-e96b-5e8b-2c3a-9f351e2e5848succeeded. Simulator run Build task log 99 showsBuildSdkVersionsexecuted andassembly-preparerwas built. The proposed source change uses per-process temporary files and atomic publication for generated sources, with Make tracking their prerequisites.Limit: normal green CI does not prove that the concurrent shared-file race was stress-reproduced and eliminated. This fix PR is still open and unmerged. Its initial Windows and iOS test failures came from separately identified registry-permission, SpringBoard, and NativeAOT crash signatures; the final green simulator rollup followed external retries. #26669's prior package retry stopped at Azure Artifacts rate limiting before its original generation race could be retested.
Source-race totals remain 3 confirmed failures across 2 distinct monitored PRs. #26687 provides a limited fix-head build outcome, not a new source-race failure. No retry or source edit was requested by this monitor. The issue body has no
Additional info to collectsection; there is no Helix work item for package generation, and Azure artifacts are retention-limited.AI-assisted CI monitoring update.
[build] Fix concurrent SDK version generation. Fixes #26685 (#26687)
dalexsoto commentedon Oct 2, 2026
Release API-baseline builds hit the unfixed SDK-generation race
New exact occurrences:
a7673fbfbcdb5330098040a443bb65d5ba8cdd9b: API run 15551292 / Detect API changes task, definition 22268, log 98, initial stage/job/task attempt 1; generation guard failed 2026-10-02T17:01:53.1945240Z.21cba66b4787ece3527d3bdffdc32dbad816eb6a: API run 15551442 / Detect API changes task, definition 22268, log 98, initial stage/job/task attempt 1; generation guard failed 2026-10-02T17:13:54.5347850Z.Both runs target release/11.0.1xx. The shared failed job is
276491e5-a85e-51e3-6c38-98e261218d1a; task30666fd6-431a-5dac-9b99-b9fef4b8eba4.Same source-related mechanism: API detection first rebuilds historical baseline
a75b11fe9cdbd66a6ddb8bfea120d006c1ec1e8dwith SDK 11.0.100-rc.2.26465.108. Each exact task log shows overlappingBuildSdkVersionsgeneration andSdkVersions.cs has been automatically re-generated, followed by MSB3073 / make generated-files exit 2, netstandard2.0. That immutable baseline still uses the shared$@.tmprule; merged fix #26687 /8918122d5be1854189baaf50af43bf5eb0bb69fdis not an ancestor of it. The message is not evidence that version constants should be blindly committed.API baselines: both raw
change-detection-1artifacts contain onlyresults/gh-comment.md, reporting that the historical baseline build failed. Neither a current-PR/commit API diff nor the separately implemented vs-stable comparison was produced. This is not a non-empty beta-Xcode breaking-change result and authorizes no binding or suppression edit.Signature:
source-related|assembly-preparer-BuildSdkVersions|multi-TFM-shared-SdkVersions.cs.tmp|generated-file-verification-MSB3073. 5 confirmed root-task occurrences across 4 distinct monitored PRs (three retained historical occurrences plus these two; unchanged observations are not new failures).Diagnostics/checklist: the refreshed issue body has no Additional info to collect section. Each occurrence retains exact PR/head/run/definition/stage/job/task/attempt/log attribution, failure time, baseline/fix ancestry, overlapping-generation lines and SHA-256 fingerprints of its raw log and nested comparison artifact. Unavailable per occurrence: no identified published baseline primary binlog; exact transient
.tmpbytes were not captured. NUnit failure results and Helix work items do not apply to this API-baseline build. Worker/IP/environment details are omitted; remote artifacts are retention-limited.Keep this issue closed: it was completed by the main-branch fix on 2026-09-24, less than two weeks ago; these failing release baselines do not contain that fix and do not invalidate its closure. Maintainers need the existing generation fix in the comparison baseline/target branch. No verified retry workaround is established.
No monitor retry or branch/source edit. This is a source-related build-system defect, not a proven flaky test or infrastructure failure; all-actor 25-lifetime/10-head accepted-request grouping also remains uncertified.
dalexsoto commentedon Oct 3, 2026
Exact same-head external attempt2 outcome
#26822 /
21cba66b4787ece3527d3bdffdc32dbad816eb6aRun15551442, Detect API changes job2, definition22268, stage/job2/2, succeeded 2026-10-03T13:05:09.8833333Z. Matching task
85bb0e1c-87a3-5d19-ec03-9df30b0ce8b4/ task-record1 / log198 succeeded; parent finished 2026-10-03T13:08:30.926104+00:00 and PR merged 2026-10-03T13:08:59Z. Azure previousAttempts links original failed job276491e5-a85e-51e3-6c38-98e261218d1a.Actual task output rebuilt historical baseline a75b11f with 11.0.100-rc.2.26465.108 and completed both comparisons. Source-GUID-matched change-detection-2 separately verifies current-PR vs that baseline (raw Constants.Version27.0.12212->27.0.12213 changes) and vs stable Ref.net11.0_27.0 /27.0.10722 (separate nonempty textual API differences). Each baseline's four actual APICompat outputs report no breaking changes. macOS independently has unnecessary-suppression notices in each comparison; these diagnostic suggestions were not executed or used to authorize suppression/binding edits. Neither textual comparison is claimed empty or transferred into the other. This bounded task success does not establish that the missing main fix reached the historical baseline or that its generation race is globally fixed.
Signature:
source-related|assembly-preparer-BuildSdkVersions|multi-TFM-shared-SdkVersions.cs.tmp|generated-file-verification-MSB3073. Classification remains source-related. 5 confirmed occurrences / 4 distinct PRs, unchanged: one existing occurrence gains outcome evidence, not a new failure.Issue-specific checklist: freshly read body has no Additional info to collect section. Authorized exact PR/head/run/stage/job/attempt/task or primary artifact/test proof is attributed above. Unavailable: accepted-request actor/grouping, complete all-actor lifetime/head counters, and proof of a source-race remediation remain unestablished. Attempts do not certify the25-lifetime/10-head budgets. No monitor retry, source/suppression edit, issue creation or reopening. Keep CLOSED/COMPLETED (closed 2026-09-24T10:30:41Z): success does not invalidate scoped closure. Direct Azure work has no Helix item; artifacts are retention-limited. Worker identifiers, addresses, credentials and raw environment blocks are withheld.
[release/10.0.1xx] [build] Fix concurrent SDK version generation. Fixes
dalexsoto commentedon Oct 15, 2026
Two more occurrences in unfixed release sources (no reopening)
Two additional root-task failures match this source-related
BuildSdkVersionsshared-output defect:05606f8c9b188400f58174903dadf592eaba40a0netstandard2.0cb041609634b0c10827937cf317dd74a8b4dd19783f2323b8059581a036a9a6405c3072c09a11b83, failingnet11.0Both are definition 22268, initial stage/job/task attempt 1. The original #26769 checkout log 46 verifies that its GitHub merge was undone to raw head
05606f8c9b. #26877 explicitly rebuilds the historical baseline with SDK 11.0.100-rtm.26480.113 before comparing APIs.Same mechanism: both sources retain the incorrect
../../commonMSBuild outputs and identicaltools/common/Make.commonblobd2dddec4ea2236345471993a06e895ae05817426, writing fixed$@.tmp. Their twomake generated-filesExec starts are 14:59:11.500814 / 14:59:11.531853Z and 12:15:22.822780 / 12:15:22.842628Z, respectively. Both then report the regeneration guard and MSB3073 / make exit 2; neither prints an actualSdkVersions.csgit diff. This does not establish that version constants need committing.API scope: neither occurrence produced the current-PR or separate stable comparison. #26877's raw
change-detection-1artifact is a 462-byte ZIP containing onlyresults/gh-comment.md, reporting failure to build baseline83f2323b. This is not a non-empty beta-Xcode breaking-change result and authorizes no binding or suppression edit.Counts: this monitor adds 2 confirmed root-task occurrences across 2 distinct monitored PRs. Together with the reconciled 5 occurrences / 4 PRs already recorded here, the canonical history is 7 occurrences / 6 distinct PRs. Successful retries and the separate missing-ZIP publisher failure are not additional source-race occurrences.
Keep CLOSED / COMPLETED: #26687 merged as
8918122d5be1854189baaf50af43bf5eb0bb69fdon 2026-09-24. GitHub ancestry comparisons and exact source inspection confirm that neither failing release source incorporates that fix. These failures do not invalidate its main-branch closure; the relevant release build/comparison source needs the existing fix. No monitor retry, source/expected-file edit, or reopening was performed.Diagnostics/checklist: the refreshed issue body has no Additional info to collect section. Per-occurrence task/checkout attribution, text execution timestamps, source rules, ancestry and original outcomes were collected. Exact transient temporary-file bytes/filesystem interleaving and locally decoded binary task intervals remain unavailable. NUnit/Helix results do not apply to these build failures. Worker/IP/environment identifiers are withheld; Azure artifacts are retention-limited.
Note
Generated by GitHub Copilot.
dalexsoto commentedon Oct 15, 2026
Two exact external retry recoveries; release/10 backport merged
The two original generation failures in the preceding update have bounded, unchanged-head external retry outcomes:
05606f8c9b188400f58174903dadf592eaba40a0BuildSdkVersionscalls and succeeds; comparison log 198 succeeds. Publisher job 2 also succeeded; the parent finished 2026-10-15 13:11:16 UTC. Checkout log 146 confirms the same raw PR head.cb041609634b0c10827937cf317dd74a8b4dd197CURRENT_HEAD/GH_HEADstill match this head.Both are definition 22268, stage/job 2, task-record 1. Each run's
change-detection-2artifact is source-GUID-matched to its own job 2; the shared job GUID alone was not used as identity.The baselines were checked separately:
05606f8cversus historical c08806f, assembly versions 27.0/27.0Ref.net10.0_26.526.5.10280, assembly 26.5 versus current 27.0cb041609versus historical 83f2323, current net11.0 referencesRef.net10.0_27.027.0.10722, independently compared with current net11.0 referencesAll four actual APICompat outputs in each comparison report no breaking changes. macOS also retains unnecessary-suppression diagnostics in each comparison; their suggestions were not executed. Neither textual comparison is claimed empty merely from a green parent or summary. The separate generator diffs were retained, not applied. This is recovery of the failed build/comparison executions, not proof that the shared-file source defect was eliminated.
Separate fix outcome: #26869, the release/10.0.1xx backport of #26687, merged externally at 12:46:04 UTC as 153ba15, with 52 successful / 3 skipped checks. Exact merged sources verify per-process temporary files, atomic publication and removal of the invalid MSBuild input/output declarations. This is not a race-stress result or evidence that other raw PR heads or immutable comparison baselines incorporate the backport.
Counts remain 7 confirmed original failures / 6 distinct PRs in the reconciled canonical history (2 / 2 in this monitor). Recoveries and a fix merge add no failures. Keep CLOSED / COMPLETED. The monitor performed no reopening, retry, source/expected-file/suppression edit or merge. At least one external accepted request per affected PR remains charged; exact all-actor grouping, actors and submission timestamps are still uncertified, and success resets neither budget.
Diagnostics/checklist: the refreshed body has no Additional info to collect section. Original temporary-file bytes/filesystem interleaving remain unavailable; successful attempts do not fill them in. No NUnit/Helix result applies to these build tasks. Worker/IP/environment identifiers are withheld; Azure artifacts are retention-limited.
Note
Generated by GitHub Copilot.
[release/11.0.1xx] [build] Fix concurrent SDK version generation. Fixes
dalexsoto commentedon Oct 15, 2026
Existing generation fix backported to release/11; no reopening
#26879, exact head 23ba4d0, merged externally into release/11.0.1xx at 2026-10-15 21:46:24 UTC as e4732a6, with 52 successful / 3 skipped checks. This is the release/11 backport of the existing main-branch fix #26687, a separate fix outcome, not another generation failure.
Exact merged-source inspection verifies the mechanism rather than assuming it from the PR title:
tools/common/Make.common, blob 08f3a5c859c85f19a4c33d83531f079f411d648bmvpublication; the original fixed shared temporary redirect is absent.tools/assembly-preparer/assembly-preparer.csproj, blob 8a272706ab17e653159f79fac50fde5b3cd2ddfbBuildSdkVersionsstill runs beforeCoreCompile; the incorrect MSBuildInputs/Outputsdeclarations are removed.Bounded CI: initial packaging 15697696 / definition 16533 succeeded 13:01:06 UTC; initial API comparison 15697700 / 22268 succeeded 13:05:59. Simulator run 15697699 / 21922 reached terminal success 21:45:51, after publisher 3 succeeded 21:45:50. Its two policy-gated retries addressed independently diagnosed tvOS infrastructure and the separate credential-redirect flake, not this source-generation defect. Their accepted counters remain 2 lifetime / 2 head; no original failure or budget was reset.
This establishes static fix incorporation and ordinary successful CI on the backport. It is not a concurrent-generation stress result, nor evidence that an existing raw PR head or immutable historical comparison baseline such as
83f2323bautomatically incorporates the backport. No constants, binding baselines or API suppressions were changed by this monitor.Keep CLOSED / COMPLETED. The original main-branch closure remains valid. Canonical failure counts remain 7 occurrences / 6 distinct PRs in the reconciled history (2 / 2 in this monitor); a fix PR and passing runs add no failures. The monitor performed no reopening, source edit, worktree, commit, push or merge.
Diagnostics/checklist: refreshed full body and non-ignored comments have no Additional info to collect section. Exact merge/branch, source blobs/mechanism and bounded pipeline outcomes were collected. Original transient temporary-file bytes/filesystem interleaving remain unavailable; passing builds do not supply them. NUnit/Helix results do not apply to the original generation build tasks. Worker/IP/environment details are withheld; Azure artifacts are retention-limited.
Note
Generated by GitHub Copilot.
dalexsoto commentedon Oct 16, 2026
Two unfixed raw-head recurrences; fixed merge previews were not the tested source
Two new initial stage/job/task 1/1/1 failures match the existing source-generation defect:
net10.0losesmake generated-filescalls 01:33:52.119338 / .122694, guard 01:33:53.303367; Build task failed 01:34:06.net11.0losesBoth own logs report
SdkVersions.cs has been automatically re-generated, make Error 1, and MSB3073 fromassembly-preparer'smake generated-filesinvocation. Exact raw sources retain shared$@.tmpoutput and incorrect incremental declarations: common Make blob d2dddec4ea2236345471993a06e895ae05817426, preparer blobs 351c53faf5cf154a31c269d2995ddecf3dd53605 / 1ac6b0ccd65a8d6729a5020bb7d57e4656a80611. This is source-related, not infrastructure, a genuinely flaky test, or evidence that constants need blindly recommitting. Exact transient bytes/filesystem interleaving were not captured; no new decoded-binlog timing is claimed.The source distinction is decisive: CI merge previews 4b3d759 / 2d51ffe, based on fixed targets 153ba15 / e4732a6, do contain the corrected per-process/atomic Make rule and target without bad
Inputs/Outputs. But the actual Undo Github merge tasks, logs 61 / 46, switch to the unfixed raw heads above at 01:27:12 / 01:31:50. BuildCURRENT_HASH_LONGand the executed target/error lines independently agree. A fixed GitHub preview therefore does not establish fixed tested source.For #26893, Detect API changes was skipped after Build failed: neither current-PR nor stable comparisons executed. Publisher task
ddab1acc-f64a-509c-ea96-e2080ffe06f6, log 123, then fails on missingchange-detection.zip; this is a dependent consequence, not another root occurrence, API compatibility failure, or Windows-copy flake. That parent finished failed 01:45:51 UTC. No binding/suppression change is authorized by these results.Reconciled counts: 9 confirmed original failures / 8 distinct PRs in canonical history, 4 / 4 in this monitor. Only these two new Build occurrences are added; skipped dependents and publisher fallout are not.
Keep CLOSED / COMPLETED. The main-branch closure remains valid: the failures execute raw source missing the existing #26687 / release backport correction, not the corrected preview trees. The source correction must be incorporated in the actually tested raw heads. No trial retry or monitor source push was performed; the mandatory exact-model review gate is currently unavailable, so no replacement reviewers, worktree, source edit, commit, push or merge were used.
Diagnostics/checklist: refreshed full body and non-ignored history have no Additional info to collect section. Own raw/merge source blobs, checkout/undo provenance, framework/call/error times and comparison/publisher states were collected. Transient temporary-file bytes, exact filesystem interleaving and new decoded-binlog relative intervals remain unavailable/unclaimed. NUnit/Helix results do not apply to these Build-task failures. Worker/IP/environment and credentials are withheld; Azure artifacts are retention-limited.
Note
Generated by GitHub Copilot.
dalexsoto commentedon Oct 16, 2026
New Maestro head has the same unfixed generation root; external Build recovery is bounded
New occurrence: #26866 advanced externally from
1a7f3813to exact head 3171c7a. Its prior green/retried outcomes are historical, not results for this revision.Run 15708048 / definition 21922 / Build packages -> Build, initial 1/1/1, log 98, independently shows two generation calls 2026-10-16 01:45:04.100217 / .233787 UTC, then the regeneration guard 01:45:05.646620 and MSB3073 at preparer line 390; this time netstandard2.0 loses. Build failed 01:45:16, the job 01:46:06, and the parent 01:59:51 UTC.
Own raw sources again retain common Make blob d2dddec4ea2236345471993a06e895ae05817426 and preparer 1ac6b0ccd65a8d6729a5020bb7d57e4656a80611: shared temporary output and incorrect incremental declarations. The fixed preview a2c6a71 contains the corrected 08f3a5c8 / 8a272706 blobs from target e4732a6. But Undo Github merge log 61 checks out the PR head; Build
CURRENT_HASH_LONG=3171c7a0...and executed line 390 agree with unfixed raw source. This is not an inherited credential/RBS failure, an API compatibility finding, or a regression of the corrected preview.Counts: 10 confirmed original failures / 9 distinct PRs in reconciled canonical history, 5 / 5 in this monitor. One new Build occurrence is added. No source edit/worktree/push/merge is permitted on this bot PR, and no source-related trial retry was submitted. The three older accepted lifetime requests remain charged; a new head does not reset them.
Separate external outcome on #26892: unchanged head 0b95e90, run 15707932 / Build job 2, 2/2/1, log 232, actually executes both original
BuildSdkVersionscalls tomake generated-filesat 01:57:47.157873 / .175329 UTC and the Build task succeeds 02:00:37, job 02:03:35. This is not inferred solely from a green wrapper. The previously skipped dependent stages are now running; the parent is not claimed recovered.That retry was external, not a monitor request. Its observable accepted-request minimum remains at least 1 lifetime / 1 head, not certified exact grouping/provenance. A passing unchanged-source execution does not eliminate the generation defect, turn it into a genuinely flaky test, authorize monitor trial reruns, erase failures, reset budgets, or satisfy the still-unavailable mandatory source-review gate.
Keep CLOSED / COMPLETED. Both new failure and passing execution concern raw source still missing the existing correction; the main-branch closure remains valid. No constants, bindings, API suppressions or CI-source-selection policy were changed by this monitor.
Diagnostics/checklist: refreshed full body/non-ignored history have no Additional info to collect section. Own new-head/source/undo/hash/error/task/parent data and the actual external generation-execution outcome were collected. Exact transient temporary-file bytes/filesystem interleaving and new decoded-binlog relative intervals remain unavailable/unclaimed. NUnit/Helix results do not apply to these Build tasks. Worker/IP/environment and credentials are withheld; artifacts are retention-limited.
Note
Generated by GitHub Copilot.
dalexsoto commentedon Oct 16, 2026
External API retry repeats the original source-build failure; no comparison ran
New failed execution, same PR/head: #26893, exact unchanged raw head 4e6f5a6, run 15707985 / definition 22268 / Detect job 2 / Build, stage/job/task 2/2/1, log 183.
Own retry log 183 shows two
make generated-filescalls at 2026-10-16 02:04:23.979692 / 02:04:24.087304 UTC, then theSdkVersions.csregeneration guard at 02:04:25.526349, make Error 1 and MSB3073 / preparer line 390 / net11.0. Build 2 failed 02:04:44 UTC, the job 02:05:26, and the parent 02:08:29.This source attribution was checked again, not inherited from attempt 1: retry Undo Github merge log 146 switches from fixed preview 2d51ffe... to raw 4e6f5a6... at 01:53:10 UTC; retry Build
CURRENT_HASH_LONGagrees. That raw head still has Make blob d2dddec4ea2236345471993a06e895ae05817426 and preparer 1ac6b0ccd65a8d6729a5020bb7d57e4656a80611, with the known shared temporary output and incorrect incremental declarations.Neither API comparison ran. Retry
Detect API changes, task 85bb0e1c-87a3-5d19-ec03-9df30b0ce8b4, was skipped after Build failed. Publisher 2, task 2d23ecaa-219a-52d7-f057-2a90cec312b0, log 223, subsequently reports ENOENT for the unproducedchange-detection-2/change-detection.zip. That is a dependent consequence, not another root failure, a Windows-copy flake, or evidence of current-PR/stable API incompatibility.Reconciled counts: 11 confirmed original failures / 9 distinct PRs, 6 / 5 in this monitor. This adds one failed attempt, not a new PR; publisher fallout and terminal status are not counted as additional failures.
The retry was external, not a monitor request. Its observable accepted-request minimum remains at least 1 lifetime / 1 on this head, not certified exact grouping: actor, actual submission time and payload remain unavailable. Dependent publisher execution adds no request charge; failure or success resets no budget.
Keep CLOSED / COMPLETED. The actual raw source still omits the existing correction, while the preview tree includes it; this does not invalidate the main-branch fix closure. No source-related trial retry, constants/binding/suppression edit, worktree, commit, push or merge was performed by this monitor. The mandatory exact-model source-review gate remains unavailable; no replacement models were used to finalize or push a correction.
Diagnostics/checklist: refreshed full body/non-ignored history has no Additional info to collect section. Retry-specific source/undo/hash, generation/error/framework and skipped-comparison/publisher/parent data were collected. Exact transient temporary-file bytes/filesystem interleaving and new decoded-binlog relative intervals remain unavailable/unclaimed. NUnit/Helix results do not apply to these Build tasks. Worker/IP/environment and credentials are withheld; artifacts are retention-limited.
Note
Generated by GitHub Copilot.
dalexsoto commentedon Oct 16, 2026
Third external API attempt recovered; current-PR comparison was skipped, not passed
#26893, unchanged raw head 4e6f5a6, run 15707985 / definition 22268 / Detect job 3, stage/job/task 3/3/1:
Original Build execution actually recovered. Build task 3991de11-94d4-59b4-e7d5-b73f5bf78128, log 284, executes both
BuildSdkVersionscalls at 2026-10-16 02:42:24.104518 / .104597 UTC, retainsCURRENT_HASH_LONG=4e6f5a67..., and succeeds 02:47:05. This is not inferred solely from a green job wrapper.The comparisons were verified separately using task e9141ecf-92be-561f-a88f-214da7c03629, log 298, and its own source-GUID-matched change-detection-3 artifact:
/Applications/Xcode_27.1.0.app, marketing 27.1. The system-requirements mismatch is recorded; the skipped comparison folder is absent.The macOS raw APICompat output retains unnecessary-suppression diagnostics; their suggested command was not executed. All four actual textual Markdown diffs are non-empty: iOS 40,043 bytes, tvOS 3,637, macOS 49,660, Mac Catalyst 15,625. The generated comment's "empty diffs" summary was not used as proof of raw output shape. No binding, suppression or baseline edit is justified by these results.
Detect job 3 succeeded 02:48:59 UTC, publisher job 3 02:52:35, and the complete API parent 02:52:41. This is bounded Build/stable-compatibility/API-pipeline recovery, not a passed current-PR comparison, whole simulator-PR recovery, or elimination of the unfixed raw generation rule.
Counts remain 11 original failures / 9 distinct PRs in reconciled canonical history, 6 / 5 in this monitor. Both retries were external, not monitor requests. Their observable accepted-request minima remain at least 2 lifetime / 2 on this head, not certified exact grouping/provenance; successful targets and dependent publishers erase no failure or charge and reset no budget.
Keep CLOSED / COMPLETED. The main-branch fix closure remains valid. The mandatory exact-model source-review gate is still unavailable, and a successful unchanged-source execution does not satisfy it. No replacement reviewers, source edit, worktree, commit, push, merge, forced historical Xcode installation or source-related monitor trial retry occurred.
Diagnostics/checklist: refreshed full body/non-ignored history have no Additional info to collect section. Original-target execution, exact raw head, comparison baseline/reference/assembly versions, all raw APICompat outputs, actual textual shapes and publisher/parent terminal metadata were collected. Original transient temporary-file bytes/filesystem interleaving remain unavailable; later successful builds do not backfill them. No new decoded-binlog timing is claimed. NUnit/Helix results do not apply to the original Build failures. Worker/IP/environment and credentials are withheld; artifacts are retention-limited.
Note
Generated by GitHub Copilot.
dalexsoto commentedon Oct 16, 2026
External Build 2 executed the original generation phase successfully; raw rules remain unfixed
#26866, unchanged raw head 3171c7a, definition 21922, run 15708048 / Build packages job 2 / Build, external stage/job/task 2/2/1, log 232.
Original generation execution actually completed, not a skipped Exec/green wrapper:
BuildSdkVersionsinvokesmake generated-filestwice at 2026-10-16 14:50:00.007106 / 14:50:00.079685 UTC. BothSdkVersions.csandProductConstants.cshave two actualGENmessages at 14:50:02.387589-14:50:02.387859, followed by both netstandard2.0 and net11.0 compilations. The Build task succeeds 14:52:59.093333 UTC, job 2 14:55:49.256667, without this invocation's regeneration-check failure.Own Undo Github merge task
d350ef53-545b-5076-5521-1cb7ecc2f7ed, log 194, explicitly checks out 3171c7a..., rather than retaining fixed-base merge previewa2c6a713.... Raw rules remain the previously verified Make.commond2dddec4.../ assembly-preparer1ac6b0cc...: this is not evidence that the raw branch incorporated the main fix. Log SHA-256 values are 977bd859f876c750c76a4279a015866fe4a3e85167a5b3776ceabef099911ddb (Build) and bb02378eb83f01c77699bb107907417b8317e917e659523cf4a34c240326c4e4 (Undo).Bounded execution recovery only. No source correction, concurrency stress result, exact temporary-file bytes/filesystem interleaving or permanent generation fix is established. This remains a source-related generation/concurrency defect, not newly classified infrastructure or a genuinely flaky product test. Runtime, whole Windows and publisher stages materialized as dependent attempt-2 work; their results and complete parent are still pending at the last scope read.
Keep CLOSED / COMPLETED with #26687's main-branch fix closure intact. Counts remain 11 original failures / 9 distinct PRs in reconciled canonical history, 6 / 5 in this monitor; this positive outcome adds no failure or erasure.
The request was external, not monitor-submitted or endorsement of a source-related trial retry. Observable minima are at least 4 lifetime / 1 on current 3171..., preserving the three older accepted monitor requests on 1a7f381...; exact all-actor grouping/actor/submission remains uncertified. Dependent stages/publishers add no request charge or budget reset. Bot source remains read-only: no source edit, worktree, generated-constant commit, push, merge, binding/suppression change or new monitor retry occurred.
Diagnostics/checklist: refreshed full body/non-ignored history and structured closure have no body Additional info to collect section. Exact original/retry execution, raw checkout, source-rule identity, command/GEN/time/framework progression and task/job outcomes were collected. Original transient temporary bytes, filesystem interleaving and concurrency-stress validation remain unavailable; no new decoded-binlog timing is claimed. NUnit/Helix do not apply to this original Build failure. Worker/IP/environment and credentials are withheld; artifacts are retention-limited.
Note
Generated by GitHub Copilot.
dalexsoto commentedon Oct 16, 2026
Generation recovered on Build 2, but its dependent runtime stage failed
#26866, unchanged raw head 3171c7a, definition 21922, run 15708048 / dependent iOS job 2 / Run tests, stage/job/task 2/2/1, log 3053.
The earlier reported actual original BuildSdkVersions execution recovery remains valid: both generation Exec calls and both target-framework compilations completed; Build task
4580825b..., log 232, succeeded 14:52:59 UTC, Build job 2 14:55:49. It was bounded execution recovery on unfixed raw rules, not proof of the main source correction or concurrency stress.The dependent iOS task failed 2026-10-16 17:36:19.773333 UTC, job 2 17:38:16.586667, and simulator stage 2 17:38:20.683333. Its three different runtime failures are one own preconnection RBSAssertion/PosterUI configuration under #26227, and two connected credential-cleanup stalls under #26221. They are not additional SdkVersions generation failures.
Historical publisher stage 2 completed succeededWithIssues at 17:44:39.433333 UTC. The Add summaries task warned that dependency metadata length 218,678 exceeded the supported 32,766; this warning is not a named product-test failure or another accepted retry. The old complete-parent terminal finish/result was not independently captured before a new external simulator/iOS attempt 3 started 17:46:10.563333 and reset the current build state. Historical simulator/publisher timelines preserve the actual stage/job outcomes; no parent time/result or current retry recovery is invented.
Keep CLOSED / COMPLETED, with #26687's main-fix closure intact. Actual raw 3171... still omits that correction; this later runtime outcome neither invalidates the closure nor establishes a source/concurrency fix. Counts remain 11 original failures / 9 distinct PRs in reconciled canonical history, 6 / 5 in this monitor. Runtime failures are counted only under their own signatures.
Both current retry requests were external, not monitor-submitted. Observable minima are at least 5 lifetime / 2 on current 3171..., preserving three older monitor requests; exact actor/group/submission accounting remains uncertified. The new equivalent iOS retry is already active, so no duplicate monitor request occurs. Dependent stages/publishers add no charge or budget reset; no failure is erased. Bot source remains read-only: no source edit, worktree, generated-constant commit, push, merge, suppression/baseline change or source-related trial retry was performed.
Diagnostics/checklist: refreshed full body/non-ignored history and structured closure have no body Additional info to collect section. Actual original generation progression, separate failed runtime scope, historical stage/job/publisher state/times, summary-warning lengths and new accepted-attempt identity were collected. Old whole-parent terminal metadata, transient generation temporary-file bytes/interleaving and concurrency-stress proof remain unavailable; new runtime blocked stacks/deeper causes are scoped under their own issues. No new binlog timing or NUnit result for the original Build failure is claimed; no Helix item applies. Worker/IP/environment, credentials and human identities are withheld; artifacts are retention-limited.
Note
Generated by GitHub Copilot.
dalexsoto commentedon Oct 16, 2026
Final accepted-run outcome after external merge: generation recovery remained bounded
#26866, raw head 3171c7a, run 15708048 / definition 21922, has now reached completed / failed at 2026-10-16 20:15:52.935390 UTC on external simulator/iOS stage/job attempt 3.
The original Build 2 / BuildSdkVersions actual generation-phase recovery already reported remains valid, but the raw generation rules remain unfixed and no concurrency stress/source correction is established. The three subsequent failed configurations from iOS 2 actually recovered on iOS 3: original CoreCLR/universal 108 produced 3,724 Passed / 0 Failed managed NUnit leaves, and original credential configurations 120 / 158 each passed all seven actual named credential cases.
The final run still failed on different Debug/CoreCLR 102, with its own preconnection RBSAssertion/PosterUI native fault under #26227. Credential recoveries are separately attributed under #26221. These are not another SdkVersions failure.
The PR had merged externally at 17:58:21 UTC into release/11.0.1xx, commit f0278a8, while its already-accepted iOS retry continued. Publisher 3 then failed independently: checkout task
9d6f084d..., log 3260, git exit 128 / missingrefs/pull/26866/mergeafter merge. Its own log SHA-256 is 292949a91bfbf7f31c3d07cd7c84fec029cdd1e706a6b6913b7d6d3fdf676848. Internal git-fetch/RetryHelper attempts are not new accepted CI requests or evidence of an SDK/runtime/authentication root.Keep CLOSED / COMPLETED with #26687's main-fix closure intact. Counts remain 11 original failures / 9 distinct PRs in reconciled canonical history, 6 / 5 in this monitor. Current runtime/checkout failures are keyed only to their own roots; successful generation or merge erases none.
The accepted out-of-scope watch is terminal; no further retry or ref restoration. Observable minima stay at least 5 lifetime / 2 on this head, exact all-actor grouping/provenance still uncertified. Existing failures, accepted charges and all action keys remain retained without refund/reset. No monitor CI, source edit, worktree, commit, push, merge, generated-constant/suppression/baseline change, authentication/retention bypass or remote deletion occurred.
Diagnostics/checklist: refreshed full body/non-ignored history and structured closure have no body Additional info to collect section. Original-generation scope, exact runtime-configuration recoveries, distinct current failure, independent post-merge checkout and terminal job/parent records were collected. Original generation temporary bytes/interleaving, concurrency-stress proof and deeper runtime initiating causes remain unavailable; no new binlog timing or actual NUnit for the original Build failure is claimed. No Helix item applies. Worker/IP/environment, credentials and human identities are withheld; artifacts are retention-limited.
Note
Generated by GitHub Copilot.