Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Repository	Step Name	Console log	Pull Request
1492477	dotnet/runtime	Checkout dotnet/runtime@refs/pull/130140/merge to s	Log	dotnet/runtime#130140
1491869	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127980/merge to s	Log	dotnet/runtime#127980
1490138	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67537/merge to s	Log	dotnet/aspnetcore#67537
1489961	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67536/merge to s	Log	dotnet/aspnetcore#67536
3012497	dotnet-maui	Checkout dotnet-maui@inflight/current to s	Log
1489822	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67485/merge to s	Log	dotnet/aspnetcore#67485
1489866	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67531/merge to s	Log	dotnet/aspnetcore#67531
1489868	dotnet/maui	Checkout dotnet/maui@refs/pull/36275/merge to s	Log	dotnet/maui#36275
1489617	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67516/merge to s	Log	dotnet/aspnetcore#67516
1489614	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67515/merge to s	Log	dotnet/aspnetcore#67515
1489644	dotnet/runtime	Checkout dotnet/runtime@refs/pull/130036/merge to s	Log	dotnet/runtime#130036
1489249	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67506/merge to s	Log	dotnet/aspnetcore#67506
1488528	dotnet/maui	Checkout dotnet/maui@refs/pull/34637/merge to s	Log	dotnet/maui#34637
3011560	dotnet-maui	Checkout dotnet-maui@inflight/candidate to s	Log
3011101	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/62529/merge to s	Log	#62529
1486344	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125436/merge to s	Log	dotnet/runtime#125436
1486361	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129988/merge to s	Log	dotnet/runtime#129988
1485979	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129836/merge to s	Log	dotnet/runtime#129836
1485710	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129965/merge to s	Log	dotnet/runtime#129965
1485618	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6452/merge to s	Log	dotnet/arcade-services#6452
1485465	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1483644	dotnet/diagnostics	Checkout dotnet/diagnostics@refs/pull/5907/merge to s	Log	dotnet/diagnostics#5907
1483783	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129912/merge to s	Log	dotnet/runtime#129912
1483675	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129904/merge to s	Log	dotnet/runtime#129904
1483514	dotnet/maui	Checkout dotnet/maui@refs/pull/36153/merge to s	Log	dotnet/maui#36153
3009079	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/62457/merge to s	Log	#62457
3009076	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/62456/merge to s	Log	#62456
1483007	dotnet/maui	Checkout dotnet/maui@inflight/candidate to s	Log
3008707	dotnet-dotnet-docker	Checkout dotnet-dotnet-docker@nightly to s	Log
1482440	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129766/merge to s	Log	dotnet/runtime#129766
3008217	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/62411/merge to s	Log	#62411
3008215	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/62410/merge to s	Log	#62410
3007886	dotnet-dotnet-buildtools-prereqs-docker	Checkout dotnet-dotnet-buildtools-prereqs-docker@dkurepa/ContinueOnBuildFailures to s	Log
1480597	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129814/merge to s	Log	dotnet/runtime#129814
1480365	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129826/merge to s	Log	dotnet/runtime#129826
1479843	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129810/merge to s	Log	dotnet/runtime#129810
1479916	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54964/merge to s	Log	dotnet/sdk#54964
1479919	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67399/merge to s	Log	dotnet/aspnetcore#67399
1479513	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51088/merge to s	Log	dotnet/sdk#51088
1479072	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129334/merge to s	Log	dotnet/runtime#129334
1478796	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54871/merge to s	Log	dotnet/sdk#54871
1478617	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54945/merge to s	Log	dotnet/sdk#54945
1478598	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54945/merge to s	Log	dotnet/sdk#54945
1478078	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54949/merge to s	Log	dotnet/sdk#54949
1477909	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67156/merge to s	Log	dotnet/aspnetcore#67156
1477916	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67157/merge to s	Log	dotnet/aspnetcore#67157
1477913	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67157/merge to s	Log	dotnet/aspnetcore#67157
1477915	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67157/merge to s	Log	dotnet/aspnetcore#67157
1476316	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126367/merge to s	Log	dotnet/runtime#126367
1475723	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129652/merge to s	Log	dotnet/runtime#129652
1474899	dotnet/maui	Checkout dotnet/maui@darc-net11.0-167dae9d-c756-48bc-a769-baa4b1f0abe2 to s	Log
1474394	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67341/merge to s	Log	dotnet/aspnetcore#67341
1473365	dotnet/sdk	Checkout dotnet/sdk@main to s	Log
3004249	dotnet-installer	Clone dotnet/installer	Log	#62258
3004194	dotnet-maui	Checkout dotnet-maui@release/10.0.1xx-sr8 to s	Log
1472925	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54892/merge to s	Log	dotnet/sdk#54892
1472559	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54890/merge to s	Log	dotnet/sdk#54890
1471366	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129563/merge to s	Log	dotnet/runtime#129563
1471288	dotnet/maui	Checkout dotnet/maui@refs/pull/35878/merge to s	Log	dotnet/maui#35878
1471235	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129587/merge to s	Log	dotnet/runtime#129587
1470679	dotnet/maui	Checkout dotnet/maui@refs/pull/35945/merge to s	Log	dotnet/maui#35945
1470376	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1469907	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129551/merge to s	Log	dotnet/runtime#129551
1469798	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67288/merge to s	Log	dotnet/aspnetcore#67288
1469371	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67279/merge to s	Log	dotnet/aspnetcore#67279
1469373	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67279/merge to s	Log	dotnet/aspnetcore#67279
1469372	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67279/merge to s	Log	dotnet/aspnetcore#67279
1469149	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54623/merge to s	Log	dotnet/sdk#54623
3002587	dotnet-sdk	Checkout dotnet-sdk@refs/pull/62190/merge to s	Log	#62190
1469207	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54723/merge to s	Log	dotnet/sdk#54723
1468036	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54761/merge to s	Log	dotnet/sdk#54761
1467884	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54729/merge to s	Log	dotnet/sdk#54729
1467015	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129260/merge to s	Log	dotnet/runtime#129260
1466217	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129454/merge to s	Log	dotnet/runtime#129454
1466332	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2612/merge to s	Log	maestro-auth-test/build-result-analysis-test#2612
1466331	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2612/merge to s	Log	maestro-auth-test/build-result-analysis-test#2612
1466164	dotnet/xharness	Checkout dotnet/xharness@refs/pull/1610/merge to s	Log	dotnet/xharness#1610
3001110	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
3000946	dotnet-sdk	Checkout dotnet-sdk@refs/pull/62118/merge to s	Log	#62118
1464983	dotnet/xharness	Checkout dotnet/xharness@refs/pull/1609/merge to s	Log	dotnet/xharness#1609
1464606	dotnet/maui	Checkout dotnet/maui@refs/pull/35898/merge to s	Log	dotnet/maui#35898
1464506	dotnet/arcade	Checkout dotnet/arcade@refs/pull/17009/merge to s	Log	#17009
1463138	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54729/merge to s	Log	dotnet/sdk#54729
1461862	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2607/merge to s	Log	maestro-auth-test/build-result-analysis-test#2607
1461861	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2607/merge to s	Log	maestro-auth-test/build-result-analysis-test#2607
1461747	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129332/merge to s	Log	dotnet/runtime#129332
1461788	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2605/merge to s	Log	maestro-auth-test/build-result-analysis-test#2605
1461787	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2605/merge to s	Log	maestro-auth-test/build-result-analysis-test#2605
1461677	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2602/merge to s	Log	maestro-auth-test/build-result-analysis-test#2602
1461676	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2602/merge to s	Log	maestro-auth-test/build-result-analysis-test#2602
1461593	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2601/merge to s	Log	maestro-auth-test/build-result-analysis-test#2601
1461592	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2601/merge to s	Log	maestro-auth-test/build-result-analysis-test#2601
1461581	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2600/merge to s	Log	maestro-auth-test/build-result-analysis-test#2600
1461580	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2600/merge to s	Log	maestro-auth-test/build-result-analysis-test#2600
1461539	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2599/merge to s	Log	maestro-auth-test/build-result-analysis-test#2599
1461538	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2599/merge to s	Log	maestro-auth-test/build-result-analysis-test#2599
1461498	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2598/merge to s	Log	maestro-auth-test/build-result-analysis-test#2598
1461497	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2598/merge to s	Log	maestro-auth-test/build-result-analysis-test#2598
1461404	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2597/merge to s	Log	maestro-auth-test/build-result-analysis-test#2597
1461405	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2597/merge to s	Log	maestro-auth-test/build-result-analysis-test#2597
Displaying 100 of 127 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	21	127

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?