The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1220069	dotnet/runtime	Send to Helix	Log
1219992	dotnet/runtime	Send tests to Helix (Unix)	Log
1220228	dotnet/runtime	Zip	Log	dotnet/runtime#122147
1220220	dotnet/runtime	Publish	Log	dotnet/runtime#118932
1220152	dotnet/runtime	Run Native AOT Test App Tests	Log	dotnet/runtime#122174
1220056	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51881
2853496	dotnet-dotnet	Build (centOSStreamContainer)	Log
1220098	dotnet/runtime	Build product	Log	dotnet/runtime#122173
2853384	dotnet-docker-tools	🟪 Generate EOL Data for all images in msdotnetcontainersstaging.azurecr.io	Log
1219932	dotnet/aspire	Run helix tests	Log
1219939	dotnet/runtime	Build product	Log	dotnet/runtime#122170
2853331	dotnet-aspire	Install MicroBuild plugin (Windows)	Log
2853239	dotnet-dotnet	Build (centOSStreamContainer)	Log
1219660	dotnet/runtime	Publish	Log	dotnet/runtime#122164
1219739	dotnet/runtime	Build product	Log	dotnet/runtime#122154
1219528	dotnet/runtime	Run Native AOT Test App Tests	Log	dotnet/runtime#121679
1219506	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#122157
1219422	dotnet/runtime	Send to Helix	Log
1219470	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#122045
1219426	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#121927
1218790	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119352
1219384	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#119634
1219362	dotnet/runtime	Send to Helix (Unix)	Log
1219301	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#117571
1219292	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#122072
1219278	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#122152
1219225	dotnet/runtime	Run Native AOT Test App Tests	Log	dotnet/runtime#122119
1219261	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#122048
1219214	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#122111
1219271	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121853
1219221	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#122119
1219207	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#121305
1219308	dotnet/xharness	Run tests in Helix	Log	dotnet/xharness#1478
1219307	dotnet/xharness	Run tests in Helix	Log	dotnet/xharness#1489
1219166	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#122127
1219171	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#122123
1219142	dotnet/runtime	Build native test components	Log	dotnet/runtime#122149
1219137	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#122147
1219104	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#121412
1219071	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#122148
1219073	dotnet/runtime	Run Native AOT Test App Tests	Log	dotnet/runtime#122148
1218858	dotnet/runtime	Run Native AOT Test App Tests	Log	dotnet/runtime#121924
1218841	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#121739
1218607	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122036
1218856	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#121924
1217987	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122094
1218813	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#119575
1218784	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121007
1218777	dotnet/runtime	Build native test components	Log	dotnet/runtime#121706
1218762	dotnet/runtime	Run Native AOT Test App Tests	Log	dotnet/runtime#122048
2852693	dotnet-dotnet	Build (centOSStreamContainer)	Log
2852870	dotnet-emsdk	Build	Log
2852541	dotnet-dotnet	Build (centOSStreamContainer)	Log
1218702	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#122072
2852820	dotnet-maui	Run Helix Tests	Log
2852802	dotnet-maui	Run Integration Tests - Build macOS	Log
1217377	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121800
1218585	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#121380
1218527	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121548
1218480	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121853
1218408	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#122128
1218396	dotnet/runtime	Publish	Log
1218439	dotnet/runtime	Send to Helix	Log
2852657	dotnet-maui	Run Helix Tests	Log
1218381	dotnet/runtime	Run Native AOT Test App Tests	Log	dotnet/runtime#121697
1218436	dotnet/runtime	Build product	Log
1217732	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122116
1218379	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#121697
1218362	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#122136
1217502	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121911
1218357	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#122135
1218354	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#122134
1218179	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122132
1218330	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#122119
2852573	dotnet-runtime	Send job to Helix (Unix)	Log
2852547	dotnet-docker-tools	🟪 Generate EOL Data for all images in msdotnetcontainersstaging.azurecr.io	Log
1218247	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#122133
1218257	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121614
1218131	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#122125
1218136	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#119432
1218118	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#122048
1218121	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#122123
1217597	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121412
1218072	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121862
1218233	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#51984
1218271	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#64433
1218232	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51935
1218227	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#51934
1218242	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#64552
1218220	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#51925
1218206	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#51847
1218111	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121853
1218068	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#122127
1218034	dotnet/runtime	Run Native AOT Test App Tests	Log	dotnet/runtime#122128
1218032	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#122128
1218013	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#111586
2852429	dotnet-try	Build	Log
1218149	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122130
1218130	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#51978
1217875	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#121973
Displaying 100 of 1980 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
47	251	1980

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez