slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1220383	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#121853
1220359	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122173
1220228	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#122147
1220220	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#118932
1220069	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
1220152	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122174
1220056	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51881
1220098	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#122173
1219939	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#122170
1219739	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#122154
1219660	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#122164
1219470	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122045
1219528	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#121679
1219506	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#122157
1219426	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121927
1219384	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#119634
1219362	dotnet/runtime	osx-arm64 Release Installer_Build_And_Test	Log
1219422	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1218790	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#119352
1219292	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122072
1219301	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#117571
1219278	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122152
1219261	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122048
1219214	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122111
1219221	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122119
1219207	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121305
1219308	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1478
1219307	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1489
1219166	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122127
1219171	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122123
1219271	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#121853
1219137	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122147
1219142	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122149
1219104	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121412
1219225	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122119
1219071	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122148
1219073	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122148
1218841	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121739
1218856	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121924
1218813	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#119575
1218607	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122036
1217987	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#122094
1218784	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#121007
1218777	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121706
1218858	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#121924
1218702	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122072
1218762	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122048
2852802	dotnet-maui	Build macOS	Log
1218585	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121380
1217377	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#121800
1218527	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#121548
1218408	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122128
1218480	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#121853
1218396	dotnet/runtime	osx-arm64 Release Installer_Build_And_Test	Log
1218439	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1218436	dotnet/runtime	osx-x64 Release NativeAOT_Libs	Log
1218379	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121697
1218362	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122136
1217732	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122116
1218357	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122135
1218381	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#121697
1218354	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122134
1217502	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#121911
1218330	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122119
1218179	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#122132
2852573	dotnet-runtime	Performance ios_scenarios iOSCoreCLR Interpreter iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1218247	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122133
1218257	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121614
1218131	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122125
1218136	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#119432
1218121	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#122123
1218118	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122048
1218072	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#121862
1218233	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#51984
1218232	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#51935
1218227	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51934
1217597	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#121412
1218220	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51925
1218206	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51847
1218068	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122127
1218032	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122128
1218013	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#111586
1218111	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#121853
1217999	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122126
1218130	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#51978
1217971	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#114983
2852388	dotnet-maui	Build macOS	Log
1217937	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122112
1217875	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121973
1217916	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121679
1217922	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121643
1217838	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#121158
2852351	dotnet-runtime	Performance ios_scenarios iOSCoreCLR Interpreter iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1217869	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122036
1217872	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122121
1217892	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122120
1218034	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122128
1216521	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122076
1217704	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122114
1217669	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122111
Displaying 100 of 682 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
36	196	682

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.