Problem with deploying the NSX Edge cluster

[pcgeek2009]

Describe the bug

The Edge Cluster Deployment Task fails. I have retried several times from both the installer and within the SDDC Manager. Below is the output from the installer:
14-02-2026 13:42:08 Add-NSX-Edge-Cluster[1504555]: [INFO] Retrying edge cluster deployment task
14-02-2026 13:42:08 Add-NSX-Edge-Cluster[1504555]: [INFO] Getting latest SDDC Manager Task ID
14-02-2026 13:42:08 Add-NSX-Edge-Cluster[1504555]: [INFO] Found existing NSX Edge Cluster deployment task in progress in SDDC Manager.
14-02-2026 13:42:10 Add-NSX-Edge-Cluster[1504555]: [INFO] Edge cluster deployment task started
14-02-2026 13:42:10 Add-NSX-Edge-Cluster[1504555]: [INFO] Getting latest SDDC Manager Task ID
14-02-2026 13:42:10 Add-NSX-Edge-Cluster[1504555]: [ERROR] Unable to obtain Task ID for this operation, please check the domainmanager logs for more info.
Exception: Failed to deploy HoloDeck

The error from the domainmanger.log file shows the following

2026-02-13T19:01:48.662+0000 DEBUG [vcf_dm,698f751bdca2fd2e213f6ab0c492ed33,b6c1] [c.v.e.s.c.s.DomainCapabilitiesApiServiceImpl,vac-scheduler-1]  Refreshed domainCapab3-4e48-9e87-9c7a7574de50, vcenterId=55b8ec02-ecf1-4775-aa2d-0642a3e56754, vcenterFqdn=vc-mgmt-a.site-a.vcf.lab, capabilities=[REPOSITORY_IMAGES, HETEROGENEOUS_CLUSTER]
2026-02-13T19:06:44.334+0000 ERROR [vcf_dm,698f756cdc7618ed985ee774d68bce9f,673e] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-7]  [JPOLP6] DEPLOY_NSXT_EDGE_FAILED Fail
2026-02-13T19:06:44.386+0000 DEBUG [vcf_dm,698f756cdc7618ed985ee774d68bce9f,673e] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-7]  Collected the following errors for 9c-572ee92f006b: [ExecutionError [errorCode=null, errorResponse=LocalizableErrorResponse(messageBundle=com.vmware.vcf.common.fsm.plugins.nsxt.messages)]]
2026-02-13T19:06:49.025+0000 DEBUG [vcf_dm,698f764421e707d552886f87e386001f,de96] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-17]  Collected the following errors for19c-572ee92f006b: [ExecutionError [errorCode=null, errorResponse=LocalizableErrorResponse(messageBundle=com.vmware.vcf.common.fsm.plugins.nsxt.messages)]]
2026-02-13T19:06:50.246+0000 DEBUG [vcf_dm,698f7648d38e75651f18520a2174621a,de78] [c.v.e.s.c.s.DomainCapabilitiesApiServiceImpl,vac-scheduler-1]  Refreshed domainCapab3-4e48-9e87-9c7a7574de50, vcenterId=55b8ec02-ecf1-4775-aa2d-0642a3e56754, vcenterFqdn=vc-mgmt-a.site-a.vcf.lab, capabilities=[REPOSITORY_IMAGES, HETEROGENEOUS_CLUSTER]
2026-02-13T19:09:55.029+0000 INFO  [vcf_dm,698f77033a9fa8bf120ff80e2a3125aa,0465] [c.v.v.q.i.util.QueryServiceMetrics,vac-scheduler-1]  QueryService metrics: [total qu+ data sources = 0 / 0 / 0, above 1+ / 2+ / 4+ secs = 0 / 0 / 0, with errors = 0]
2026-02-13T19:10:36.665+0000 ERROR [vcf_dm,698f76499e70e777959093dbe764fa23,1b0c] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-15]  [434T1U] DEPLOY_NSXT_EDGE_FAILED Fai
2026-02-13T19:10:36.669+0000 DEBUG [vcf_dm,698f76499e70e777959093dbe764fa23,1b0c] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-15]  Collected the following errors for19c-572ee92f006b: [ExecutionError [errorCode=null, errorResponse=LocalizableErrorResponse(messageBundle=com.vmware.vcf.common.fsm.plugins.nsxt.messages)]]
2026-02-13T19:10:47.930+0000 DEBUG [vcf_dm,698f772c2039867cdc746740a9b772ae,602d] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-2]  Collected the following errors for 9c-572ee92f006b: [ExecutionError [errorCode=null, errorResponse=LocalizableErrorResponse(messageBundle=com.vmware.vcf.common.fsm.plugins.nsxt.messages)]]
2026-02-13T19:10:48.081+0000 ERROR [vcf_dm,698f756a61ca41496a46fdf526cdf638,4746] [c.v.v.n.f.a.DeployAndConfigureMultipleNsxtEdgeNodesAction,dm-exec-35]  Exception Int
2026-02-13T19:10:48.082+0000 ERROR [vcf_dm,698f756a61ca41496a46fdf526cdf638,4746] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-35]  [DTEC27] ORCH_TASK_INTERRUPTED The t
Caused by: com.vmware.evo.sddc.orchestrator.exceptions.OrchException: Internal error: Failed to initiate execution
2026-02-13T19:10:48.259+0000 DEBUG [vcf_dm,698f756a61ca41496a46fdf526cdf638,4746] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-35]  Collected the following errors fora010105-9c57-16f4-819c-5729ee50002d: [ExecutionError [errorCode=null, errorResponse=LocalizableErrorResponse(messageBundle=com.vmware.evo.sddc.orchestrator.model.error
2026-02-13T19:10:48.326+0000 DEBUG [vcf_dm,698f773885142d3cc9df7a7297cbbaf4,9e92] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-25]  Collected the following errors fora010105-9c57-16f4-819c-5729ee50002d: [ExecutionError [errorCode=null, errorResponse=LocalizableErrorResponse(messageBundle=com.vmware.evo.sddc.orchestrator.model.error
2026-02-13T19:10:48.468+0000 INFO  [vcf_dm,698f77380c6f3539a4ea58e43cfff9be,394f] [c.v.e.s.o.core.OrchestratorImpl,dm-exec-25]  An error occurred in Execution ID e84b8
2026-02-13T19:11:51.509+0000 DEBUG [vcf_dm,698f7776ce82e86980d65afa197a5335,c20f] [c.v.e.s.c.s.DomainCapabilitiesApiServiceImpl,vac-scheduler-1]  Refreshed domainCapab3-4e48-9e87-9c7a7574de50, vcenterId=55b8ec02-ecf1-4775-aa2d-0642a3e56754, vcenterFqdn=vc-mgmt-a.site-a.vcf.lab, capabilities=[REPOSITORY_IMAGES, HETEROGENEOUS_CLUSTER]
2026-02-13T19:16:53.239+0000 DEBUG [vcf_dm,698f78a373b6c642b466ab67a7aa8ac3,638e] [c.v.e.s.c.s.DomainCapabilitiesApiServiceImpl,vac-scheduler-1]  Refreshed domainCapab3-4e48-9e87-9c7a7574de50, vcenterId=55b8ec02-ecf1-4775-aa2d-0642a3e56754, vcenterFqdn=vc-mgmt-a.site-a.vcf.lab, capabilities=[REPOSITORY_IMAGES, HETEROGENEOUS_CLUSTER]

Reproduction steps

1. Deployment using the 9.0.2 installer on the 9.0.1 installation
2. Performing a full stack install with Automation and Supervisor
3. Used the following command for deployment: "New-HoloDeckInstance -Version '9.0.1.0' -InstanceID 'gray' -WorkloadDomainType 'SharedSSO' -NsxEdgeClusterMgmtDomain -NsxEdgeClusterWkldDomain -DeployVcfAutomation -DeploySupervisor"
   ...

Expected behavior

Edge cluster should deploy without issue

Additional context

No response

[dhruv-tyagi-broadcom]

Can you login to VCF Operations and look at the Tasks section to see why the NSX Edge cluster deployment failed? You can check the same thing in SDDC Manager Tasks section as well

[pcgeek2009]

Here are the screenshots of both the SDDC manager and operations. Also note, there seems to be password verification errors as well for some reason.

[dhruv-tyagi-broadcom]

Definitely something going wrong in the environment. I hope there's enough physical resources available to the nested deployment?

Are you able to restart the NSX Edge cluster deployment from the SDDC Manager UI by clicking the restart button for that task?

The password sync errors should not have happened as well

[pcgeek2009]

I do not appear to be using anywhere near my resources. This is a HP Proliant Gen 9 server with Dual Intel Xeon E5-2697Av4 CPU's with 16 cores each (64 threads total) and 1TB of memory.

[pcgeek2009]

I have restarted it in the SDDC manager

[pcgeek2009]

This is the same environment I used 5.2 in without any issue. I did run a health check and the errors only seem to be on the NSX Edge management

root@sddcmanager-a [ /home/vcf ]# /opt/vmware/sddc-support/sos --password-health
Welcome to Supportability and Serviceability(SoS) utility!
Performing SoS operation for mgmt-a domain components
Health Check : /var/log/vmware/vcf/sddc-support/healthcheck-2026-02-16-04-29-59-726963
Health Check log : /var/log/vmware/vcf/sddc-support/healthcheck-2026-02-16-04-29-59-726963/sos.log
NOTE : The Health check operation was invoked without --skip-known-host-check, additional identity checks will be included for Connectivity Health, Password Health and Certificate Health Checks because of security reasons.

SDDC Manager : sddcmanager-a.site-a.vcf.lab
+-------------------------+-----------+
| Stage | Status |
+-------------------------+-----------+
| Bringup | Completed |
| Management Domain State | Completed |
+-------------------------+-----------+
+--------------------+----------+
| Component | Identity |
+--------------------+----------+
| SDDC-Manager | 10.1.1.5 |
| Number of Servers | 4 |
+--------------------+----------+
Password Expiry Status : RED
+-----+-------------------------------------+-----------------+-------------------+--------------+-----------------+-----------------------+
| SL# | Component | User | Last Changed Date | Expiry Date | Expires in Days | State |
+-----+-------------------------------------+-----------------+-------------------+--------------+-----------------+-----------------------+
| 1 | ESXI : esx-01a.site-a.vcf.lab | svc-vcf-esx-01a | Feb 11, 2026 | Never | Never | GREEN |
| | | root | Feb 11, 2026 | Never | Never | GREEN |
| 2 | ESXI : esx-02a.site-a.vcf.lab | svc-vcf-esx-02a | Feb 11, 2026 | Never | Never | GREEN |
| | | root | Feb 11, 2026 | Never | Never | GREEN |
| 3 | ESXI : esx-03a.site-a.vcf.lab | svc-vcf-esx-03a | Feb 11, 2026 | Never | Never | GREEN |
| | | root | Feb 11, 2026 | Never | Never | GREEN |
| 4 | ESXI : esx-04a.site-a.vcf.lab | svc-vcf-esx-04a | Feb 11, 2026 | Never | Never | GREEN |
| | | root | Feb 11, 2026 | Never | Never | GREEN |
| 5 | NSX : edge-mgmt-01a.site-a.vcf.lab | - | - | - | - | Failed to get details |
| 6 | NSX : edge-mgmt-02a.site-a.vcf.lab | - | - | - | - | Failed to get details |
| 7 | NSX : nsx-mgmt-a.site-a.vcf.lab | root | Feb 14, 2026 | May 15, 2026 | 88 days | GREEN |
| | | audit | Feb 14, 2026 | May 15, 2026 | 88 days | GREEN |
| 8 | SDDC : sddcmanager-a.site-a.vcf.lab | vcf | Feb 14, 2026 | Feb 14, 2027 | 363 days | GREEN |
| | | backup | Feb 14, 2026 | Feb 14, 2027 | 363 days | GREEN |
| | | root | Feb 14, 2026 | Feb 14, 2027 | 363 days | GREEN |
| 9 | vCenter : vc-mgmt-a.site-a.vcf.lab | root | Feb 14, 2026 | May 15, 2026 | 87 days | GREEN |
+-----+-------------------------------------+-----------------+-------------------+--------------+-----------------+-----------------------+

Legend:

GREEN - No attention required, health status is NORMAL
YELLOW - May require attention, health status is WARNING
RED - Requires immediate attention, health status is CRITICAL

Health Check completed successfully for : [VCF-SUMMARY]
Operation failed for : [PASSWORD-CHECK(2/16 Failed)]
For detailed report please refer : /var/log/vmware/vcf/sddc-support/healthcheck-2026-02-16-04-29-59-726963/report.json

[pcgeek2009]

I noticed William Lam's X post today that the full 9.02 is available. I have downloaded and will work to redeploy with that version to see if this is resolved. I did notice another post from him about resource lock issues when deploying Holodeck and had planned to try that workaround in case that was the issue, but then noticed the post about the new binaries.

[pcgeek2009]

So, with the 9.0.2 package, I seem to still have somewhat of an issue with the edge cluster, but this is now for the workload domain. It may be resource related though. Here is the output you suggested on the previous ticket.

Kube-Performance Command.txt

[dhruv-tyagi-broadcom]

So you can ignore the "No coredump" info message. That is not related.

At this point, you have 2 options. Wait for memory to free up till 64 GB after which the deployment will automatically move forward. Or manually increase the memory on the nested host to speed it up

[pcgeek2009]

So, if it has been that way for over 24 hours, I am guessing I will need to put each one in maintenance mode and add resources?

[dhruv-tyagi-broadcom]

yes, start with the ones that have the NSX Edge VM running on it. Ideally those should be enough

[pcgeek2009]

So I increased 1 host to 100G and 14 CPU's. The cluster appears to have more free resources, but the edge fails about 43%