Unable to launch enclave: AesmCode(ServiceUnavailable_30) error

[carelsarthur]

After successfully using the Fortanix EDP runtime for several months, I am unable to launch enclaves since this week.
I contacted Fortanix about this and they suggested me to file an issue over here.
I haven't performed any updates/installed new things, so I am puzzled as to why enclaves suddenly don't launch anymore.

The issue can be reproduced by running the "sgx-detect" command (tool from Fortanix ) or by trying to run an enclave using Fortanix Rust EDP (see below).

This is the output I get at the moment when trying to run a program inside an enclave:

  > cargo run --target x86_64-fortanix-unknown-sgx -Zbuild-std

   Finished dev [unoptimized + debuginfo] target(s) in 0.03s
   Running `ftxsgx-runner-cargo target/x86_64-fortanix-unknown-sgx/debug/test-app
   Error: AesmCode(ServiceUnavailable_30)

   The EINITTOKEN provider didn't provide a token

   While loading SGX enclave
   ERROR: while running "ftxsgx-runner" "target/x86_64-fortanix-unknown-sgx/debug/test-app.sgxs" got exit status: 1 

Relevant output & commands:

CPU version: Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz

> uname -r -v
5.10.0-1008-oem #9-Ubuntu SMP Tue Dec 15 14:22:38 UTC 2020
> lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.4 LTS
Release:	20.04
Codename:	focal

Using sgx-detect:

> sgx-detect --verbose

Detecting SGX, this may take a minute...
✔  SGX instruction set
 ✔  CPU support
 ✔  CPU configuration
 ✔  Enclave attributes
 ✔  Enclave Page Cache
 SGX features
   ✘  SGX2  ✘  EXINFO  ✘  ENCLV  ✘  OVERSUB  ✘  KSS  
   Total EPC size: 93.5MiB
✘  Flexible launch control
 ✘  CPU support
✘  SGX system software
 ✔  SGX kernel device (/dev/sgx)
 ✔  libsgx_enclave_common
 ✔  AESM service
 ✘  Able to launch enclaves
   ✘  Debug mode

🕮  SGX system software > Able to launch enclaves > Debug mode
The enclave could not be launched.

debug: failed to load report enclave
debug: cause: failed to load report enclave
debug: cause: The EINITTOKEN provider didn't provide a token
debug: cause: aesm error code ServiceUnavailable_30

More information: https://edp.fortanix.com/docs/installation/help/#run-enclave-debug

> dmesg -T | grep sgx

[Do Mär 10 12:40:56 2022] isgx: loading out-of-tree module taints kernel.
[Do Mär 10 12:40:56 2022] isgx: module verification failed: signature and/or required key missing - tainting kernel
[Do Mär 10 12:40:56 2022] intel_sgx: Intel SGX Driver v2.11.1
[Do Mär 10 12:40:56 2022] intel_sgx INT0E0C:00: EPC bank 0x70200000-0x75f80000
[Do Mär 10 12:40:56 2022] intel_sgx:  can not reset SGX LE public key hash MSRs
[Do Mär 10 12:40:56 2022] intel_sgx: second initialization call skipped

> systemctl status aesmd.service

● aesmd.service - Intel(R) Architectural Enclave Service Manager
    Loaded: loaded (/lib/systemd/system/aesmd.service; enabled; vendor preset: enabled)
    Active: active (running) since Thu 2022-03-10 12:40:59 CET; 5h 38min ago
   Process: 1120 ExecStartPre=/opt/intel/sgx-aesm-service/aesm/linksgx.sh (code=exited, status=0/SUCCESS)
   Process: 1150 ExecStartPre=/bin/mkdir -p /var/run/aesmd/ (code=exited, status=0/SUCCESS)
   Process: 1153 ExecStartPre=/bin/chown -R aesmd:aesmd /var/run/aesmd/ (code=exited, status=0/SUCCESS)
   Process: 1155 ExecStartPre=/bin/chmod 0755 /var/run/aesmd/ (code=exited, status=0/SUCCESS)
   Process: 1157 ExecStartPre=/bin/chown -R aesmd:aesmd /var/opt/aesmd/ (code=exited, status=0/SUCCESS)
   Process: 1160 ExecStartPre=/bin/chmod 0750 /var/opt/aesmd/ (code=exited, status=0/SUCCESS)
   Process: 1162 ExecStart=/opt/intel/sgx-aesm-service/aesm/aesm_service (code=exited, status=0/SUCCESS)
  Main PID: 1199 (aesm_service)
     Tasks: 4 (limit: 23646)
    Memory: 8.2M
    CGroup: /system.slice/aesmd.service
            └─1199 /opt/intel/sgx-aesm-service/aesm/aesm_service

Mär 10 18:12:32 arthur-Lenovo-ideapad-520-15IKB aesm_service[1199]: InKernel LE loaded
Mär 10 18:12:32 arthur-Lenovo-ideapad-520-15IKB aesm_service[1199]: InKernel LE loaded
Mär 10 18:12:32 arthur-Lenovo-ideapad-520-15IKB aesm_service[1199]: InKernel LE loaded
Mär 10 18:12:32 arthur-Lenovo-ideapad-520-15IKB aesm_service[1199]: InKernel LE loaded
Mär 10 18:12:32 arthur-Lenovo-ideapad-520-15IKB aesm_service[1199]: InKernel LE loaded
Mär 10 18:12:32 arthur-Lenovo-ideapad-520-15IKB aesm_service[1199]: InKernel LE loaded
Mär 10 18:12:32 arthur-Lenovo-ideapad-520-15IKB aesm_service[1199]: InKernel LE loaded
Mär 10 18:12:32 arthur-Lenovo-ideapad-520-15IKB aesm_service[1199]: InKernel LE loaded
Mär 10 18:12:32 arthur-Lenovo-ideapad-520-15IKB aesm_service[1199]: InKernel LE loaded
Mär 10 18:14:58 arthur-Lenovo-ideapad-520-15IKB aesm_service[1199]: InKernel LE loaded

> dpkg -l | grep aesm

ii  libsgx-aesm-epid-plugin                       2.15.101.1-focal1                     amd64        EPID Quote Plugin for Intel(R) Software Guard Extensions AESM Service
ii  libsgx-aesm-launch-plugin                     2.15.101.1-focal1                     amd64        Launch Plugin for Intel(R) Software Guard Extensions AESM Service
ii  libsgx-aesm-pce-plugin                        2.15.101.1-focal1                     amd64        PCE Plugin for Intel(R) Software Guard Extensions AESM Service
ii  sgx-aesm-service                              2.15.101.1-focal1                     amd64        Intel(R) Software Guard Extensions AESM Service

Kind regards

[llly]

uname -r -v
5.10.0-1008-oem #9-Ubuntu SMP Tue Dec 15 14:22:38 UTC 2020
✔ SGX kernel device (/dev/sgx)

Did you recently update Linux kernel?
This kernel seems to be a customized 5.10 and contains inkernel SGX driver. You can see that there are two different sgx driver in dmesg isgx and intel_sgx.

[gausk]

@carelsarthur We recently updated intel-sgx-dkms package to install driver based on intel/linux-sgx-driver#138. This driver support both EPID and DCAP attestation with SGX and SGX2 but works only on node that has Flexible Launch Control enabled. As your node does not have FLC enabled so you should downgrade intel-sgx-dkms to older version.
To downgrade to OOT driver( support only EPID) run below command:

sudo apt install -y --allow-downgrades intel-sgx-dkms=2.11-1
ls -ltr /dev/isgx
sudo systemctl restart aesmd

Let us know if are still facing issue after following above suggestion.

[carelsarthur]

This fixed the issue.
Thank you!