Symbol missing issue with 1.3 version onwards in SLES and Intel Datacenter Max GPU on Aurora

[servesh]

We are seeing a symbol missing with xpu-smi newer than 1.3 version.

$ ./xpu-smi
./xpu-smi: symbol lookup error: /home/servesh/Tests/xpu-smi/usr/lib64/libxpum.so.1: undefined symbol: _ZN6spdlog7details15periodic_workerD1Ev

There seems to be a dependency on spdlog which isn't listed in the rpm spec,

$ rpm -qpR ./xpu-smi-1.3.2-20250825.080849.0d946904.x86_64.rpm
warning: ./xpu-smi-1.3.2-20250825.080849.0d946904.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 1b79da4d: NOKEY
/bin/sh
/bin/sh
/bin/sh
/bin/sh
/bin/sh
/bin/sh
intel-gsc >= 0.9.5
intel-level-zero-gpu >= 1.3.23726
level-zero >= 1.7.9.1
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(PayloadIsXz) <= 5.2-1

nor in the library list,

ldd ./xpu-smi
	linux-vdso.so.1 (0x00007ffed0d29000)
	libigsc.so.0 => /usr/lib64/libigsc.so.0 (0x00007f2d8a95e000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f2d8a959000)
	libxpum.so.1 => /home/servesh/Tests/xpu-smi/usr/lib64/libxpum.so.1 (0x00007f2d8a374000)
	libstdc++.so.6 => /opt/aurora/25.190.0/spack/unified/0.10.0/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib64/libstdc++.so.6 (0x00007f2d8a115000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f2d89fc8000)
	libgcc_s.so.1 => /opt/aurora/25.190.0/spack/unified/0.10.0/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib64/libgcc_s.so.1 (0x00007f2d89fa3000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2d89f7f000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f2d89d8a000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f2d8a9a9000)
	libmetee.so.5.0.0 => /usr/lib64/libmetee.so.5.0.0 (0x00007f2d89d80000)
	libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007f2d89d49000)
	libze_loader.so.1 => /usr/lib64/libze_loader.so.1 (0x00007f2d89c1f000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f2d89c15000)

ldd /home/servesh/Tests/xpu-smi/usr/lib64/libxpum.so.1
	linux-vdso.so.1 (0x00007fff489f2000)
	libze_loader.so.1 => /usr/lib64/libze_loader.so.1 (0x00007f5e5032e000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f5e50329000)
	libigsc.so.0 => /usr/lib64/libigsc.so.0 (0x00007f5e502f9000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5e502d5000)
	libstdc++.so.6 => /opt/aurora/25.190.0/spack/unified/0.10.0/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib64/libstdc++.so.6 (0x00007f5e50076000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f5e4ff29000)
	libgcc_s.so.1 => /opt/aurora/25.190.0/spack/unified/0.10.0/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib64/libgcc_s.so.1 (0x00007f5e4ff04000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f5e4fd0f000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f5e50a56000)
	libmetee.so.5.0.0 => /usr/lib64/libmetee.so.5.0.0 (0x00007f5e4fd05000)
	libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007f5e4fcce000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f5e4fcc2000)

It appears the if I pass the spdlog as LD_LIBRARY_PATH, this still doesn't help. The only way to get things to work is LD_PRELOAD the spdlog library.

LD_PRELOAD=/opt/aurora/25.190.0/spack/unified/0.10.0/install/linux-sles15-x86_64/gcc-13.3.0/spdlog-1.10.0-kjmwhnz/lib64/libspdlog.so ./xpu-smi
Intel XPU System Management Interface -- v1.3
Intel XPU System Management Interface provides the Intel datacenter GPU model. It can also be used to update the firmware.
Intel XPU System Management Interface is based on Intel oneAPI Level Zero. Before using Intel XPU System Management Interface, the GPU driver and Intel oneAPI Level Zero should be installed rightly.

It would also be good to understand why xpu-smi is deprecated for SLES and Intel Datacenter Max GPU. Aurora is the largest installation of Intel GPUs. Both the drivers and OneAPI SDK continue to support the system for atleast few more years. Many of users rely on xpu-smi as a means to access the counter data and having continued support is crucial. If we run into a bug or an issue at later date we will be stuck with an older version.

[saik-intel]

@servesh we will look internally and update you. Can you please see if 1.2.X is works for you for Intel Data Center Max GPU and update back ?

[servesh]

@saik-intel Yes, 1.2.X works. The issue is users tend to check versions and point out we are using older tools. Would be good to support xpu-smi updates for SLES and PVC GPUs.

[eero-t]

XPUM builds its own version of spdlog: https://github.com/intel/xpumanager/tree/master/third_party/spdlog

So it should not matter what is installed on the host.

Missing symbol is destructor:

$ echo _ZN6spdlog7details15periodic_workerD1Ev | c++filt
spdlog::details::periodic_worker::~periodic_worker()

I do not see such symbol for the Ubuntu version, not a single spdlog symbol, although there are references to other log destructors:

$ dpkg -s libxpum1 | grep Version
Version: 1.3.2-0ubuntu1~24.04~ppa2

$ dpkg -L libxpum1 | grep libxpum.so
/usr/lib/x86_64-linux-gnu/libxpum.so.1.3.2
/usr/lib/x86_64-linux-gnu/libxpum.so.1

$ readelf -sW /usr/lib/x86_64-linux-gnu/libxpum.so.1 | c++filt | grep spd

$ readelf -sW /usr/lib/x86_64-linux-gnu/libxpum.so.1 | c++filt | grep log.*~ | grep -vi topolog
   355: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND std::logic_error::~logic_error()@GLIBCXX_3.4 (4)

Is your build patched XPUM version?

[servesh]

@eero-t Using the RPM provided in the releases, https://github.com/intel/xpumanager/releases/download/v1.3.2/xpu-smi-1.3.2-20250825.080849.0d946904.x86_64.rpm

This is on a SLES system.

$ readelf -sW libxpum.so.1.3.2 | c++filt | grep spd
   277: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND spdlog::details::periodic_worker::~periodic_worker()
   420: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND spdlog::sinks::ansicolor_stdout_sink<spdlog::details::console_mutex>::ansicolor_stdout_sink(spdlog::color_mode)
   591: 000000000015dde0    61 FUNC    WEAK   DEFAULT   13 void spdlog::logger::warn<char const*&, char const*>(fmt::v11::fstring<char const*&, char const*>::t, char const*&, char const*&&)
   654: 00000000001cbbf0    58 FUNC    WEAK   DEFAULT   13 void spdlog::logger::info<int&>(fmt::v11::fstring<int&>::t, int&)
   701: 00000000001cc510   727 FUNC    WEAK   DEFAULT   13 void spdlog::logger::log_<long double&, long double&>(spdlog::source_loc, spdlog::level::level_enum, fmt::v11::basic_string_view<char>, long double&, long double&)
...
(truncated)

$ readelf -sW libxpum.so.1.3.2 | c++filt | grep log.*~ | grep -vi topolog
   277: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND spdlog::details::periodic_worker::~periodic_worker()
   508: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND std::logic_error::~logic_error()@GLIBCXX_3.4 (2)

[eero-t]

Oh, it's problem also with releases. XPUM should be building about same for all distros, using the builder containers making sure that environments are matching, and which do not have additional (distro) spdlog devel files against which XPUM could be accidentally linked against.

If I understand CMake file correctly, internal spdlog version should be linked as private: https://github.com/intel/xpumanager/blob/master/core/CMakeLists.txt#L184

Only potentially relevant change I can see between 1.2.41 and 1.3.0 is the target_link_directories addition:
6cb90fb#diff-8f52846645b80806e11fbe318820659ca6d81d2fe305d815bb97f62d2102e1fd

But I have no idea whether that could change linking visibility.

[eero-t]

Yes, 1.2.X works.

Only potentially relevant change I can see between 1.2.41 and 1.3.0 is the target_link_directories addition:

@servesh Just to verify, was it 1.2.41 that you tested, or some earlier 1.2.x version (which one)?

(I'm asking because find -name CMakeLists.txt | grep -v -e third_party/ -e doc/ | xargs gitk shows another linking related change in v1.2.37.)

[eero-t]

I extracted all the recent RPM based release files with rpm2cpio | cpio -idmv and checked them. Regardless of distro version, all recent xpu-xmi release RPMs suffer from the same undefined symbol(s) issue.

Between XPUM V1.2.42 => V1.3.0 releases, libxpum got 1 undefined spdlog symbol. This is potentially due to changes in CMakeLists.txt files.

Between XPUM V1.3.1 => v1.3.2 releases, another undefined spdlog symbol appeared. Most likely reason for that is vendored spdlog sources being updated to latest v1.15.3 spdlog release (from earlier 2021 spdlog v1.9.2 release).

Interesting question is why those releases worked in CI...

[servesh]

@eero-t Yes, we deploy the rpm with extraction and base a module around it on Aurora and has worked on 1.2.x releases so far.

I also have the same Q on how it worked in the CI...

[eero-t]

Btw. While release files need to be dowloaded manually, checking several versions is easy:

$ for i in ~/Downloads/xpu-smi-1*el7*.rpm; do rpm2cpio $i | cpio -idmv; done
$ mkdir -p centos7  &&  mv usr/lib64/libxpum.so.1.* centos7/  &&  rm -r usr/
$ for i in centos7/libxpum*; do echo $i:; readelf -sW $i | c++filt | grep spdlog | grep UND; done
centos7/libxpum.so.1.2.42:
centos7/libxpum.so.1.3.0:
   356: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND spdlog::sinks::ansicolor_stdout_sink<spdlog::details::console_mutex>::ansicolor_stdout_sink(spdlog::color_mode)
centos7/libxpum.so.1.3.1:
   356: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND spdlog::sinks::ansicolor_stdout_sink<spdlog::details::console_mutex>::ansicolor_stdout_sink(spdlog::color_mode)
centos7/libxpum.so.1.3.2:
   253: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND spdlog::details::periodic_worker::~periodic_worker()
   367: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND spdlog::sinks::ansicolor_stdout_sink<spdlog::details::console_mutex>::ansicolor_stdout_sink(spdlog::color_mode)

One cannot generalize such check for CI though, set of builtin libs could change (see #104), and all of them do not have a clear namespace to check for.

I also have the same Q on how it worked in the CI...

If tests are done as part of builds, tested SW may be using components from the build, instead of being constricted just to what's in the release package, and what its deps bring in.

[eero-t]

If you grep the XPUM sources for those undefined symbols: git grep -e periodic_worker -e ansicolor_stdout_sink

They are references just within spdlog sources, not elsewhere in XPUM. I.e. they seem to be spdlog internal symbols.

It's also interesting that none of the Ubuntu 22.04 / 24.04 releases has such issues, but Debian10 packages do have (just checked those):

$ for i in ~/Downloads/xpu-smi_1.3.*deb10*deb; do ar x $i; tar xf data.tar.gz; rm *.tar.*; done
$ mkdir -p debian10  &&  mv usr/lib64/libxpum.so.1.* debian10/  &&  rm -r usr/
$ for i in debian10/libxpum.so.1.*; do echo $i:; readelf -sW $i | c++filt | grep spdlog | grep UND; done
debian/libxpum.so.1.3.0:
   323: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND spdlog::sinks::ansicolor_stdout_sink<spdlog::details::console_mutex>::ansicolor_stdout_sink(spdlog::color_mode)
debian/libxpum.so.1.3.1:
   323: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND spdlog::sinks::ansicolor_stdout_sink<spdlog::details::console_mutex>::ansicolor_stdout_sink(spdlog::color_mode)
debian/libxpum.so.1.3.2:
   228: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND spdlog::details::periodic_worker::~periodic_worker()
   335: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND spdlog::sinks::ansicolor_stdout_sink<spdlog::details::console_mutex>::ansicolor_stdout_sink(spdlog::color_mode)

[servesh]

Thanks for digging. I'm not really sure why the mix in behavior. Is there CMake version difference between the different CI systems for each OS?

[saik-intel]

@servesh While the commercial lifecycle of these GPUs is winding down, Intel remains fully committed to supporting existing customers. This includes addressing issues encountered on supported configurations such as Aurora. We recognize the importance of tools like xpu-smi for accessing counter data, and we understand the concerns about relying on previous versions. Continued support for operational tooling is essential, and we encourage users to report any issues so they can be addressed within the scope of ongoing support.

[servesh]

@saik-intel Thanks! It will boost user confidence to keep this tool active and being supported with new releases for PVC GPUs. We have 60000 of them, so hopefully that shows the size of your target userbase. 😉

[eero-t]

Thanks for digging. I'm not really sure why the mix in behavior. Is there CMake version difference between the different CI systems for each OS?

The 6 XPUM builder containers build a specific CMake version, instead of using distro version, so that should not matter:

$ grep CMAKE_INSTALLER_URL= builder/* | cut -d/ -f6 | uniq -c
6  v3.20

However, Debian 10 (Buster) is the release before current Debian "oldoldstable", so it's as old as the RHEL, and those have much older toolchains than Ubuntu 22.04 / 24.04.

Their default GCC versions are: GCC4.8 [1] on RHEL7, GCC7.5 on SLES15, GCC8 on Debian10 & RHEL8, GCC11 on RHEL9 & Ubuntu 22.04, and GCC13 on Ubuntu 24.04.

As RHEL9 & Ubuntu 22.04 are using same GCC11 version, that does not quite explain only RHEL having issue though...

Comparing symbols in RHEL & 22.04 libxpum, they have otherwise exactly the same spdlog symbols except for the 2 extra undefined ones on RHEL.

However, that's not the only difference. RHEL one includes these 2 additional symbols (inline function & non-static function):

  3514: 000000000039d060  1264 OBJECT  <OS specific>: 10 DEFAULT   13 nlohmann::detail::dtoa_impl::get_cached_power_for_binary_exponent(int)::kCachedPowers
  1607: 00000000001595a0   169 FUNC    GLOBAL DEFAULT   11 xpum::teeInitAndConnectByPath(_TEEHANDLE*, uuid_le const*, char const*)

And hundreds of duplicate symbols belonging to different standard libs (GLIBC: 182, GLIBCXX: 138, CXXABI: 27, std: 8, GCC: 2). Almost like things would have been linked multiple times, and on one distro that would be cleaned up, but not on another one...

=> Maybe these distros default to different linker flags (e.g. --as-needed)?

@servesh could you attach output of gcc -dumpspecs of your SLES system here, so I can compare its linker options against what Ubuntu uses?

---

[1] In case GCC C++ standards support matters, according to: https://gcc.gnu.org/projects/cxx-status.html

• C++14: stable in GCC 5, default in GCC 6, required by XPUM: https://github.com/intel/xpumanager/blob/master/CMakeLists.txt#L36
• C++17: stable in GCC 9, default in GCC 11, mentioned few times in XPUM sources
• C++20: support is experimental

PS. RHEL GCC versions were listed on: https://access.redhat.com/solutions/19458