CUDA_ERROR_NO_DEVICE: no CUDA capable device is detected

[MSajek]

Hi @stefanprodic,
When I run
python ${EXE}/SWARM_read_level.py -m ${MOD} --sam ${SAM} --fasta ${FASTA} --raw ${BLOW5} -o ${OUTPUT_DIR}/${OUTPUT_FILE}
I have the following message in stderr:

2025-11-24 10:25:19.180331: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0. 2025-11-24 10:25:19.640229: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2025-11-24 10:25:19.640291: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2025-11-24 10:25:19.717590: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2025-11-24 10:25:19.869597: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2025-11-24 10:25:23.825152: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2025-11-24 10:25:31.913383: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:274] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected 2025-11-24 10:25:31.913414: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:129] retrieving CUDA diagnostic information for host: gpu07 2025-11-24 10:25:31.913419: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:136] hostname: gpu07 2025-11-24 10:25:31.913496: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:159] libcuda reported version is: 570.86.15 2025-11-24 10:25:31.913512: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:163] kernel reported version is: 570.86.15 2025-11-24 10:25:31.913517: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:241] kernel version seems to match DSO: 570.86.15

I used CUDA 11.8.
The output file is created, I assume that using CPU instead of GPU, right?
What can I do to make GPU visible?
Thanks in advance.
Best,
Marcin

[stefanprodic]

Hi Marcin,

It seems like your tensorflow is not configured for GPU, in case you are on a HPC which does not provide tensorflow module configured for your GPU it is the best and easiest to ask the support team to install and configure it system-wide. If that is an option than you don't have to read the rest :)

If you are running it on your own machine and are struggling to configure tensorflow, there is a workaround to avoid compilation/configuration by pulling nvidia tensorflow image from the nvidia website which has all drivers set up, and install pysam into that container. You would need a linux device with root access to install docker. Here is what worked for me using ubuntu:

On local system with root access follow docker installation for your OS as per:
https://docs.docker.com/engine/install/

# Pull the tensorflow container
mkdir tf_pysam && cd tf_pysam
sudo docker pull nvcr.io/nvidia/tensorflow:24.01-tf2-py3

# setup Dockerfile to install pysam into the tf container  
echo "FROM nvcr.io/nvidia/tensorflow:24.01-tf2-py3" > Dockerfile && echo "RUN pip install pysam==0.22.1" >> Dockerfile

sudo docker build -t tf_pysam:latest .

You may be able to run SWARM straight from this image if your GPU is on the system with root access and have docker configured for nvidia-container-toolkit (could not test):

sudo docker run \
    -v /PATH/TO/tf_pysam:/workspace \
    -w /workspace \
    tf_pysam:latest \
    python SWARM_read_level.py -m pU --sam $SAM --fasta $FASTA --raw $RAW -o $OUT

I was not able to test running GPUs locally as my PC doesn't have an nvidia GPU and HPC has no root access (no docker) so I used singularity (version 3.11.3):

sudo docker save -o tf_pysam.tar tf_pysam:latest       # this will produce tf_pysam.tar
sudo chown username:username /PATH/tf_pysam/tf_pysam.tar # may need to change ownership from root to user
scp tf_pysam.tar user@server:/PATH/TO/tf_pysam/ # upload tar to your server

Now on the HPC server build sif file to run with singularity:

cd /PATH/TO/tf_pysam/ 
module load singularity
export SINGULARITY_CACHEDIR=/PATH/TO/tf_pysam/
singularity build tensorflow_24.01-tf2-py3-pysam.sif docker-archive://tf_pysam.tar

You can run SWARM on GPU with singularity:

module load singularity
singularity exec --nv /PATH/TO/tf_pysam/tensorflow_24.01-tf2-py3-pysam.sif python SWARM_read_level.py -m pU --sam $SAM --fasta $FASTA --raw $RAW -o $OUT

This worked using a V100 GPU and runs as fast as with system-wide tensorflow installation.

Let me know if you are running SWARM on HPC or locally and if you were able to run it via docker/singularity.

Best,
Stefan

[MSajek]

Hi Stefan,
Thank you very much. It was run on HPC. I will ask support to install proper tensorflow and in case of any issues try your singularity approach.
Btw., did you check on any data if predictions generated on cpu vs gpu are the same/similar or show some substantial differences?

Best,
Marcin

[stefanprodic]

Hi Marcin,

That's good to know, hope they can install it. Please note that the latest version of tensorflow that I tested running SWARM was 2.15.0.

I haven't used CPU inference a lot but every time I did (on small subsets of reads) I got identical results compared to GPU.

Best,
Stefan