Installation

Prerequisites

• OpenMPI/MPICH2
• Python>=3.7
  • Cython
  • numpy
  • mpi4py
  • cupy>=12 (optional)
  • torch>=2 (optional)
• QUDA
  • CMake>=3.18
  • Git
  • GCC>=7 / Clang>=5 (supports C++17)
  • CUDA-Toolkit>=11

For detailed instructions to set the environment, refer to https://github.com/CLQCD/PyQUDA/wiki/Environment.

Build and install QUDA

git clone https://github.com/lattice/quda.git
mkdir -p quda/build
pushd quda/build
export GPU_ARCH=sm_60
cmake -DCMAKE_BUILD_TYPE=RELEASE \
    -DQUDA_GPU_ARCH=${GPU_ARCH} -DQUDA_MPI=ON \
    -DQUDA_COVDEV=ON -DQUDA_MULTIGRID=ON \
    -DQUDA_DIRAC_DEFAULT_OFF=ON \
    -DQUDA_DIRAC_WILSON=ON -DQUDA_DIRAC_CLOVER=ON \
    -DQUDA_DIRAC_STAGGERED=ON -DQUDA_DIRAC_LAPLACE=ON \
    -DQUDA_CLOVER_DYNAMIC=OFF -DQUDA_CLOVER_RECONSTRUCT=OFF
cmake --build . -j32 && cmake --install .
popd

The GPU_ARCH environ should be set to the architecture of your device. The default installation path of QUDA will be /path/to/quda/build/usqcd. We will use this path then.

Install CuPy or PyTorch

PyQUDA requires cupy or torch to handle the data on GPU memory.

Choose ONE OF the commands below depending on your CUDA toolkit version to install cupy.

python3 -m pip install "cupy>=12" # build from source, not recommend
python3 -m pip install "cupy-cuda110>=12" # for CUDA v11.0
python3 -m pip install "cupy-cuda111>=12" # for CUDA v11.1
python3 -m pip install "cupy-cuda11x>=12" # for CUDA v11.2 ~ v11.8
python3 -m pip install "cupy-cuda12x>=12" # for CUDA v12.x

Or use the command blow to install torch.

python3 -m pip install "torch>=2"

Install PyQUDA

git clone --recursive https://github.com/CLQCD/PyQUDA.git
pushd PyQUDA
export QUDA_PATH=/path/to/quda/build/usqcd
python3 -m pip install .
popd

Here we use an environment variable QUDA_PATH to tell PyQUDA where to find the libquda.so.

Test the installation

Chroma is needed to generate the reference files used by test scripts.

A precompiled chroma executable for most Linux distros is included in the repository, and you need to fetch it with git-lfs. Note that git<2 seems not to work well with git-lfs.

git lfs install
git lfs pull

The reference files used by test scripts should be generated by running Chroma with test.*.ini.xml. Then the corresponding Python script test.*.py will check the QUDA result with Chroma.

tests/bin/chroma -i tests/test.clover.ini.xml
python3 tests/test.clover.py

Or just run the Python script which automatically calls Chroma.

python3 tests/test.clover.chroma.py

You may also want to initialize pyquda with command line arguments instead of hardcoding init parameters in scripts.

python3 -m pyquda tests/test.clover.cli.py --lattice 4 4 4 8 --t-boundary -1 --anisotropy 2.593684210526316 --backend cupy

The command line with arguments above is written in the shebang line.

./tests/test.clover.cli.py