This repository contains the Open Source Software (OSS) components of PCIe-based AI accelerators (support for PCIe-based AI accelerators such as GPUs, FPGAs, ASICs from NVIDIA, AMD, Xilinx, Intel, IBM, Qualcomm, Samsung, Huawei and more) for both inference and training. Democratizing access to multimodal large language models, proving that accessing high-performance LLM inference is achievable beyond centralized data centers and on the hardware people already own or plan to own thereby maximizing hardware efficiency and increasing accessibility. These open source software components are a subset of the General Availability (GA) release with some extensions and bug-fixes.
-
Set Up the PCIe AI Accelerator Install the required drivers and SDKs for your PCIe accelerator. For example: NVIDIA GPUs: Install CUDA and cuDNN. Intel accelerators: Install OpenVINO Toolkit. Xilinx FPGAs: Install Vitis AI runtime. Ensure the PCIe device is visible via tools like lspci (Linux) or equivalent commands.
-
Modify Training Code Use the appropriate deep learning framework and ensure device targeting is set to the PCIe accelerator. Examples:
-
and 4. code files
-
Use Accelerator-Specific Optimizations For NVIDIA GPUs: Use TensorRT for inference optimization. For Intel FPGAs: Use OpenVINO's optimized inference engine. For Xilinx FPGAs: Use Vitis AI tools for quantization and deployment. For AMD GPUs: Use ROCm libraries and tools for machine learning, computer vision, and mathematical computations.
-
Monitor and Debug Use monitoring tools to ensure efficient usage of the PCIe accelerator: NVIDIA: nvidia-smi Intel: OpenVINO Benchmark Tool Xilinx: Vitis AI Profiler AMD: ROCProfiler and ROCTrace
