Failed to load ZLUDA: Could not find module C:\Program Files\AMD\ROCm\6.2\bin\hiprtc0507.dll

[CutestEchidna]

What happened?

Hello everyone! I am using Onetrainer to train Lora for SDXL. I run other GUIs for neural networks through ZLUDA and ROCM 6.2 (ComfyUI, A1111), but as I noticed, Onetrainer defaults to the version for ROCM 5.7.

So, I edited the ZLUDAInstaller.py file and added a link to the ZLUDA installation for 6.2. Everything installed without any problems, but when I launch Onetrainer, I get the following error

`activating venv E:\OneTrainer\venv
Using Python "E:\OneTrainer\venv\Scripts\python.exe"
Checking Python version...
Python 3.10.11

Warning: Deprecated Python version found. Update to 3.11.0 or newer
Starting UI...
Failed to load ZLUDA: Could not find module 'C:\Program Files\AMD\ROCm\6.2\bin\hiprtc0507.dll' (or one of its dependencies). Try using the full path with constructor syntax.`

When I try to start training, nothing happens, or rather, it says “training,” but there is no progress (it's definitely not compiling, I waited 2 hours).

I have a file called hiprtc0602.dll in the ROCM folder.

In general, I would be very grateful if you could help and tell me which file needs to be corrected so that Onetrainer can use ROCM 6.2.

I apologize in advance if this is a silly question or has already been answered. I am new to this and have not been able to find the answer online.

What did you expect would happen?

Onetrainer launches with ROCM 6.2

Relevant log output

Generate and upload debug_report.log

No response

[O-J1]

Converted from bug report as its a support question not a bug report (as it contains no repro steps nor system info)

[CS1o]

There was already a commit to add Support for ROCm HIP SDK 6.1 and 6.2 and all upcoming versions here:
https://github.com/Nerogar/OneTrainer/pull/422/commits
Maybe you find all necessary changes there.
They didnt merged it sadly.

But there is an outdated fork of OneTrainer from lshqqtiger which supports 6.2 just fine.
Here is the setup guide for it:
https://github.com/CS1o/Stable-Diffusion-Info/wiki/Lora-Trainer-Setup-Guides#amd-onetrainer-with-zluda
It works fine for training.

I really wish the OneTrainer devs would just add the fix from the commit or checkout the fork for AMD Support.

[CutestEchidna]

Thanks, I used this fork before, but a few weeks ago when training a certain model I started getting an error after saving the intermediate result. So I decided that it was a sign to try the official version

[CutestEchidna]

I was able to get Onetrainer up and running on ROCM 6.2, for anyone who might need a solution:

Before running the install.bat file (or if you already installed it just delete the "venv" folder), go to :\OneTrainer\modules\zluda and edit the ZLUDAinstaller.py file

In the HIP TARGETS line, rename 'hiprtc0507.dll' with 'hiprtc0602.dll'

In the urllib.request.urlretrieve line, paste the link to the latest version of ZLUDA from lshqqytiger
ZLUDA

In the same file, replace all references to ROCM 5.7 with 6.2

Here is my version of the ZLUDAinstaller.py file:

import ctypes
import os
import platform
import shutil
import urllib.request
import zipfile

RELEASE = f"rel.{os.environ.get('ZLUDA_HASH', '11cc5844514f93161e0e74387f04e2c537705a82')}"
DLL_MAPPING = {
    'cublas.dll': 'cublas64_11.dll',
    'cusparse.dll': 'cusparse64_11.dll',
    'nvrtc.dll': 'nvrtc64_112_0.dll',
}
HIP_TARGETS = ['rocblas.dll', 'rocsolver.dll', 'hiprtc0602.dll']
ZLUDA_TARGETS = ('nvcuda.dll', 'nvml.dll')


def get_path() -> str:
    return os.path.abspath(os.environ.get('ZLUDA', '.zluda'))


def find_hip_sdk() -> str | None:
    program_files = os.environ.get('ProgramFiles', r'C:\Program Files')  # noqa: SIM112
    hip_path_default = rf'{program_files}\AMD\ROCm\6.2'
    if not os.path.exists(hip_path_default):
        hip_path_default = None
    return os.environ.get('HIP_PATH', hip_path_default)


def install(zluda_path: os.PathLike) -> None:
    if os.path.exists(zluda_path):
        return

    if platform.system() != 'Windows': # TODO
        return

    urllib.request.urlretrieve(f'https://github.com/lshqqytiger/ZLUDA/releases/download/rel.5e717459179dc272b7d7d23391f0fad66c7459cf/ZLUDA-windows-rocm6-amd64.zip', '_zluda')
    with zipfile.ZipFile('_zluda', 'r') as archive:
        infos = archive.infolist()
        for info in infos:
            if not info.is_dir():
                info.filename = os.path.basename(info.filename)
                archive.extract(info, '.zluda')
    os.remove('_zluda')


def make_copy(zluda_path: os.PathLike) -> None:
    for k, v in DLL_MAPPING.items():
        if not os.path.exists(os.path.join(zluda_path, v)):
            try:
                os.link(os.path.join(zluda_path, k), os.path.join(zluda_path, v))
            except Exception:
                shutil.copyfile(os.path.join(zluda_path, k), os.path.join(zluda_path, v))


def load(zluda_path: os.PathLike) -> None:
    hip_path = find_hip_sdk()
    if hip_path is None:
        raise RuntimeError('Could not find AMD HIP SDK, please install it from https://www.amd.com/en/developer/resources/rocm-hub/hip-sdk.html')
    for v in HIP_TARGETS:
        ctypes.windll.LoadLibrary(os.path.join(hip_path, 'bin', v))
    for v in ZLUDA_TARGETS:
        ctypes.windll.LoadLibrary(os.path.join(zluda_path, v))
    for v in DLL_MAPPING.values():
        ctypes.windll.LoadLibrary(os.path.join(zluda_path, v))

You may also need to create copies of the files and change them as in the screenshot

[iDeker]

[CutestEchidna]

you need to uninstall torch in venv folder, and then instal another version

i was able to launch it with this version:
pip install torch==2.3.0+cu118 torchvision==0.18.0 torchaudio==2.3.0 --extra-index-url https://download.pytorch.org/whl/cu118

then you might have a problem with "numpy"
pip install --force-reinstall -v "numpy==1.25.2"

[CS1o]

Update: Lshqqtiger updated the OneTrainer Zluda fork two weeks ago. Works again!