TorchRuntimeError: Dynamo failed to run FX node with fake tensors

[umdsgy]

Hi there, I am running TabDPT regression from Tabarena using a tabular dataset and got the following error:

torch._dynamo.exc.TorchRuntimeError: Dynamo failed to run FX node with fake tensors: call_function (*(FakeTensor(..., device='cuda:0', size=(1, 8, 1600, 96)), FakeTensor(..., device='cuda:0', size=(1, 8, 1280, 96)), FakeTensor(..., device='cuda:0', size=(1, 8, 1280, 96), dtype=torch.bfloat16)), **{}): got RuntimeError('No available kernel. Aborting execution.')
from user code:
File "C:\Users\GS\miniconda3\envs\ag\Lib\site-packages\tabdpt\utils.py", line 29, in wrapper
return func(self, *args, **kwargs)
File "C:\Users\GS\miniconda3\envs\ag\Lib\site-packages\tabdpt\model.py", line 74, in forward
src = layer(src, eval_pos)
File "C:\Users\GS\miniconda3\envs\ag\Lib\site-packages\tabdpt\model.py", line 137, in forward
attn = F.scaled_dot_product_attention(q, k, v).transpose(1, 2)
Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
[W302 10:10:39.000000000 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator ())

I am running it on Windows and torch 2.9.

torch.__version__ '2.9.1' torch.version.cuda '12.8'

Here is my gpu info:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 591.86 Driver Version: 591.86 CUDA Version: 13.1 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Driver-Model | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 5080 WDDM | 00000000:01:00.0 On | N/A |
| 30% 41C P1 120W / 360W | 8093MiB / 16303MiB | 100% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

Any clue how to address this?

[shuvo14051]

Hey, I ran into the same issue even after installing via pip install tabdpt. What worked for me was modifying the model.py file inside the site-packages.
I replaced this line (you can find at the bottom):

attn = F.scaled_dot_product_attention(q, k, v).transpose(1, 2)

with this

with torch.backends.cuda.sdp_kernel(enable_flash=False, enable_mem_efficient=False, enable_math=True):
    attn = F.scaled_dot_product_attention(q, k, v).transpose(1, 2)

After this change, it ran without errors. Might be worth trying on your end.

[umdsgy]

@shuvo14051 Thanks for your solution. It also works for me with an additional setting torch._dynamo.config.suppress_errors = True since I dont have a compatible triton package.