RuntimeError: CUDA error: an illegal memory access was encountered

[HengyiWang]

Hi Steven,

When I try to integrate your treefilterv2 into my network, it causes the RuntimeError: CUDA error: an illegal memory access was encountered:

File "TfTest.py", line 238, in
main()
File "TfTest.py", line 212, in main
eval_loss, pred_l, attn_l = eval(ms, m4_all, r3e_all, r2e_all, Ms_all, scale_space, num_frames, device1)
File "TfTest.py", line 117, in eval
out, attn = model.forward(m4_in, r3e_in, r2e_in, mask, scale_space, n_scale=True)
File "/homes/hw002/Test/testloader/Tree.py", line 269, in forward
return self.nscale_forward(inputs, r3e, r2e, gts, scales)
File "/homes/hw002/Test/testloader/Tree.py", line 185, in nscale_forward
p, feats = self.Decoder(inputs[str(s)], r3e[str(s)], r2e[str(s)]) # .item()
File "/homes/hw002/.conda/envs/pytorch36/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in call_impl
result = self.forward(*input, **kwargs)
File "/homes/hw002/Test/testloader/Tree.py", line 140, in forward
m3 = self.tf2(m3, r2)
File "/homes/hw002/.conda/envs/pytorch36/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/homes/hw002/LearnableTreeFilterV2/cvpods/layers/tree_filter_v2.py", line 78, in forward
feature = self.tree_filter_layer(feature, embed, tree, guide_embed, beta)
File "/homes/hw002/.conda/envs/pytorch36/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/homes/hw002/LearnableTreeFilterV2/cvpods/layers/tree_filter_core.py", line 235, in forward
sorted_parent, sorted_child)
File "/homes/hw002/LearnableTreeFilterV2/cvpods/layers/tree_filter_core.py", line 47, in forward
_C.tree_filter_refine_forward(feature_in, edge_weight, self_weight, sorted_index, sorted_parent, sorted_child)
RuntimeError: CUDA error: an illegal memory access was encountered

It is wired since this happens during my test time, if I use only 1 treefilter layer, my network will stop at frame 42 of the test sequence and if I use 2 treefilters, it will stop at the frame 21 of the test sequence.

Could you help me to find out the reason why this happens?
Thanks a lot for your help, I wish you all the best!