[Question]: the Necessity of NVIDIA Driver Config NVreg_EnableStreamMemOPs and PeerMappingOverride

[michaelchen1996]

Question

I have a question regarding the necessity of specific NVIDIA configuration parameters.

NVreg_EnableStreamMemOPs=1 and NVreg_RegistryDwords="PeerMappingOverride=1;"

In our tests, after removing these two parameters from initramfs, we still observe the following outputs:

NIC buffer will be on GPU memory
NIC handler will be GPU

From my observations, the version of cuStreamWaitValue64 used in nvshmem is 11070_v2, as indicated by:

LOAD_SYM(table, cuStreamWriteValue64, 11070, _v2, 1);
LOAD_SYM(table, cuStreamWaitValue64, 11070, _v2, 1);

According to the NVIDIA CUDA Driver API documentation, the NVreg_EnableStreamMemOPs=1 parameter is only required for v1 of these functions.

Additionally, does the PeerMappingOverride parameter only take effect in the case of non-root users?

So, are the parameters NVreg_EnableStreamMemOPs=1 and NVreg_RegistryDwords="PeerMappingOverride=1;" still necessary for GPU to handle NIC?

What impact (if any) would there be without these parameters—for instance, on performance?

Thank you!

[benhg]

Starting with 3.4.5, you do not need them anymore, for CPU-assisted IBGDA. You will still need them for fully-device-assisted IBGDA. See the release notes: https://docs.nvidia.com/nvshmem/release-notes-install-guide/release-notes/release-3405.html#key-features-and-enhancements

[michaelchen1996]

In our tests, after removing these two parameters from initramfs, we still observe the following outputs:
NIC buffer will be on GPU memory
NIC handler will be GPU

Does the output NIC handler will be GPU imply that it's fully-device-assisted IBGDA?