Address a deadlock issue in multi-GPU scenario

[mangupta]

• Fixes SWDEV-219322

[jeffdaily]

The error message for the failing tests is curious. error while loading shared libraries: libhsa-runtime64.so.1: cannot open shared object file: No such file or directory

[jeffdaily]

I kicked off a unit test run on my local system to see if I could reproduce the failures.

[srinivamd]

Some comment would help. It seems calling 'wait()' is equivalent to holding qmutex and wait_no_lock() which is what is lines 4185 below after locking rocrQueuesMutex. Does this fix imply that rocrQueuesMutex should NOT be held before qmutex as it may cause deadlock? If so, shouldn't line 4185 locking of qmutex be removed?
Or, could moving acquiring of rocrQueuesMutex to just before calling removeRocrQueue() help?

[jeffdaily]

wait_no_lock() could potentially call EnqueueMarkerNoLock(). If the HSAQueue does not hold a rocr queue, it will end up calling createOrstealRocrQueue() and lock rocrQueuesMutex.

Moving the lock on rocrQueuesMutex until just before line 4206 might work, too.

[scchan]

The error message for the failing tests is curious. error while loading shared libraries: libhsa-runtime64.so.1: cannot open shared object file: No such file or directory

That's a known problem that @david-salinas will address

[scchan]

@jeffdaily do we still need an explicit wait if we clear the asyncops vector?

[jeffdaily]

@jeffdaily do we still need an explicit wait if we clear the asyncops vector?

Isn't the wait is necessary due to HCC_OPT_FLUSH?