Re. multiprocessing code

[ShahRutav]

Hi,

Thanks for open-sourcing your work and releasing the codebase!

I was wondering if this torch.cuda.synchronize() (Here) plays any role? Upon removing it, I got some visible training performance improvement.

[Max-Fu]

Hi,

This synchronization ensures that the metrics/losses are correctly synchronized. I referenced MAE when implementing the distributed training. I am not sure if removing this can guarantee the correctness of the model's gradient update, but it is worth a try.

[ShahRutav]

I see. Thanks for the pointer!
I noticed that the gradient updates are performed before the synchronization call (here). The tensor board logging already has all_reduce_mean (here). I will give it a try! Thanks

[Max-Fu]

Let me know if it works! I will keep this issue open :)