Switch to native ParaView MPI parallelization with pvserver

[unkaktus]

Currently, we spawn multiple subprocesses of pvpython which process their block of frames. It requires the management code (multiprocessing pool + subprocess), and it is not super scalable.
I am convinced that the parallelization implemented in pvserver is faster, but at least one would not have to maintain separate code for job management.

In this case, the architecture would look like this:

• Starting pvserver with mpirun (e.g. on HPC cluster)
• Connecting from gwpv to pvserver via TCP using Connect() method
• Rendering frames sequentially, and pvserver takes care (mostly) of parallelizing the rendering

It would require having a parallel-aware data source, which I believe is not hard to implement. ParaView says in the documentation, that with the D3 filter even non-parallel-aware data sources can benefit from using pvserver.

Here is the documentation page: https://docs.paraview.org/en/latest/ReferenceManual/parallelDataVisualization.html#paraview-architecture

[nilsvu]

Do I understand correctly that this means rendering one frame at a time, and parallelizing that? Or does pvserver also support rendering multiple frames in parallel?

[unkaktus]

I don't know for sure, but I believe that pvserver can also handle multiple clients concurrently. That would mean that we can render multiple frames in parallel while utilizing parallelization of the render inside the frame. And leaving the subprocess code as-is, but rendering the frames using pvserver.

Is that often useful to render frames concurrently?

[nilsvu]

Yes it is quite important to render multiple frames in parallel, since that's what is often called "embarrassingly parallel" -- the frames don't depend on each other, so if the computational resources are unconstrained they parallelize perfectly. On the other hand, parallelization within each frame is limited by how well it is implemented. Doing both forms of parallelization is only useful when you have more cores available than you have frames to render, or when you're constrained by other resources like RAM or GPU memory. (This may very well be the case depending on the system.)