[9.0] Subprocess: use psutil for killing processes

[fstagni]

This is a rewrite of few functions in subprocess that need very careful evaluation.

For reasons not understood (at least not by me) the logic of killing a pid and its sub-processes is not completing (not always, at least). I blame the very old logic used there, but of course I might be wrong. In any case this is a rewriting using psutil.

I had to change one test in Test_JobWrapper. I have not fully understood why, but I do not see how the previous code could work. cc @aldbr

BEGINRELEASENOTES

CHANGE: Subprocess: use psutil for killing processes

ENDRELEASENOTES

[aldbr]

Actually here you could directly call psutil.wait_procs as it was done before, no?
This is shown in the psutil documentation (https://psutil.readthedocs.io/en/latest/index.html#psutil.wait_procs) and this was done like this before (I think the else block was correct but it was just not used by the JobWrapper.

[aldbr]

Well actually, forget about my other comment: any reason for not applying this comment?

[fstagni]

The 2 are not equivalent?

[aldbr]

Sorry I should have provided further details.
Here you sequentially terminate and wait for each process to terminate. If there are n processes, and if they do not terminate, you are going to wait for n * 60s from what I understand.

Whereas you could reduce the waiting time from o(n) to o(1) by doing:

for p in children:
    p.terminate()
psutil.wait_procs(children, timeout=60)

This is what it's currently done and what is recommended in the documentation.
(The problem with the current code is the first bloc after the if pgid != os.getpgrp():, because we terminate the processes but we don't kill them I think, the else bloc looks fine to me).