Legate fails to detect network when launched with aprun

[elliottslaughter]

One of our machines uses PBS/Torque as the job scheduler and aprun to launch jobs. In this environment, based on the behavior of our scripts, it appears that Legate is failing to detect that we're in a multi-node run. Thus the script executes multiple discrete copies, resulting in various bad behavior (slow, out of memory, filesystem races, etc.).

@manopapad pointed me to the code that attempts to detect the network, and we can clearly see aprun is not supported:

legate/src/cpp/legate/runtime/detail/argument_parsing/util.cc

Lines 29 to 33 in 60b41a2

	unsigned int num_ranks()
	{
	constexpr EnvironmentVariable<std::uint32_t> OMPI_COMM_WORLD_SIZE{"OMPI_COMM_WORLD_SIZE"};
	constexpr EnvironmentVariable<std::uint32_t> MV2_COMM_WORLD_SIZE{"MV2_COMM_WORLD_SIZE"};
	constexpr EnvironmentVariable<std::uint32_t> SLURM_NTASKS{"SLURM_NTASKS"};

So this draws two larger questions.

1. First, we need a way to initialize when the network is not one of these. A manual flag or some sort of hint might be an option.
2. Second, we should have a standardized documentation that covers: (a) how to debug whether the network is working, (b) how to run with an unsupported launcher, (c) advice for checking that all the rank/core/GPU/etc. bindings are working when you're in this scenario.

[elliottslaughter]

As a potential solution I suggested initializing MPI instead of grepping for known environment variables. This has downsides, and I wouldn't want it to be automatic, but I feel like it might be the best general solution. A lot of these odd systems with quirky job launchers have MPI, and it's often the only thing that works reliably.

You could put it behind a flag, e.g.: --launcher=mpi and then that tells Legate, "attempt to initialize MPI, and assume the user has already launched the job via whatever job scheduler is on the system".

[elliottslaughter]

I've been doing some research on aprun specifically. This won't generalize to other launchers but may help unblock our immediate issues in the short term.

A simple 2-node, 2-rank run found the following environment variables:

PMI_RANK=0
PMI_SIZE=2
PMI_LOCAL_RANK=0
PMI_LOCAL_SIZE=1

PMI is a standardized job launching framework, so I'm guessing these are documented somewhere.

PALS_NODEID=0
PALS_RANKID=0
PALS_DEPTH=1
PALS_LOCAL_RANKID=0
PALS_LOCAL_SIZE=1

These ones seem to be specific to the HPE software environment. They are documented here: https://support.hpe.com/hpesc/public/docDisplay?docId=dp00005984en_us&page=user/pbs/PALS.html&docLocale=en_US

I guess there is a choice to be made about how much to generalize vs specialize to particular job launchers. In the short term, adding support for some or all of these variables might resolve our issues. In the longer term, I fear that we'll repeat this again with some new machine down the line if we don't resolve this more properly.

[manopapad]

@elliottslaughter I pushed some updates to handle aprun, you can see the changes here 1ec9333. They should become available under -c legate-nightly (please note new channel) as legate=25.11.00.dev12. Please try it out and let me know if it works for you. We can open follow-up issues for your longer-term asks above.

[elliottslaughter]

I can't test due to issues with realm-gex-wrapper availability for the new Legate, as explained in our private channel. Let me know if you want me to replicate the information here.

Correction: it's not actually the gex wrapper, it's a regression in Legate related to checking for the number of GPUs in order to calculate a conservative obcount. At any rate, this is currently blocked.

[manopapad]

@elliottslaughter the regression should be fixed on the nightly packages if you want to try again.