Interoperability with other programming models

[minsii]

Goal: define a new interoperability section

Context: Appendix D has been removed because the contents are outdated. We want to make a new interoperability section that defines the interoperability of OpenSHMEM with other programming models such as MPI, UPC, CAF, in order to help portable hybrid OpenSHMEM + X programs.

The section is planed to cover:

• Standardization of features / useful APIs for portable hybrid program
• Clarification of undefined behavior caused by mixed use of different models
• Advice to users about the ways to guarantee program portability
• Advice to runtime developers about the ways to optimize performance

A PR is created for MPI interoperability. See minsii#1

[minsii]

Updates in 20180904's Thread WG call:

Slides:
SHMEM_Interoperability_20180904.pptx

Summary of feedbacks (mainly for progress property SHMEM_PROGRESS_UNIFIED|SEPARATE):

• Do we also need define the progress model for SHMEM ?
• Is the progress property only used for performance optimization ?
• The survey (see slides) covers only SHMEM + MPI, the other programming models (e.g., UPC, CAF) may not be progressed in a consistent way. Maybe we should rename the property values to SHMEM_PROGRESSES_MPI|MPI_PROGRESSES_SHMEM

[spophale]

Notes form WG Meeting Tue Sep 4th:

• Atomicity guarantees between MPI and SHMEM operations will not be part of the Spec - user responsibility.
• Removed section on data allocation by one programming model and used by the other.
• Open Question : How to interpret SHMEM_PROGRESS_UNIFIED and SHMEM_PROGRESS_SEPARATE
• Current focus OSH + MPI, other programming models will be added later when we have more information.

[bcernohous]

Try to explain myself after a bit more thought.

Open Question : How to interpret SHMEM_PROGRESS_UNIFIED and SHMEM_PROGRESS_SEPARATE

I think SHMEM queries SHMEM attributes, not MPI/UPC/Fortran.

I think there are 3 attributes (+detail/sub attributes if desired).

1. SHMEM make progress on xxx :

• SHMEM_PROGRESS_ALL

  or individually

  • SHMEM_PROGRESS_MPI
  • SHMEM_PROGRESS_UPC
    .. to whatever level of detail

2. SHMEM needs other programming models, or the user applications, to make progress:

• SHMEM_REQUIRES_PROGRESS (or SHMEM_PROGRESS_MANUAL)

3. SHMEM always makes progress (no dependency on other programming models or application)

• SHMEM_PROGRESS_AUTO

For 2), I don't think we can define if MPI progress (MPI_TEST) make progress on SHMEM, only that SHMEM needs progress.

e.g.

Cray XC supports Cray MPI, Intel MPI, OpenMPI (and sometime MVAPICH).

I don't see how Cray SHMEM defines 'unified progress' or anything about MPI, only that SHMEM itself needs (or doesn't need) progress from other programming models/application.

(edit) and I don't see SHMEM defining whether or not MPI makes auto progress or required manual calls to MPI_Test/wait/.

[anshumang]

From section 4.1 [Progress of OpenSHMEM Operations] in spec (v 1.4), one interpretation is that progress without API call is required. If so, SHMEM_REQUIRES_PROGRESS (or SHMEM_PROGRESS_MANUAL) is not allowed by the spec and SHMEM_PROGRESS_AUTO is guaranteed. If the interpretation of the spec allows relaxed progress, why not query that property outside the scope of interop?

Regarding SHMEM_PROGRESS_ALL (or SHMEM_PROGRESS_X) : If an application is written to rely on this property, isn't the programmer also assuming a specific OpenSHMEM implementation? If so, why not leave this as a property that can only be queried by a vendor-specific shmemx_ API?

[jeffhammond]

OSHMPI has unfinished support for MPMD and it is straightforward to extend that to DPM (dynamic process management). There isn't any serious impediment to doing it, but I had no motivation other than curiosity. If somebody in the OpenSHMEM community wants to know how this works, I might be able to implement it for testing.

[jeffhammond]

@minsii Regarding progress, this is rather complicated and perhaps not amenable to a simple boolean descriptor. For example, OpenSHMEM might progress MPI RMA but not two-sided. How do we express that?

[minsii]

@bcernohous Thanks a lot for the detailed explanation. However, it might be inappropriate if we consider the progress property only from SHMEM. This is an information we want to have only for hybrid SHMEM + X program. As @anshumang mentioned, the OpenSHMEM spec actually guanratees that the progress of SHMEM can only be SHMEM_PROGRESS_AUTO.

@anshumang Thanks for pointing out the progress spec. After reading the spec again, I agree with your interpretation. I.e., a SHMEM implementation is required to make progress even when the program does not make API call. It is a stronger progress guarantee than that in MPI. Then the info we really need to query is "whether SHMEM makes progress for X".

Regarding shmemx suggestion, I do not think the query API has to be vendor-specific, however, the vendor can define different values for the property (e.g., SHMEM_PROGRESS_MPI|SHMEM_PROGRESS_UPC, or none)

Another issue I am reading from Bob's comment is that, although an runtime implementation may support both MPI and SHMEM, the user may link with another implementation of one of the models at runtime (or preload at execution time). E.g., MVAPICH gives both MPI and SHMEM and supports SHMEM_PROGRESS_MPI, but the user may link (or preload) another binary of MPI which breaks SHMEM_PROGRESS_MPI. Thus, the properties cannot be statically set.

I guess the runtime may figure out some properties (e.g., progress, pe mapping) after shmem_init & MPI_Init, but the ordering of init calls has to be exposed in a different way.

[minsii]

@jeffhammond I do not think OpenSHMEM itself can support dynamic progress because shmem_init is required to be a collective call over all PEs(section 9.1.1 in spec 1.4).

Example of unsupported dynamic process:

P0                                      
shmem_init();
MPI_Comm_spawn(P1) or fork(P1); 

P1
shmem_init();

However, it might be allowed if some PEs are forked separately (e.g., through fork() or MPI_comm_spawn) but all the PEs call shmem_init collectively. This is being discussed in #223

Example that might be supported:

P0                                      
MPI_Comm_spawn(P1) or fork(P1); 
shmem_init();

P1
shmem_init();

[bcernohous]

the OpenSHMEM spec actually guanratees that the progress of SHMEM can only be SHMEM_PROGRESS_AUTO.

That was my original understanding but from today's call it seemed like that wasn't universal.

So I think the only question is whether shmem makes progress for XXXX. And that's a hard thing to answer. As you've suggested, it's not static.

I'd guess it's very limited (my vendor shmem + my vendor mpi) so it a query really useful vs knowing which libraries you linked with? Header versions?

[jeffhammond]

Yes, @minsii, I was thinking of a case where an MPI application spawned islands, each of which would call shmem_init and act independent of one another. It is not possible to have symmetric variables across the original and spawned communicators with the current symmetric heap API, although one can of course do it in MPI-3 RMA.

[jeffhammond]

@bcernohous Indeed, I agree that SHMEM_PROGRESS_AUTO is required by any high-quality implementation and that we shouldn't spend a lot of time worrying about other ones.

[minsii]

@bcernohous I think the runtime can figure out some properties after all hybrid models are initialized in program. The supported properties include:

• progress: e.g., SHMEM_PROGRESS_MPI|SHMEM_PROGRESS_UPC, or none
• pe-mapping: e.g., FORTRAN_IMAGES_ARE_PES|MPI_WORLD_RANKS_ARE_PES, MPI_WORLD_RANKS_ARE_PES, or none

The assumption is that, only unified runtime (e.g., one who implements both SHMEM and X) wants to return not-none value. Once both SHMEM and X(es) are initialized, the runtime will know whether the internal X is being used. E.g., the vendor runtime developer can simply set a global variable is_vendor_X_initiated=1 when X_init() is called. Thus, the user can query valid properties after all initialization calls. I think it is still useful.

The only property we cannot support is initialization-ordering. Because the user program needs this info before any initialization call, but the runtime cannot figure out correct value until initialized.

[minsii]

Slides used for threads WG meeting on September 18:
SHMEM_Interoperability_20180918.pptx

Comments from today's meeting:

• Dynamic loaded external library: It helps the developer to return query properties if the query API can be called only after initialized both SHMEM and X model(s). The runtime may choose to support query before initialization calls (e.g., return the ordering of initialization property) if the runtime could detect dynamic loaded external library in a different way.
• Utilization of the query API: It is unclear how the query API can help user program's portability. Maybe we can prepare code examples for it.
• Dynamic process support: We could also extend SHMEM to support inter-group communication (e.g., across two MPI inter-communicators). This can be left as a future work.

[minsii]

@naveen-rn and I have discussed offline about the query API. Here is a summary of the comments:

• We want to introduce generic interoperability features in OpenSHMEM spec. If the features rely on some internal details of a SHMEM implementation, it does not seem to be generic. E.g., initialization ordering and pe-rank mapping are implementation-specific details.
• When SHMEM / MPI libraries are dynamically loaded, the above info can be incorrect unless the SHMEM implementation chooses to workaround it. In the worst case, a SHMEM implementation may always return unsupported.

Alternative options:

1. Instead of querying initialization-ordering, we could introduce shmem_initialized and shmem_finalized similar to MPI_Initialized and MPI_Finalized. We suggest the hybrid program always query runtime status before making the init/finalize call.
2. For pe-rank mapping, the user program should always query ranks/pes by using the standard APIs and maintain the mapping in the user program.

I especially like the idea of shmem_initialized|finalized(). It seems to be a more generic API than querying the ordering. It also resolves the initialization problem.

For pe-rank mapping, the query feature could help user program only when the SHMEM implementation supports it thus a portable program has to still write the manual version. Besides, it saves cost only at program init time (e.g., no info exchange). The benefit of this feature seems to be limited and likely increases code complexity in SHMEM implementation. Thus, I would suggest that we discard the proposal of pe-mapping in the initial version and go with option-2.

[minsii]

A pull request has been created to prepare the spec document draft: minsii#1