Dynamic allocation of GASPI segments

[krzikalla]

I have a library, where I can't precalculate the memory footpring in a GASPI segment. Thus the idea is to create segments on demand. Does the following code work in principle:

void* GetMemory(size_t size)   // collective called by all members of group
{
  void* p = Allocate(size, seg_id);
  if (p == 0)
  {
    gaspi_segment_avail_local(&seg_id);  // from the extension
    gaspi_segment_alloc(seg_id, size, GASPI_MEM_UNINITIALIZED);
    gaspi_segment_ptr(seg_id, &p);
    for (auto i : group)
      gaspi_segment_register(seg_id, i, GASPI_BLOCK);
  }
  std::vector<unsigned int> ids(group.size(), 0);
  remote_ids[my_rank] = seg_id;
  gaspi_allreduce(ids.data(), ids.data(), 1, GASPI_OP_SUM, GASPI_TYPE_UINT, group, GASPI_BLOCK)); 
}

What it does: it tries to allocate the requested size on the already existing segment seg_id. If there is not enough memory in that old segment, a new seg_id is locally retrieved, bind to newly allocated memory and then registered at some remote ranks (stored in group). Afterwards all members of the group enter the gaspi_allreduce to exchange their segment ids, so that the ids can be used for communcation. The allreduce serves also as a barrier here (see gaspi_segment_create).

Question: works gaspi_segment_register as written here? That is, is it a one-sided call, which does either nothing on the remote rank or the remote rank receives the register request somehow magically?

Wish: If I'd use gaspi_segment_bind (with some memory from the outside), I have difficulties to release the segment. I cannot use segment_delete, since the memory is not owned by the segment. I could of course just release the memory at the outside, which renders the segment id invalid. But then I cannot reuse the segment id, since something like gaspi_segment_unbind is missing, isn't it?

[krzikalla]

Based on the GPI implementation gaspi_segment_unbind is unnecessary, since gaspi_segment_delete already handles bound and allocated segments differently. This should be made clear in the GASPI spec. At least the first sentence of 7.3.1

The synchronous local blocking procedure gaspi_segment_delete releases the
resources of a previously allocated memory segment.

is misleading. It must be read allocated or bound.

[mrahn]

The the question about segment_register: The allreduce implies GetMemory is called by all members of the group. So all new segments are registered with all remote ranks in the group. This looks valid (even though I don't know about registration with the creating rank). Also the segment_register is not using magic but an established connection, according to the spec. But yes, this implies an active component.

[mrahn]

About the unbind: There is no other way delete could behave. But yes, maybe worth mentioning it. Would you mind compiling the small snippets and change sets into a proposal for the forum?

[krzikalla]

Indeed the idea is to discuss and collect the issues here, turn them into proposals and then close them here.

[krzikalla]

Regarding unbind: delete could also (erroneously) try to deallocate the segment memory.