DM-52370: Extended replica allocation services

The new services extend chunk allocations (placement) to include all
existing replicas for the chunks in question instead of just one
allocation per chunk. The extended services are meant to be used by
the ingest workflows to push multiple replicas of a chunk in scenarios
where the replication level is bigger than 1 and multiple replicas
already exist in Qserv. The older services would fail in such scenarios.

Minor refactoring on the older code.

Fixed a bug in the JSON schema of  of the chunks allocation service
that was returning the key "location" instead of "locations". Migrated
the integration test accordingly.

Fixed a bug in the integration test.

Migratd the documentation on the Ingest API.

Author: iagaponenko

doc/ingest/api/index.rst

@@ -1,7 +1,7 @@
 
 .. note::
 
-   Information in this guide corresponds to the version **51** of the Qserv REST API. Keep in mind
+   Information in this guide corresponds to the version **52** of the Qserv REST API. Keep in mind
    that each implementation of the API has a specific version. The version number will change
    if any changes to the implementation or the API that might affect users will be made.
    The current document will be kept updated to reflect the latest version of the API.

doc/ingest/api/reference/rest/controller/table-location.rst

@@ -119,10 +119,66 @@ The service also supports an alternative method accepting a transaction identifi
 
 If a request succeeded, the System would respond with the following JSON object:
 
+.. code-block::
+
+    {   "location" : {
+            "chunk" :          <number>,
+            "worker" :         <string>,
+            "host" :           <string>,
+            "host_name" :      <string>,
+            "port" :           <number>,
+            "http_host" :      <string>,
+            "http_host_name" : <string>,
+            "http_port" :      <number>
+        },
+        ...
+    }
+
+Where, the object represents a worker where the Ingest system requests the workflow to forward the chunk contributions.
+See an explanation of the attributes in:
+
+- :ref:`table-location-connect-params`
+
+
+.. _table-location-chunks-one-multi:
+
+Single chunk allocation (all replicas of a chunk)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following service is meant to be used for allocating/locating (potentially) multiple replicas of a single chunk-multi:
+
+..  list-table::
+    :widths: 10 90
+    :header-rows: 0
+
+    * - ``POST``
+      - ``/ingest/chunk-multi``
+
+Where the request object has the following schema, in which a client would have to provide the name of a database:
+
+.. code-block::
+
+    {   "database" : <string>,
+        "chunk" :    <number>
+    }
+
+The service also supports an alternative method accepting a transaction identifier (transactions are always associated with the corresponding databases):
+
+.. code-block::
+
+    {   "transaction_id" : <number>,
+        "chunk" :          <number>
+    }
+
+**Note** the difference in the object schema - unlike the single-chunk allocator, this one expects an array of chunk numbers.
+
+If a request succeeded, the System would respond with the following JSON object:
+
 .. code-block::
 
     {   "locations" : [
-            {   "worker" :         <string>,
+            {   "chunk" :          <number>,
+                "worker" :         <string>,
                 "host" :           <string>,
                 "host_name" :      <string>,
                 "port" :           <number>,
@@ -134,11 +190,12 @@ If a request succeeded, the System would respond with the following JSON object:
         ]
     }
 
-Where, the object represents a worker where the Ingest system requests the workflow to forward the chunk contributions.
+Where, each object in the array represents a particular worker where the corresponding replica of the chunk is located.
 See an explanation of the attributes in:
 
 - :ref:`table-location-connect-params`
 
+
 .. _table-location-chunks-many:
 
 Multiple chunks allocation
@@ -161,7 +218,7 @@ Where the request object has the following schema, in which a client would have
         "chunks" :   [<number>, <number>, ... <number>]
     }
 
-Like the above-explained case of the single chunk allocation service, this one also supports an alternative method accepting
+Like the above-explained case of other chunk allocation service, this one also supports an alternative method accepting
 a transaction identifier (transactions are always associated with the corresponding databases):
 
 .. code-block::
@@ -194,6 +251,66 @@ Where, each object in the array represents a particular worker. See an explanati
 
 - :ref:`table-location-connect-params`
 
+
+
+.. _table-location-chunks-many-multi:
+
+Multiple chunks allocation (all replicas of each chunk)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For allocating/locating all available replicas of each chunk in the requested collectionone would have to use the following service:
+
+..  list-table::
+    :widths: 10 90
+    :header-rows: 0
+
+    * - ``POST``
+      - ``/ingest/chunks-multi``
+
+Where the request object has the following schema, in which a client would have to provide the name of a database:
+
+.. code-block::
+
+    {   "database" : <string>,
+        "chunks" :   [<number>, <number>, ... <number>]
+    }
+
+Like the above-explained case of other chunk allocation services, this one also supports an alternative method accepting
+a transaction identifier (transactions are always associated with the corresponding databases):
+
+.. code-block::
+
+    {   "transaction_id" : <number>,
+        "chunks" :        [<number>, <number>, ... <number>]
+    }
+
+**Note** the difference in the object schema - unlike the single-chunk allocator, this one expects an array of chunk numbers, where
+each chunk may have multiple replicas. In the later case the service will return multiple entries for the same chunk number.
+
+The resulting object  has the following schema:
+
+.. code-block::
+
+    {   "locations" : [
+            {   "chunk" :          <number>,
+                "worker" :         <string>,
+                "host" :           <string>,
+                "host_name" :      <string>,
+                "port" :           <number>,
+                "http_host" :      <string>,
+                "http_host_name" : <string>,
+                "http_port" :      <number>
+            },
+            ...
+        ]
+    }
+
+Where, each object in the array represents a particular worker. See an explanation of the attributes in:
+
+- :ref:`table-location-connect-params`
+
+
+
 .. _table-location-connect-params:
 
 Connection parameters of the workers

python/lsst/qserv/admin/replication_interface.py

@@ -330,8 +330,9 @@ def ingest_chunk_config(self, transaction_id: int, chunk_id: str) -> ChunkLocati
                 )
             ),
         )
+        loc = res["location"]
         return ChunkLocation(
-            res["chunk"], res["host"], str(res["port"]), res["http_host"], str(res["http_port"])
+            loc["chunk"], loc["host"], str(loc["port"]), loc["http_host"], str(loc["http_port"])
         )
 
     def ingest_chunk_configs(self, transaction_id: int, chunk_ids: list[int]) -> list[ChunkLocation]:
@@ -363,7 +364,7 @@ def ingest_chunk_configs(self, transaction_id: int, chunk_ids: list[int]) -> lis
             ChunkLocation(
                 loc["chunk"], loc["host"], str(loc["port"]), loc["http_host"], str(loc["http_port"])
             )
-            for loc in res["location"]
+            for loc in res["locations"]
         ]
 
     def ingest_regular_table(self, transaction_id: int) -> list[RegularLocation]: