Global state to app.state to enable multiple APIs in a single process

.github/workflows/ci.yml

@@ -94,8 +94,7 @@ jobs:
         sleep 15
 
     - name: Test server, including OPTIONAL base URLs
-      # Pin validator action until merge of Materials-Consortia/optimade-validator-action#175
-      uses: Materials-Consortia/optimade-validator-action@dc4a7b6a83da42e5341a1a401e36d83375550fb7
+      uses: Materials-Consortia/optimade-validator-action@v2
       with:
         port: 3213
         path: /
@@ -109,8 +108,7 @@ jobs:
         sleep 15
 
     - name: Test index server, including OPTIONAL base URLs
-      # Pin validator action until merge of Materials-Consortia/optimade-validator-action#175
-      uses: Materials-Consortia/optimade-validator-action@dc4a7b6a83da42e5341a1a401e36d83375550fb7
+      uses: Materials-Consortia/optimade-validator-action@v2
       with:
         port: 3214
         path: /

docs/api_reference/server/create_app.md

@@ -0,0 +1,3 @@
+# create_app
+
+::: optimade.server.create_app

docs/deployment/.pages

@@ -1,4 +1,5 @@
 title: "Deployment"
 nav:
     - integrated.md
+    - multiple_apps.md
     - container.md

docs/deployment/integrated.md

@@ -3,7 +3,7 @@
 The `optimade` package can be used to create a standalone web application that serves the OPTIMADE API based on a pre-configured MongoDB backend.
 In this document, we are going to use `optimade` differently and use it to add an OPTIMADE API implementation alongside an existing API that employs an Elasticsearch storage layer.
 
-Let's assume we already have a *FastAPI* application that runs an unrelated web service, and that we use an Elasticsearch backend that contains all structure data, but not necessarily in a form that OPTIMADE expects.
+Let's assume we already have a _FastAPI_ application that runs an unrelated web service, and that we use an Elasticsearch backend that contains all structure data, but not necessarily in a form that OPTIMADE expects.
 
 ## Providing the `optimade` configuration
 
@@ -13,7 +13,7 @@ If you run `optimade` code inside another application, you might want to provide
 Let's say you have a file `optimade_config.json` as part of the Python module that you use to create your OPTIMADE API.
 
 !!! tip
-    You can find more detailed information about configuring the `optimade` server in the [Configuration](../configuration.md) section.
+You can find more detailed information about configuring the `optimade` server in the [Configuration](../configuration.md) section.
 
 Before importing any `optimade` modules, you can set the `OPTIMADE_CONFIG_FILE` environment variable to refer to your config file:
 
@@ -37,30 +37,21 @@ structures.structures_coll = MyElasticsearchStructureCollection()
 
 You can imagine that `MyElasticsearchStructureCollection` either sub-classes the default `optimade` Elasticsearch implementation ([`ElasticsearchCollection`][optimade.server.entry_collections.elasticsearch.ElasticCollection]) or sub-classes [`EntryCollection`][optimade.server.entry_collections.entry_collections.EntryCollection], depending on how deeply you need to customize the default `optimade` behavior.
 
-## Mounting the OPTIMADE Python tools *FastAPI* app into an existing *FastAPI* app
+## Mounting the OPTIMADE Python tools _FastAPI_ app into an existing _FastAPI_ app
 
-Let's assume you have an existing *FastAPI* app `my_app`.
+Let's assume you have an existing _FastAPI_ app `my_app`.
 It already implements a few routers under certain path prefixes, and now you want to add an OPTIMADE implementation under the path prefix `/optimade`.
 
-First, you have to set the `root_path` in the `optimade` configuration, so that the app expects all requests to be prefixed with `/optimade`.
+The primary thing to modify is the `base_url` to match the new subpath. The easiest is to just update your configuration file or env parameters.
 
-Second, you simply mount the `optimade` app into your existing app `my_app`:
+Then one can just simply do the following:
 
 ```python
-from optimade.server.config import CONFIG
+from optimade.server.main import main as optimade
 
-CONFIG.root_path = "/optimade"
-
-from optimade.server import main as optimade
-
-optimade.add_major_version_base_url(optimade.app)
 my_app.mount("/optimade", optimade.app)
 ```
 
-!!! tip
-    In the example above, we imported `CONFIG` before `main` so that our config was loaded before app creation.
-    To avoid the need for this, the `root_path` can be set in your JSON config file, passed as an environment variable, or declared in a custom Python module (see [Configuration](../configuration.md)).
-
-See also the *FastAPI* documentation on [sub-applications](https://fastapi.tiangolo.com/advanced/sub-applications/).
+See also the _FastAPI_ documentation on [sub-applications](https://fastapi.tiangolo.com/advanced/sub-applications/).
 
-Now, if you run `my_app`, it will still serve all its routers as before and in addition it will also serve all OPTIMADE routes under `/optimade/` and the versioned URLs `/optimade/v1/`.
+Now, if you run `my_app`, it will still serve all its routers as before and in addition it will also serve all OPTIMADE routes under `/optimade/`.

docs/deployment/multiple_apps.md

@@ -0,0 +1,38 @@
+# Serve multiple OPTIMADE APIs within a single python process
+
+One can start multiple OPTIMADE API apps within a single FastAPI instance and mount them at different subpaths.
+
+This is enabled by the `create_app` method that allows to override parts of the configuration for each specific app, and set up separate loggers.
+
+Here's a simple example that sets up two OPTIMADE APIs and an Index Meta-DB respectively at subpaths `/app1`, `/app2` and `/idx`.
+
+```python
+from fastapi import FastAPI
+
+from optimade.server.config import ServerConfig
+from optimade.server.create_app import create_app
+
+parent_app = FastAPI()
+
+base_url = "http://127.0.0.1:8000"
+
+conf1 = ServerConfig()
+conf1.base_url = f"{base_url}/app1"
+conf1.mongo_database = "optimade_1"
+app1 = create_app(conf1, logger_tag="app1")
+parent_app.mount("/app1", app1)
+
+conf2 = ServerConfig()
+conf2.base_url = f"{base_url}/app2"
+conf2.mongo_database = "optimade_2"
+app2 = create_app(conf2, logger_tag="app2")
+parent_app.mount("/app2", app2)
+
+conf3 = ServerConfig()
+conf3.base_url = f"{base_url}/idx"
+conf3.mongo_database = "optimade_idx"
+app3 = create_app(conf3, index=True, logger_tag="idx")
+parent_app.mount("/idx", app3)
+```
+
+Note that `ServerConfig()` returns the configuration based on the usual sources - env variables or json file (see [Configuration](../configuration.md) section).

docs/getting_started/use_cases.md

@@ -6,16 +6,12 @@ The [Materials Project](https://materialsproject.org) uses `optimade-python-tool
 
 `optimade-python-tools` handles filter parsing, database query generation and response validation by running the reference server implementation with minimal configuration.
 
-[*odbx*](https://odbx.science), a small database of results from crystal structure prediction calculations, follows a similar approach.
+[_odbx_](https://odbx.science), a small database of results from crystal structure prediction calculations, follows a similar approach.
 This implementation is open source, available on GitHub at [ml-evs/odbx.science](https://github.com/ml-evs/odbx.science).
 
 ## Serving multiple databases
 
-[Materials Cloud](https://materialscloud.org) uses `optimade-python-tools` as a library to provide an OPTIMADE API entry to archived computational materials studies, created with the [AiiDA](https://aiida.net) Python framework and published through their archive.
-In this case, each individual study and archive entry has its own database and separate API entry.
-The Python classes within the `optimade` package have been extended to make use of AiiDA and its underlying [PostgreSQL](https://postgresql.org) storage engine.
-
-Details of this implementation can be found on GitHub at [aiidateam/aiida-optimade](https://github.com/aiidateam/aiida-optimade).
+[Materials Cloud](https://materialscloud.org) uses `optimade-python-tools` as a library to provide an OPTIMADE API entries to 1) their main databases create with the [AiiDA](https://aiida.net) Python framework; and 2) to user-contributed data via the Archive platform. Separate OPTIMADE API apps are started for each database, mounted as separate endpoints to a parent FastAPI instance. For converting the underying data to the OPTIMADE format, the [optimade-maker](https://github.com/materialscloud-org/optimade-maker) toolkit is used.
 
 ## Extending an existing API
 

optimade/client/client.py

@@ -511,11 +511,12 @@ def _binary_search_count_async(
                 )
             )
 
+            # if we got any data, we are below the target value
+            below = bool(result[base_url].data)
+
             self._progress.disable = self.silent
 
-            window, probe = self._update_probe_and_window(
-                window, probe, bool(result[base_url].data)
-            )
+            window, probe = self._update_probe_and_window(window, probe, below)
 
             if window[0] == window[1] and window[0] == probe:
                 return probe
@@ -557,16 +558,15 @@ def _update_probe_and_window(
             raise RuntimeError(
                 "Invalid arguments: must provide all or none of window, last_probe and below parameters"
             )
-
         probe: int = last_probe
 
         # Exit condition: find a range of (count, count+1) values
         # and determine whether the probe was above or below in the last guess
         if window[1] is not None and window[1] - window[0] == 1:
             if below:
-                return (window[0], window[0]), window[0]
-            else:
                 return (window[1], window[1]), window[1]
+            else:
+                return (window[0], window[0]), window[0]
 
         # Enclose the real value in the window, with `None` indicating an open boundary
         if below:
@@ -578,12 +578,13 @@ def _update_probe_and_window(
         if window[1] is None:
             probe *= 10
 
-        # Otherwise, if we're in the window and the ends of the window now have the same power of 10, take the average (102 => 108) => 105
-        elif round(math.log10(window[0])) == round(math.log10(window[0])):
+        # Otherwise, if we're in the window and the ends of the window now have the same power of 10 (or within +-1),
+        # take the average (102 => 108) => 105
+        elif abs(math.log10(window[1]) - math.log10(window[0])) <= 1:
             probe = (window[1] + window[0]) // 2
         # otherwise use logarithmic average (10, 1000) => 100
         else:
-            probe = int(10 ** (math.log10(window[1]) + math.log10(window[0]) / 2))
+            probe = int(10 ** ((math.log10(window[1]) + math.log10(window[0])) / 2))
 
         return window, probe
 

optimade/filtertransformers/base_transformer.py

@@ -82,7 +82,7 @@ class BaseTransformer(Transformer, abc.ABC):
 
     """
 
-    mapper: type[BaseResourceMapper] | None = None
+    mapper: BaseResourceMapper | None = None
     operator_map: dict[str, str | None] = {
         "<": None,
         "<=": None,
@@ -106,7 +106,7 @@ class BaseTransformer(Transformer, abc.ABC):
     _quantity_type: type[Quantity] = Quantity
     _quantities = None
 
-    def __init__(self, mapper: type[BaseResourceMapper] | None = None):
+    def __init__(self, mapper: BaseResourceMapper | None = None):
         """Initialise the transformer object, optionally loading in a
         resource mapper for use when post-processing.
 

optimade/filtertransformers/elasticsearch.py

@@ -101,7 +101,7 @@ class ElasticTransformer(BaseTransformer):
 
     def __init__(
         self,
-        mapper: type[BaseResourceMapper],
+        mapper: BaseResourceMapper,
         quantities: dict[str, Quantity] | None = None,
     ):
         if quantities is not None:

optimade/server/config.py

@@ -175,6 +175,7 @@ class ServerConfig(BaseSettings):
         extra="allow",
         env_file_encoding="utf-8",
         case_sensitive=False,
+        validate_assignment=True,
     )
 
     debug: Annotated[
@@ -370,12 +371,13 @@ class ServerConfig(BaseSettings):
             list[str | dict[Literal["name", "type", "unit", "description"], str]],
         ],
         Field(
+            default_factory=dict,
             description=(
                 "A list of additional fields to be served with the provider's prefix "
                 "attached, broken down by endpoint."
             ),
         ),
-    ] = {}
+    ]
     aliases: Annotated[
         dict[Literal["links", "references", "structures"], dict[str, str]],
         Field(
@@ -540,32 +542,6 @@ def check_license_info(cls, value: Any) -> AnyHttpUrl | None:
 
         return value
 
-    @model_validator(mode="after")
-    def use_real_mongo_override(self) -> "ServerConfig":
-        """Overrides the `database_backend` setting with MongoDB and
-        raises a deprecation warning.
-        """
-        use_real_mongo = self.use_real_mongo
-
-        # Remove from model
-        del self.use_real_mongo
-
-        # Remove from set of user-defined fields
-        if "use_real_mongo" in self.model_fields_set:
-            self.model_fields_set.remove("use_real_mongo")
-
-        if use_real_mongo is not None:
-            warnings.warn(
-                "'use_real_mongo' is deprecated, please set the appropriate 'database_backend' "
-                "instead.",
-                DeprecationWarning,
-            )
-
-            if use_real_mongo:
-                self.database_backend = SupportedBackend.MONGODB
-
-        return self
-
     @model_validator(mode="after")
     def align_mongo_uri_and_mongo_database(self) -> "ServerConfig":
         """Prefer the value of database name if set from `mongo_uri` rather than
@@ -621,10 +597,3 @@ def settings_customise_sources(
             ConfigFileSettingsSource(settings_cls),
             file_secret_settings,
         )
-
-
-CONFIG: ServerConfig = ServerConfig()
-"""This singleton loads the config from a hierarchy of sources (see
-[`customise_sources`][optimade.server.config.ServerConfig.settings_customise_sources])
-and makes it importable in the server code.
-"""