Add lockfiles to RAiDER

.circleci/config.yml

@@ -27,16 +27,27 @@ jobs:
             # it even in non-interactive terminals
             "${HOME}/conda/bin/mamba" shell hook --shell bash --root-prefix "${HOME}/conda" \
               > /etc/profile.d/mamba-raider-circleci.sh
-            echo "mamba activate RAiDER" >> /etc/profile.d/mamba-raider-circleci.sh
+
+      - run:
+          name: Install conda-lock
+          shell: /bin/bash -xl
+          command: |
+            mamba install --channel=conda-forge --name=base conda-lock
 
       - run:
           name: Create RAiDER environment
           no_output_timeout: 60m
           shell: /bin/bash -xl
           command: |
             PYTHON_VERSION="<< parameters.python-version >>"
-            sed -i "/python>=/c\ - python=${PYTHON_VERSION}" environment.yml
-            mamba env create -f environment.yml
+            if [ "${PYTHON_VERSION}" == "3.12" ]; then
+              LOCKFILE_PATH="conda-lock.yml"
+            else
+              LOCKFILE_PATH=".circleci/conda-lock-${PYTHON_VERSION}.yml"
+            fi
+            mamba activate base
+            conda-lock install --prefix "${HOME}/conda/envs/RAiDER" "${LOCKFILE_PATH}"
+            echo "mamba activate RAiDER" >> /etc/profile.d/mamba-raider-circleci.sh
 
       - run:
           name: Install RAiDER and check environment

.circleci/regenerate-locks.py

@@ -0,0 +1,64 @@
+"""
+Script to assist in regenerating each Python version's lockfiles after making
+changes to environment.yml.
+"""
+
+import re
+import subprocess
+import tempfile
+from pathlib import Path
+
+from tqdm import tqdm
+
+
+# RAiDER's supported Python versions. The last entry in the list will be placed
+# in the project root.
+VERSIONS = [
+    '3.9',
+    '3.10',
+    '3.11',
+    '3.12',
+]
+
+# The dependencies entry for python.
+# First group: " - python"
+# Second group: ">=3.8" or any version specification
+PATTERN_PYTHON_DEP = re.compile(r'^(\s*-\s*python)([<>=~]?=?.+$)', re.MULTILINE)
+
+ENVIROMENT_YML_PATH = Path('environment.yml')
+
+
+def generate_lock(out_path: Path, version: str, template: str) -> None:
+    """
+    Use conda-lock to generate a lockfile for this version of the
+    environment.yml file, and place it at the specified output path.
+    """
+    with tempfile.TemporaryDirectory() as tmp_dir_str:
+        tmp_dir = Path(tmp_dir_str)
+        env_path = tmp_dir / 'environment.yml'
+
+        # Hardcode a copy of the environment.yml file to this Python version
+        with env_path.open('w', encoding='utf-8') as f_env:
+            f_env.write(re.sub(PATTERN_PYTHON_DEP, f'\\1=={version}', template))
+
+        # Platforms explicitly listed in order to exclude win-64, since isce3
+        # and wand (and therefore RAiDER) are not compatible with Windows.
+        cmd = f'conda-lock --file {env_path} --lockfile {out_path} -p linux-64 -p osx-64 -p osx-arm64'
+        result = subprocess.run(cmd.split(' '), stdout=subprocess.PIPE, text=True)
+        if result.returncode != 0:
+            raise Exception(f'Command exited with non-zero status code {result.returncode}: "{cmd}"')
+
+
+def main() -> None:
+    with ENVIROMENT_YML_PATH.open(encoding='utf-8') as fin:
+        yml_content = fin.read()
+
+    for i, version in tqdm(enumerate(VERSIONS), total=len(VERSIONS), unit='lockfiles written'):
+        if i < len(VERSIONS) - 1:
+            generate_lock(Path(f'.circleci/conda-lock-{version}.yml'), version, yml_content)
+        else:
+            generate_lock(Path('conda-lock.yml'), version, yml_content)
+
+
+if __name__ == '__main__':
+    main()

CHANGELOG.md

@@ -19,10 +19,9 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 * [732](https://github.com/dbekaert/RAiDER/pull/732) - Updated `hrrr.py` to resolve a deprecation warning about `unary_union`.
 * [731](https://github.com/dbekaert/RAiDER/pull/731) - Fixed fetch routine for GMAO.
 
-### Changed
-* [746](https://github.com/dbekaert/RAiDER/pull/746) - Upgraded numpy version to >2.
-
 ### Added
+* [762](https://github.com/dbekaert/RAiDER/pull/762) - Added lockfiles through conda-lock to the repo, documentation, and CircleCI runners.
+* [746](https://github.com/dbekaert/RAiDER/pull/746) - Added support for numpy v2.
 * [725](https://github.com/dbekaert/RAiDER/pull/725) - Added rules to ignore all test artifacts in git.
 * [728](https://github.com/dbekaert/RAiDER/pull/728) - Added downloader tests for HRES, GMAO, and MERRA2.
 * [726](https://github.com/dbekaert/RAiDER/pull/726) - Updated code and documentation for changes to the CDS API.

CONTRIBUTING.md

@@ -48,7 +48,7 @@ RAiDER distributes two conda packages, `raider-base` a lighter-weight package th
 
 Fork RAiDER from GitHub UI, and then
 
-```
+```sh
 git clone https://github.com/dbekaert/RAiDER.git
 cd RAiDER
 git remote add my_user_name https://github.com/my_user_name/RAiDER.git
@@ -58,7 +58,7 @@ git remote add my_user_name https://github.com/my_user_name/RAiDER.git
 
 Fork RAiDER-docs from GitHub UI, and then
 
-```
+```sh
 git clone https://github.com/dbekaert/RAiDER-docs.git
 cd RAiDER-docs
 git remote add my_user_name https://github.com/my_user_name/RAiDER-docs.git
@@ -67,7 +67,7 @@ git remote add my_user_name https://github.com/my_user_name/RAiDER-docs.git
 
 ### Updating your local master against upstream master ###
 
-```
+```sh
 git checkout master
 git fetch origin
 # Be careful: this will loose all local changes you might have done now
@@ -78,9 +78,9 @@ git reset --hard origin/master
 
 [Here](https://thoughtbot.com/blog/git-interactive-rebase-squash-amend-rewriting-history) is a great tutorial if you are new to rewriting history with git.
 
-```
+```sh
 git checkout master
-(potentially update your local master against upstream, as described above)
+# (potentially update your local master against upstream, as described above)
 git checkout -b my_new_feature_branch
 
 # do work. For example:
@@ -114,11 +114,11 @@ git push my_user_name my_new_feature_branch
 ### Formatting and linting with [Ruff](https://docs.astral.sh/ruff/) ###
 
 Format your code to follow the style of the project with:
-```
+```sh
 ruff format
 ```
 and check for linting problems with:
-```
+```sh
 ruff check
 ```
 Please ensure that any linting problems in your changes are resolved before
@@ -130,17 +130,28 @@ editor.
 ### Issue a pull request from GitHub UI ###
 commit locally and push. To get a reasonable history, you may need to
 
-```
+```sh
 git rebase -i master
 ```
 
 , in which case you will have to force-push your branch with 
 
-```
+```sh
 git push -f origin my_new_feature_branch
 ```
 
 Once a pull request is issued it will be reviewed by multiple members before it will be approved and integrated into the main.
 
+### Changing dependencies ###
+If your branch makes changes to `environment.yml`, we ask that you regenerate
+and commit the lockfiles to reflect your changes for other users.
+This step also helps ensure that the project can still be solved in all of our
+supported Python versions after your changes.
+```sh
+python .circleci/regenerate_locks.py
+git add conda-lock.yml
+git add .circleci/conda-lock*.yml
+```
+
 ### Things you should NOT do
 (For anyone with push rights to RAiDER or RAiDER-docs) Never modify a commit or the history of anything that has been committed to https://github.com/dbekaert/RAiDER and https://github.com/dbekaert/RAiDER-docs.

README.md

@@ -47,29 +47,29 @@ conda activate RAiDER
 ### Using the Docker image
 
 RAiDER provides a [docker container image](https://docs.docker.com/get-started/) with all the necessary dependencies pre-installed. To get the latest released version: 
-```
+```sh
 docker pull ghcr.io/dbekaert/raider:latest
 ```
 a specific release version (>=v0.2.0 only):
-```
+```sh
 docker pull ghcr.io/dbekaert/raider:0.2.0
 ```
 or the current development version:
-```
+```sh
 docker pull ghcr.io/dbekaert/raider:test
 ```
 
 To run `raider.py` inside the container:
-```
+```sh
 docker run -it --rm ghcr.io/dbekaert/raider:latest
 ```
 To mount your current directory inside the container so that files will be written back to your local machine:
-```
+```sh
 docker run -it -v ${PWD}:/home/raider/work --rm ghcr.io/dbekaert/raider:latest
 cd work
 ```
 To jump into a `bash` shell inside the container:
-```
+```sh
 docker run -it --rm --entrypoint /bin/bash ghcr.io/dbekaert/raider:latest -l
 ```
 
@@ -99,12 +99,26 @@ TODO
 Contributions are welcome and heartily encourage! See our [contributing guide](https://github.com/dbekaert/RAiDER/blob/dev/CONTRIBUTING.md).
 
 ### Development install
-For development, we recommend installing directly from source.
-```
+For development, we recommend installing directly from source.  
+Solving can take a long time (upwards of an hour), so we also recommend 
+taking advantage of the included lockfile to install faster.
+```sh
+# Clone the project
 git clone https://github.com/dbekaert/RAiDER.git
 cd RAiDER
-conda env create -f environment.yml
+
+# Install conda-lock (if not already installed)
+pip install condax
+condax install conda-lock
+
+# Create RAiDER environment using the lockfile
+conda-lock install
+# Or solve and create from scratch without lockfile:
+# conda env create -f environment.yml
+
 conda activate RAiDER
+
+# Symlink project folder in as a pip package in the environment
 python -m pip install -e .
 ```
 For more details on installing from source see [here](https://github.com/dbekaert/RAiDER/blob/dev/docs/Installing_from_source.md).