Re-organize environment files, lock command environment, and add docs

Building from the locked command environment saves some time.

.github/workflows/test-and-build.yml

@@ -37,7 +37,7 @@ jobs:
       - name: "Install Conda environment"
         uses: "mamba-org/setup-micromamba@v1"
         with:
-          environment-file: "conda-lock.yml"
+          environment-file: "environments/main/conda-lock.yml"
           # When using a lock-file, we have to set an environment name.
           environment-name: "qgreenland-ci"
           cache-environment: true

Dockerfile

@@ -4,7 +4,7 @@
 
 FROM axiom/docker-luigi:3.0.3-alpine AS luigi
 # This build stage only exists to grab the luigi run script. Luigi dependency
-# itself is specified in `environment.yml`
+# itself is specified in our conda environment.
 # TODO: Why is this necessary? Does `luigid` not come along with the conda package?
 
 FROM mambaorg/micromamba:1.4.2 AS micromamba
@@ -15,7 +15,6 @@ USER root
 # `git` is required for analyzing the current version
 # `make` is required for building sphinx docs
 # `texlive-latex-extra` is required for pdf doc builds
-# TODO: Remove `make`
 RUN apt-get update && apt-get install -y \
   git \
   make \
@@ -32,9 +31,9 @@ COPY --chown=$MAMBA_USER:$MAMBA_USER . .
 # tolerate this user mismatch.
 RUN git config --global --add safe.directory "${TASKS_DIR}"
 
-# Set up the Luigi task environment and the command environment
-RUN micromamba install -y -n base -f conda-lock.yml
-RUN micromamba create -y -f environment.cmd.yml
+# Set up the main environment and the command environment
+RUN micromamba install --yes --name "base" --file "environments/main/conda-lock.yml"
+RUN micromamba create --yes --name "qgreenland-cmd" --file "environments/command/conda-lock.yml"
 
 # Cleanup
 RUN micromamba clean --all --yes

environments/README.md

@@ -0,0 +1,20 @@
+# QGreenland's `conda` environments
+
+QGreenland has many pieces that require different dependencies, and those are defined in
+these different environments.
+
+Environments are defined in `environment.yml` files, and pinned to allow packages to
+upgrade over time without too much unexpected breakage. Environments are locked in
+`conda-lock.yml` files with exact pins and checksums for reproducibility.
+
+
+## Locking
+
+In any directory containing an `environment.yml` file, run `conda-lock` to lock that
+environment.
+
+
+## Notes
+
+The `main` and `command` environments are only locked for the `linux-64` platform
+because it's expected that they will only be used in a Linux container context.

environments/command/README.md

@@ -0,0 +1,18 @@
+# QGreenland's "command" `conda` environment
+
+This environment exists for running `CommandStep` layer data processing steps.
+
+The original rationale for creation of this environment was conflict between
+QGreenland's [main environment](/environments/main/README.md) dependencies and
+dependencies needed for layer data processing.
+
+
+## TODO
+
+* Ideally all data processing, including Python steps, would run in this environment for
+  reproducibility.  When that's implemented, consider renaming this environment to the
+  "step" environment?  In the long term, we may consider extracting the framework
+  elements to a library, and in that case we'd want the end-user to define data
+  processing steps, and the dependencies required to execute those steps, preferably
+  with a lock-file defining the exact environment that was _actually_ used. So this
+  pattern of defining a data-processing lock-file is probably here to stay.