Add first version of documentation for SIA

.github/workflows/ci.yaml

@@ -54,6 +54,46 @@ jobs:
           tox-envs: "py,coverage-report,typing"
           tox-requirements: requirements/tox.txt
 
+  docs:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          # Ensure the documentation gets the right version.
+          fetch-depth: 0
+      - uses: lsst-sqre/run-tox@v1
+        with:
+          python-version: "3.12"
+          tox-envs: docs
+          tox-requirements: "requirements/tox.txt"
+          cache-key-prefix: docs
+      # Only attempt documentation uploads for long-lived branches, tagged
+      # releases, and pull requests from ticket branches.  This avoids version
+      # clutter in the docs and failures when a PR doesn't have access to
+      # secrets.
+      - uses: lsst-sqre/ltd-upload@v1
+        with:
+          project: sia
+          dir: "docs/_build/html"
+          username: ${{ secrets.LTD_USERNAME }}
+          password: ${{ secrets.LTD_PASSWORD }}
+        if: >
+          github.event_name != 'merge_group'
+          && (github.event_name != 'pull_request'
+              || startsWith(github.head_ref, 'tickets/'))
+  linkcheck:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v4
+      - name: Check links
+        uses: lsst-sqre/run-tox@v1
+        with:
+          python-version: "3.12"
+          tox-envs: docs-linkcheck
+          tox-requirements: "requirements/tox.txt"
+
   build:
     runs-on: ubuntu-latest
     needs: [lint, test]

Makefile

@@ -1,6 +1,6 @@
 .PHONY: help
 help:
-	@echo "Make targets for example"
+	@echo "Make targets for sia"
 	@echo "make init - Set up dev environment"
 	@echo "make run - Start a local development instance"
 	@echo "make update - Update pinned dependencies and run make init"
@@ -17,6 +17,16 @@ init:
 	uv pip install --upgrade pre-commit
 	pre-commit install
 
+# This is defined as a Makefile target instead of only a tox command because
+# if the command fails we want to cat output.txt, which contains the
+# actually useful linkcheck output. tox unfortunately doesn't support this
+# level of shell trickery after failed commands.
+.PHONY: linkcheck
+linkcheck:
+	sphinx-build -W --keep-going -n -T -b linkcheck docs    \
+            docs/_build/linkcheck                               \
+            || (cat docs/_build/linkcheck/output.txt; exit 1)
+
 .PHONY: run
 run:
 	tox run -e run
@@ -29,9 +39,20 @@ update-deps:
 	pip install --upgrade uv
 	uv pip install --upgrade pre-commit
 	pre-commit autoupdate
-	uv pip compile --upgrade --universal --generate-hashes		\
+	uv pip compile --upgrade --generate-hashes			\
+	    --output-file requirements/main.txt requirements/main.in
+	uv pip compile --upgrade --generate-hashes			\
+	    --output-file requirements/dev.txt requirements/dev.in
+	uv pip compile --upgrade --generate-hashes			\
+	    --output-file requirements/tox.txt requirements/tox.in
+
+# Useful for testing against a Git version of Safir.
+.PHONY: update-deps-no-hashes
+update-deps-no-hashes:
+	pip install --upgrade uv
+	uv pip compile --upgrade					\
 	    --output-file requirements/main.txt requirements/main.in
-	uv pip compile --upgrade --universal --generate-hashes		\
+	uv pip compile --upgrade					\
 	    --output-file requirements/dev.txt requirements/dev.in
-	uv pip compile --upgrade --universal --generate-hashes		\
-	    --output-file requirements/tox.txt requirements/tox.in
+	uv pip compile --upgrade					\
+	    --output-file requirements/tox.txt requirements/tox.in

changelog.d/20241023_001701_steliosvoutsinas_DM_47058.md

@@ -0,0 +1,5 @@
+<!-- Delete the sections that don't apply -->
+
+### New features
+
+- Added documentation for the SIA application (/docs)

docs/admin/index.rst

@@ -0,0 +1,129 @@
+##############
+Administration
+##############
+
+
+Helm configuration
+==================
+
+The SIAv2 service is deployed via Phalanx, which requires a ``values-{env}.yaml`` configuration file to be provided. Most of the configuration is fairly standard and common with other applications in Phalanx and has defaults in the ``values.yaml``.
+
+The main configuration that needs to be provided is the list of Butler Repositories (``butlerDataCollections``), and all the relevant attributes for each repository which are listed below:
+
+butlerDataCollections
+---------------------
+
+**config** (required)
+    A path (HTTP or File Path) to the Obscore configuration.
+
+    Example: ``"https://raw.githubusercontent.com/lsst-dm/dax_obscore/refs/heads/main/configs/dp02.yaml"``
+
+**label** (required)
+    The label for the Butler repository.
+
+    Example: ``"dp02"``
+
+**name** (required)
+    Name of the Butler repository. This is used by the app to configure the path to the SIAv2 service for this repository.
+    For example ``"dp02"`` will result in ``"https://data-dev.lsst.cloud/api/sia/dp02"``
+
+**butler_type** (required)
+    The type of Butler repository.
+
+    Example: ``"REMOTE"`` | ``"DIRECT"``
+
+**repository** (required)
+    A path (HTTP or File Path) to the Butler repository.
+
+    Example: ``"https://data-dev.lsst.cloud/api/butler/configs/dp02.yaml"``
+
+**datalink_url** (optional)
+    An HTTP path to overwrite the Datalink endpoint in the obscore repository with.
+
+    Example: ``"https://data-dev.lsst.cloud/api/datalink/links?ID=butler%3A//dp02/{id}"``.
+
+**default_instrument** (required)
+    The default instrument for this repository.
+
+    Example: ``"LSSTCam-imSim"``. Note that this will be deprecated in future releases.
+
+directButlerEnabled
+-------------------
+
+**bool** - Whether the instance of the SIA app supports any Direct Butler repositories.
+
+Note: The SIA application supports Remote Butler, Direct Butler or a mixture of the two. However if a Direct Butler repository is defined then the directButlerEnabled needs to be enabled.
+
+Ingresses
+==================
+
+**Anonymous Paths**
+
+By default SIA sets up an anonymous ingress for these paths:
+
+- path: "/openapi.json"
+
+- path: "/{collection}/capabilities"
+
+- path: "/{collection}/availability"
+
+**Authenticated Paths**
+
+All other paths of the app, including /query are authenticated:
+
+- path: /
+
+
+
+Vault secrets
+=============
+
+The SIA applications by default does not require any secrets to be setup by an administrator, as the secrets that it does expect are copied over from other Phalanx applications.
+An administrator can however define an environment specific secrets.yaml file and overwrite these if they wish.
+Note that the secrets currently defined in ``secrets.yaml`` are only applicable to the case that the service is supporting a Direct Butler Repository, otherwise they are not used.
+
+These secrets are listed below:
+
+**aws-credentials.ini**:
+  Copied from nublado if config.directButlerEnabled is set to true
+
+**butler-gcs-idf-creds.json**:
+  Copied from nublado if config.directButlerEnabled is set to true
+
+**postgres-credentials.txt**:
+  Copied from nublado if config.directButlerEnabled is set to true
+
+**slack-webhook**:
+  Copied from mobu if config.slackAlerts is set to true
+
+
+Deployment Information
+======================
+
+SIA will deploy one pod by default, if the ``config.directButlerEnabled`` is set to **false**, which is the SIA FastAPI application. This will be the main FastAPI web-service running the SIA application. Any relevant logs during operations will be in the logs of this pod.
+If ``directButlerEnabled`` is set to **true**, a utility pod ``fix-secret-permissions`` used to setup the permissions for the Direct Butler secrets will also be spawned.
+
+
+Use with a Remote Butler
+==========================
+
+To connect the SIA app with a Remote Butler repository, the values in the configuration need to be specified as described, ensuring their ``butler_type`` is set to **REMOTE** and that the ``config.directButlerEnabled`` field is set to **false**. Then when a user wishes to run a query they need to provide a Bearer token as with several other apps in Phalanx (see TAP for example). This is possible with most python clients (pyvo/astropy) and is handled automatically for the user when accessing via a browser having previously logged in.
+If a token is not provided the app will reject the request with an authorization error.
+
+
+Use with a Direct Butler
+==========================
+
+If an operator of the SIA application wishes to use the app with Direct Butler access rather than remote, they can specify the above parameters making sure to set ``butler_type`` to **DIRECT** & also setting ``config.directButlerEnabled`` to **true**.
+When this is used the app no longer expects to read in a user token from the headers to use when setting up the Butler access, and instead uses the authentication credentials from the ``sia`` secrets to authenticate with the Butler Postgres database.
+
+
+Service Self-Description
+========================
+
+SIA v2 Services are expected to return a self-description VOTable document with metadata describing the input & output of the service, including where appropriate ranges or enumeration of possible values.
+This SIA app is configured to generate that response via a combination of information gathered from the specific Butler Repository of the service being queried, and the repository attributes defined in the configuration values for the environment. In the future we may end-up generating everything from the Butler repository to ensure we avoid any unexpected behavior.
+
+
+
+

docs/api.rst

@@ -0,0 +1,5 @@
+########
+REST API
+########
+
+This is a stub page for the API.

docs/conf.py

@@ -0,0 +1 @@
+from documenteer.conf.guide import *

docs/documenteer.toml

@@ -0,0 +1,12 @@
+[project]
+title = "SIA"
+copyright = "2024 Association of Universities for Research in Astronomy, Inc. (AURA)"
+base_url="https://sia.lsst.io"
+github_url="https://github.com/lsst-sqre/sia"
+
+[project.openapi]
+openapi_path = "_static/openapi.json"
+doc_path = "api"
+
+[project.python]
+package = "sia"

docs/index.rst

@@ -0,0 +1,43 @@
+.. toctree::
+   :maxdepth: 1
+   :hidden:
+
+   usage/index
+   admin/index
+   api
+
+###########
+SIA
+###########
+SIA is an implementation of the IVOA `Simple Image Access v2`_ protocol as a `FastAPI`_ web service, designed to be deployed as part of the Rubin Science Platform through `Phalanx`_.
+It provides a capability of discovering, retrieving and gathering metadata for image holdings, via the use of the ObsCore Data Model.
+Users will in most cases be accessing the service via other client tools like **Firefly** or **TopCat** which abstract the data access flow behind user-friendly interfaces.
+
+The default configuration uses the `dax_obscore`_ package and interacts with a `Butler`_ repository to find images matching specific criteria. While the current release supports both remote and direct (local) Butler repositories, our primary focus has been on the Remote Butler use-case, resulting in more mature support for this option.
+The application expects as part of the configuration a list of Butler Data Collections, each of which expects a number of attributes which define how to access the repository.
+
+The github repository for `SIA`_ is where the application is developed. See the `CHANGELOG`_ for a list of versions and changes to the application.
+
+The system architecture & design considerations have been documented in https://github.com/lsst-sqre/sqr-095.
+
+.. grid:: 1
+
+   .. grid-item-card:: Usage
+      :link: usage/index
+      :link-type: doc
+
+      Learn how to use the SIAv2 Service.
+
+   .. grid-item-card:: Admin
+      :link: admin/index
+      :link-type: doc
+
+      Learn how to setup the application.
+
+.. _Simple Image Access v2: https://www.ivoa.net/documents/SIA/
+.. _FastAPI: https://fastapi.tiangolo.com/
+.. _Phalanx: https://github.com/lsst-sqre/phalanx
+.. _dax_obscore: https://github.com/lsst-dm/dax_obscore
+.. _Butler: https://github.com/lsst/daf_butler
+.. _SIA: https://github.com/lsst-sqre/sia
+.. _CHANGELOG: https://github.com/lsst-sqre/sia/blob/main/CHANGELOG.md