Add first version of docs

.github/workflows/ci.yaml

@@ -54,6 +54,46 @@ jobs:
           tox-envs: "py,coverage-report,typing"
           tox-requirements: requirements/tox.txt
 
+  docs:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          # Ensure the documentation gets the right version.
+          fetch-depth: 0
+      - uses: lsst-sqre/run-tox@v1
+        with:
+          python-version: "3.12"
+          tox-envs: docs
+          tox-requirements: "requirements/tox.txt"
+          cache-key-prefix: docs
+      # Only attempt documentation uploads for long-lived branches, tagged
+      # releases, and pull requests from ticket branches.  This avoids version
+      # clutter in the docs and failures when a PR doesn't have access to
+      # secrets.
+      - uses: lsst-sqre/ltd-upload@v1
+        with:
+          project: sia
+          dir: "docs/_build/html"
+          username: ${{ secrets.LTD_USERNAME }}
+          password: ${{ secrets.LTD_PASSWORD }}
+        if: >
+          github.event_name != 'merge_group'
+          && (github.event_name != 'pull_request'
+              || startsWith(github.head_ref, 'tickets/'))
+  linkcheck:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v4
+      - name: Check links
+        uses: lsst-sqre/run-tox@v1
+        with:
+          python-version: "3.12"
+          tox-envs: docs-linkcheck
+          tox-requirements: "requirements/tox.txt"
+
   build:
     runs-on: ubuntu-latest
     needs: [lint, test]

changelog.d/20241023_001701_steliosvoutsinas_DM_47058.md

@@ -0,0 +1,5 @@
+<!-- Delete the sections that don't apply -->
+
+### New features
+
+- Added documentation for the SIA application (/docs)

docs/admin/index.rst

@@ -0,0 +1,122 @@
+##############
+Administration
+##############
+
+
+Helm configuration
+==================
+
+The SIAv2 service is deployed via Phalanx, which requires a ``values-{env}.yaml`` configuration file to be provided. Most of the configuration is fairly standard and common with other applications in Phalanx and has defaults in the ``values.yaml``.
+
+The main configuration that needs to be provided is the Butler Repositories, and all the relevant attributes for each repository which are listed below:
+
+butlerDataCollections
+---------------------
+
+**config** (required)
+    A path (HTTP or File Path) to the Obscore configuration.
+
+    Example: ``"https://raw.githubusercontent.com/lsst-dm/dax_obscore/refs/heads/main/configs/dp02.yaml"``
+
+**label** (required)
+    The label for the Butler repository.
+
+    Example: ``"dp02"``
+
+**name** (required)
+    Name of the Butler repository. This is used by the app to configure the path to the SIAv2 service for this repository.
+    For example ``"dp02"`` will result in ``"https://data-dev.lsst.cloud/api/sia/dp02"``
+
+**butler_type** (required)
+    The type of Butler repository.
+
+    Example: ``"REMOTE"`` | ``"DIRECT"``
+
+**repository** (required)
+    A path (HTTP or File Path) to the Butler repository.
+
+    Example: ``"https://data-dev.lsst.cloud/api/butler/configs/dp02.yaml"``
+
+**datalink_url** (optional)
+    An HTTP path to overwrite the Datalink endpoint in the obscore repository with.
+
+    Example: ``"https://data-dev.lsst.cloud/api/datalink/links?ID=butler%3A//dp02/{id}"``.
+
+**default_instrument** (required)
+    The default instrument for this repository.
+
+    Example: ``"LSSTCam-imSim"``. Note that this will be deprecated in future releases.
+
+
+Ingresses
+==================
+
+**Anonymous Paths**
+
+By default SIA sets up an anonymous ingress for these paths:
+
+- path: "/openapi.json"
+
+- path: "/{collection}/capabilities"
+
+- path: "/{collection}/availability"
+
+**Authenticated Paths**
+
+All other paths of the app, including /query are authenticated:
+
+- path: /
+
+
+
+Vault secrets
+=============
+
+The SIA applications by default does not require any secrets to be setup by an administrator, as the secrets that it does expect are copied over from other Phalanx applications.
+An administrator can however define an environment specific secrets.yaml file and overwrite these if they wish.
+
+These secrets are listed below:
+
+**aws-credentials.ini**:
+  Copied from nublado if config.directButlerEnabled is set to true
+
+**butler-gcs-idf-creds.json**:
+  Copied from nublado if config.directButlerEnabled is set to true
+
+**postgres-credentials.txt**:
+  Copied from nublado if config.directButlerEnabled is set to true
+
+**slack-webhook**:
+  Copied from mobu if config.slackAlerts is set to true
+
+
+Deployment Information
+======================
+
+SIA will deploy one pod by default, if the config.directButlerEnabled is set to false, which is the SIA FastAPI application. This will be the relevant web-service running the SIA application. Any relevant logs during operations will be in the logs of this pod.
+If directButlerEnabled is set to true, a utility pod "fix-secret-permissions" used to setup the permissions for the Direct Butler secrets will also be spawned.
+
+
+Use with a Remote Butler
+==========================
+
+To connect the SIA app with a Remote Butler repository, the values in the configuration need to be specified as described, ensuring the butler_type is set to "REMOTE". Then when a user wishes to run a query they need to provide a Bearer token as with several other apps in Phalanx (see TAP for example). This is possible with most python clients (pyvo/astropy) and is handled automatically for the user when accessing via a browser having previously logged in.
+If a token is not provided the app will reject the request with an authorization error.
+
+
+Use with a Direct Butler
+==========================
+
+If an operator of the SIA application wishes to use the app with Direct Butler access rather than remote, they can specify the above parameters making sure to set butler_type to "DIRECT" & also setting config.directButlerEnabled to true.
+When this is used the app no longer expects to read in a user token from the headers to use when setting up the Butler access, and instead uses the authentication credentials from the sia secrets to authenticate with the Butler Postgres database.
+
+
+Service Self-Description
+========================
+
+SIA v2 Services are expected to return a self-description VOTable document with metadata describing the input & output of the service, including where appropriate ranges or enumeration of possible values.
+This SIA app is configured to generate that response via a combination of information gathered from the specific Butler Repository of the service being queried, and the repository attributes defined in the configuration values for the environment. In the future we may end-up generating everything from the Butler repository to ensure we avoid any unexpected behavior.
+
+
+
+

docs/api.rst

@@ -0,0 +1,5 @@
+########
+REST API
+########
+
+This is a stub page for the API.

docs/conf.py

@@ -0,0 +1 @@
+from documenteer.conf.guide import *

docs/documenteer.toml

@@ -0,0 +1,12 @@
+[project]
+title = "SIA"
+copyright = "2024 Association of Universities for Research in Astronomy, Inc. (AURA)"
+base_url="https://sia.lsst.io"
+github_url="https://github.com/lsst-sqre/sia"
+
+[project.openapi]
+openapi_path = "_static/openapi.json"
+doc_path = "api"
+
+[project.python]
+package = "sia"

docs/index.rst

@@ -0,0 +1,43 @@
+.. toctree::
+   :maxdepth: 1
+   :hidden:
+
+   usage/index
+   admin/index
+   api
+
+###########
+SIA
+###########
+SIA is an implementation of the IVOA `Simple Image Access v2`_ protocol as a `FastAPI`_ web service, designed to be deployed as part of the Rubin Science Platform through `Phalanx`_.
+
+The default configuration uses the `dax_obscore`_ package and interacts with a `Butler`_ repository to find images matching specific criteria.
+
+While the current release supports both remote and direct (local) Butler repositories, our primary focus has been on the Remote Butler use-case, resulting in more mature support for this option.
+
+The application expects as part of the configuration a list of Butler Data Collections, each of which expects a number of attributes which define how to access the repository.
+
+SIA is developed on `SIA`_. It is deployed via Phalanx.
+
+The system architecture & design considerations have been documented in https://github.com/lsst-sqre/sqr-095.
+
+.. grid:: 1
+
+   .. grid-item-card:: Usage
+      :link: usage/index
+      :link-type: doc
+
+      Learn how to use the SIAv2 Service.
+
+   .. grid-item-card:: Admin
+      :link: admin/index
+      :link-type: doc
+
+      Learn how to setup the application.
+
+.. _Simple Image Access v2: https://www.ivoa.net/documents/SIA/
+.. _FastAPI: https://fastapi.tiangolo.com/
+.. _Phalanx: https://github.com/lsst-sqre/phalanx
+.. _dax_obscore: https://github.com/lsst-dm/dax_obscore
+.. _Butler: https://github.com/lsst/daf_butler
+.. _SIA: https://github.com/lsst-sqre/sia>

docs/usage/index.rst

@@ -0,0 +1,129 @@
+#####
+Usage
+#####
+
+
+``sia`` is an image-access API complying with the IVOA SIA (v2) specification.
+For documentation on the protocol and details on usage including input parameters & output see the IVOA `SIAv2`_ specification.
+
+Parameters
+=============
+
+Example of commonly used parameters:
+
+:POS: Central position of the region of interest (ICRS RA,Dec in degrees)
+:SIZE: Size of the search region in degrees
+:BAND: Energy bandpass to search
+:TIME: Time interval to search
+
+For a full list of parameters see the IVOA `SIAv2`_ specification
+
+HTTP Methods
+=============
+Both POST & GET methods are implemented for the /query API in accordance to the specification
+
+HTTP GET Example::
+
+    https://data-dev.lsst.cloud/api/sia/dp02/query?POS=CIRCLE+55.7467+-32.2862+0.05
+
+HTTP POST Example::
+
+    curl -X POST "https://data-dev.lsst.cloud/api/sia/dp02/query" \
+	-H "Authorization: Bearer your_token_here" \
+	-d "POS=CIRCLE+55.7467+-32.2862+0.05"
+
+Response
+=============
+
+SIAv2 responses will typically be in VOTable format, containing:
+
+- Metadata about the service
+
+Metadata like the fields of the response data will be included in the response assuming it did not produce an error table.
+If MAXREC is set to 0, the self-description VOTable will be returned, which contains detailed information on the expected parameters, including range of possible values where appropriate and the result fields.
+
+- Table of available image products matching the query
+
+The results will include in the "access_url" field a link that can be used to retrieve each image.
+The format (access_format) will either be a datalink (x-votable+xml;content=datalink) if the results are a datalink or the image content-type if the link is a direct download link to the image.
+
+Example response structure::
+
+
+    <VOTABLE>
+      <RESOURCE type="results">
+        <TABLE>
+          <FIELD name="access_url" datatype="char" ucd="meta.ref.url"/>
+          <FIELD name="access_format" datatype="char" ucd="meta.code.mime"/>
+          <!-- Other metadata fields -->
+          <DATA>
+            <TABLEDATA>
+              <TR>
+                <TD>http://example.com/image1.fits</TD>
+                <TD>image/fits</TD>
+                <!-- Other metadata values -->
+              </TR>
+              <!-- More table rows -->
+            </TABLEDATA>
+          </DATA>
+        </TABLE>
+      </RESOURCE>
+    </VOTABLE>
+
+VOTable is the only format supported in this current version. In the future this may be extended to support other formats depending on requests and feedback from the community.
+
+Errors
+=============
+
+SIAv2 services use standard HTTP status codes. Common errors:
+
+:400: Bad Request (invalid parameters)
+:404: Not Found (no matching data)
+:500: Internal Server Error
+
+Bad request errors (400), which in most cases will be invalid parameters provided via a query are displayed as a VOTable error.
+
+The other two error types indicate either an invalid URL or an unexpected server-side issue which needs to be resolved so we do not format these as VOTables.
+
+Example Error VOTable::
+
+    <VOTABLE xmlns="http://www.ivoa.net/xml/VOTable/v1.3" version="1.3">
+      <RESOURCE>
+        <INFO name="QUERY_STATUS" value="ERROR">UsageFault: Unrecognized shape in POS string 'other_shape'</INFO>
+      </RESOURCE>
+    </VOTABLE>
+
+
+Discovery
+=============
+
+The expectation is that production SIAv2 services will be registered and discoverable via VO clients, however if they are not, use of the service will require users to input the SIA service URL manually or any clients using it to hard code this value.
+
+
+Query using Pyvo
+===================
+
+Querying the SIAv2 service is also possible using the `pyvo`_ package.
+Here is an example of such a query::
+
+    from pyvo.dal import SIA2Service
+    service = SIA2Service(sia_url, auth)
+    t1 = Time("60550.31803461111", format='mjd').to_datetime()
+    t2 = Time("60550.31838182871", format='mjd').to_datetime()
+    results = service.search(pos=(55.7467, -32.2862, 0.05), time=[t1, t2])
+    results.to_table()
+
+Result can then be parsed to download the images using the access_url field in
+the response::
+
+    dataLinkUrl = random.choice(results).access_url
+    from pyvo.dal.adhoc import DatalinkResults
+    dl_result = DatalinkResults.from_result_url(
+        dataLinkUrl, session=auth
+    )
+    image_url = dl_result.getrecord(0).get('access_url')
+
+
+.. _Butler: https://github.com/lsst/daf_butler
+.. _pyvo: https://pyvo.readthedocs.io/en/latest/
+.. _SIAv2: https://www.ivoa.net/documents/SIA/

requirements/dev.in

@@ -18,6 +18,7 @@ pydantic
 pytest
 pytest-asyncio
 pytest-cov
+documenteer[guide]
 
 # Documentation
 scriv