Merge pull request #445 from lsst-sqre/tickets/DM-34097

[DM-34097] Restructure Gafaelfawr code

CHANGELOG.rst

@@ -2,10 +2,24 @@
 Change log
 ##########
 
-3.6.1 (unreleased)
+Versioning follows `semver <https://semver.org/>`__.
+Versioning assumes that Gafaelfawr is installed and configured via `Phalanx <https://phalanx.lsst.io/>`__, so only changes to configuration changes exposed in the Helm values file are considered breaking changes.
+The internal configuration format may change in minor releases.
+
+4.0.0 (2022-03-25)
 ==================
 
+As of this release, the only supported mechanism for installing Gafaelfawr is as part of the Vera C. Rubin Science Platform, using `Phalanx <https://github.com/lsst-sqre/phalanx/>`__.
+
+- The Gafaelfawr token lifetime is now configured with ``config.tokenLifetimeMinutes`` instead of ``config.issuer.expMinutes``.
+- The internal OpenID Connect server now puts the numeric UID in a ``uid_number`` claim rather than ``uidNumber`` for consistency with the naming scheme of other claims.
+- InfluxDB 1.x token generation is now configured with ``config.influxdb.enabled`` and ``config.influxdb.username`` without the ``issuer`` component.
+- Drop support for restricting the upstream OpenID Connect provider to specific key IDs.
+  This prevents upstream key rotation for dubious security benefit given that Gafaelfawr still verifies the issuer URL and then reaches out to its ``.well-known`` endpoints to retrieve the public key and verify the key signature.
+- Log token scopes as proper lists instead of space- or comma-separated strings.
+- Return 404 with a proper error if the OpenID Connect server routes are accessed when Gafaelfawr is not configured to act as an OpenID Connect server.
 - Drop support for Python 3.9.
+- Update dependencies.
 
 3.6.0 (2022-02-24)
 ==================

docs/api.rst

@@ -31,8 +31,6 @@ API reference
 
 .. automodapi:: gafaelfawr.factory
 
-.. automodapi:: gafaelfawr.issuer
-
 .. automodapi:: gafaelfawr.keypair
 
 .. automodapi:: gafaelfawr.middleware.state
@@ -61,6 +59,8 @@ API reference
 
 .. automodapi:: gafaelfawr.services.admin
 
+.. automodapi:: gafaelfawr.services.influxdb
+
 .. automodapi:: gafaelfawr.services.kubernetes
 
 .. automodapi:: gafaelfawr.services.oidc
@@ -87,5 +87,3 @@ API reference
    :include-all-objects:
 
 .. automodapi:: gafaelfawr.util
-
-.. automodapi:: gafaelfawr.verify

docs/applications.rst

@@ -239,6 +239,8 @@ The OpenID Connect client should be configured to request only the ``openid`` sc
 No other scope is supported.
 The client must be able to authenticate by sending a ``client_secret`` parameter in the request to the token endpoint.
 
+The JWT returned by the Gafaelfawr OpenID Connect server will include the authenticated username in the ``sub`` and ``preferred_username`` claims, and the numeric UID in the ``uid_number`` claim.
+
 Chronograf example
 ------------------
 
@@ -284,7 +286,7 @@ Authenticating to InfluxDB
    These tokens will only work with InfluxDB 1.x.
 
 Gafaelfawr optionally supports issuing tokens for InfluxDB 1.x authentication.
-To enable this support, set ``issuer.influxdb.enabled`` to true in :ref:`helm-settings`.
+To enable this support, set ``config.influxdb.enabled`` to true in :ref:`helm-settings`.
 Then, create an ``influxdb-secret`` Vault secret key with the shared key that InfluxDB uses to verify the token.
 This can be any string of characters, such as the results of ``os.urandom(32).hex()``.
 The same secret must be configured in the `InfluxDB configuration file <https://docs.influxdata.com/influxdb/v1.8/administration/authentication_and_authorization/>`__.
@@ -295,4 +297,4 @@ The result is a JSON object containing a ``token`` key, the contents of which ar
 
 The token will contain a ``username`` claim matching the user's Gafaelfawr username and will expire at the same time as the token or session used to authenticate to this route.
 
-If you want all InfluxDB tokens to contain the same ``username`` field so that you can use a single generic InfluxDB account, set ``issuer.influxdb.username`` to that value in :ref:`helm-settings`.
+If you want all InfluxDB tokens to contain the same ``username`` field so that you can use a single generic InfluxDB account, set ``config.influxdb.username`` to that value in :ref:`helm-settings`.

docs/configuration.rst

@@ -2,8 +2,8 @@
 Installation guide
 ##################
 
-Gafaelfawr was written to run inside a Kubernetes environment.
-While there is nothing intrinsic in Gafaelfawr that would prevent it from working in some other environment, only installation on Kubernetes has been documented or tested.
+Gafaelfawr was written to run within the Vera C. Rubin Science Platform.
+While there is nothing intrinsic in Gafaelfawr that would prevent it from working in some other environment, only installation via `Phalanx <https://github.com/lsst-sqre/phalanx>`__ is supported or has been tested.
 
 Prerequisites
 =============
@@ -86,8 +86,8 @@ You will need the following information from the OpenID Connect provider:
 Vault secrets
 =============
 
-The standard Helm chart for Gafaelfawr (described below) assumes that you will use `Vault`_ to store secrets and `Vault Secrets Operator`_ to materialize those secrets in Kubernetes.
-If you are using it, create a Vault secret with the following keys:
+Gafaelfawr uses secrets stored in `Vault`_ and uses `Vault Secrets Operator`_ to materialize those secrets in Kubernetes.
+The Phalanx installer expects a Vault secret named ``gafaelfawr`` in the relevant Science Platform environment containing the following keys:
 
 ``bootstrap-token``
     A Gafaelfawr token created with ``gafaelfawr generate-token``.
@@ -107,16 +107,16 @@ If you are using it, create a Vault secret with the following keys:
     This is only required if you're using GitHub for authentication.
 
 ``influxdb-secret`` (optional)
+    Only used if the Helm chart parameter ``config.influxdb.enabled`` is set to true.
     The shared secret to use for issuing InfluxDB tokens.
     See :ref:`influxdb` for more information.
-    You can omit this if you don't need InfluxDB token support.
 
 ``oidc-client-secret``
     The secret for an OpenID Connect authentication provider.
     This is only required if you're using generic OpenID Connect for authentication.
 
 ``oidc-server-secrets`` (optional)
-    Only used if the Helm chart parameter ``oidcServer.enabled`` is set to true.
+    Only used if the Helm chart parameter ``config.oidcServer.enabled`` is set to true.
     The JSON representation of the OpenID Connect clients.
     Must be a JSON list of objects, each of which must have ``id`` and ``secret`` keys corresponding to the ``client_id`` and ``client_secret`` parameters sent by OpenID Connect clients.
     See :ref:`openid-connect` for more information.
@@ -130,35 +130,37 @@ If you are using it, create a Vault secret with the following keys:
     Generate with :py:meth:`cryptography.fernet.Fernet.generate_key`.
 
 ``signing-key``
+    Only used if the Helm chart parameter ``config.oidcServer.enabled`` is set to true.
     The PEM-encoded RSA private key used to sign internally-issued JWTs.
     Generate with ``gafaelfawr generate-key``.
 
-You will reference the path to this secret in Vault when configuring the Helm chart later.
-
 .. _helm-settings:
 
-Helm deployment
-===============
-
-The supported way of deploying Gafaelfawr is to use the Helm chart in the `Rubin Observatory charts repository <https://lsst-sqre.github.io/charts/>`__.
-The Helm chart only supports GitHub or CILogon as identity providers.
+Helm configuration
+==================
 
-To use that chart, you will need to provide a ``values.yaml`` file or otherwise set various Helm values.
+The supported way of deploying Gafaelfawr is as a Phalanx service, using the Helm chart in `the Phalanx repository <https://github.com/lsst-sqre/phalanx/tree/master/services/gafaelfawr/>`__.
+You will need to provide a ``values-<environment>.yaml`` file for your Phalanx environment.
 Below are the most-commonly-used settings.
-For a complete reference, see `the Helm chart documentation <https://github.com/lsst-sqre/charts/tree/master/charts/gafaelfawr>`__.
+For a complete reference, see `the Helm chart documentation <https://github.com/lsst-sqre/phalanx/tree/master/services/gafaelfawr>`__.
+For examples, see the other ``values-<environment>.yaml`` files in that directory.
 
-For examples, see `the configuration for the LSST Science Platform deployments <https://github.com/lsst-sqre/lsp-deploy/blob/master/services/gafaelfawr>`__.
+In the below examples, the full key hierarchy is shown for each setting.
+For example:
 
-.. _basic-settings:
+.. code-block::
 
-Basic settings
---------------
+   config:
+     cilogon:
+       test: true
 
-Set the path in Vault where the Gafaelfawr secret is stored:
+When writing a ``values-<environment>.yaml`` chart, you should coalesce all settings so that each level of the hierarchy appears only once.
+For example, there should be one top-level ``config:`` key and all parameters that start with ``config.`` should go under that key.
 
-.. code-block:: yaml
+.. _basic-settings:
 
-   vaultSecretsPath: "secret/path/in/vault"
+Basic settings
+--------------
 
 Set the URL to the PostgreSQL database that Gafaelfawr will use:
 
@@ -170,22 +172,6 @@ Set the URL to the PostgreSQL database that Gafaelfawr will use:
 Do not include the password in the URL; instead, put the password in the ``database-password`` key in the Vault secret.
 If you are using Cloud SQL with the Cloud SQL Auth Proxy (see :ref:`cloudsql`), use ``localhost`` for the hostname portion.
 
-Set the hostname that Gafaelfawr will be protecting:
-
-.. code-block:: yaml
-
-   config:
-     host: "hostname.example.com"
-   ingress:
-     host: "hostname.example.com"
-
-You can omit ``ingress.host`` if you aren't using named virtual hosts and want all routes to be registered for ``*``.
-The ``/auth``, ``/login``, ``/logout``, ``/oauth2/callback``, and ``/.well-known/jwks.json`` routes will be claimed under this host (or under ``*`` if it is not given) by the Gafaelfawr ingress configuration.
-If ``config.oidcServer.enabled`` is set to true, the ``/.well-known/openid-configuration`` route will also be claimed.
-
-If you need to configure TLS options or annotations for the ingress, use ``ingress.annotations`` and ``ingress.tls``.
-The syntax is the same as the ``metadata.annotations`` and ``spec.tls`` attributes of a Kubernetes ``Ingress`` resource.
-
 To add additional information to the error page from a failed login, set ``config.errorFooter`` to a string.
 This string will be embedded verbatim, inside a ``<p>`` tag, in all login error messages.
 It may include HTML and will not be escaped.
@@ -199,6 +185,16 @@ Production deployments should use at least two replicas.
 
    replicaCount: 2
 
+Change the token lifetime by setting ``config.tokenLifetimeMinutes``.
+The default is 1380 (23 hours).
+
+.. code-block:: yaml
+
+   config:
+     tokenLifetimeMinutes: 43200  # 30 days
+
+This setting will also affect the lifetime of tokens issued by the OpenID Connect server and the InfluxDB token issuer, if enabled.
+
 Finally, you may want to define the initial set of administrators:
 
 .. code-block:: yaml
@@ -342,7 +338,7 @@ The default includes:
        "user:token": "Can create and modify user tokens"
 
 which are used internally by Gafaelfawr, plus the scopes that are used by the Rubin Science Platform.
-You can add additional scopes by adding more key/value pairs to the ``config.knownScopes`` object in ``values.yaml``.
+You can add additional scopes by adding more key/value pairs to the ``config.knownScopes`` object in ``values-<environment>.yaml``.
 
 Once the scopes are configured, you will need to set up a mapping from groups to scope names.
 
@@ -485,23 +481,22 @@ OpenID Connect server
 
 Gafaelfawr can act as an OpenID Connect identity provider for relying parties inside the Kubernetes cluster.
 To enable this, set ``config.oidcServer.enabled`` to true.
-If this is set, ``oidc-server-secrets`` must be set in the Gafaelfawr Vault secret.
+If this is set, ``oidc-server-secrets`` and ``signing-key`` must be set in the Gafaelfawr Vault secret.
 See :ref:`openid-connect` for more information.
 
 InfluxDB tokens
 ---------------
 
-To enable issuing of InfluxDB tokens, set ``config.issuer.influxdb.enabled``.
-To force all InfluxDB tokens to be issued with the same username, instead of the username requesting the token, set ``config.issuer.influxdb.username``.
+To enable issuing of InfluxDB tokens, set ``config.influxdb.enabled``.
+To force all InfluxDB tokens to be issued with the same username, instead of the username requesting the token, set ``config.influxdb.username``.
 For example:
 
 .. code-block:: yaml
 
    config:
-     issuer:
-       influxdb:
-         enabled: true
-         username: "influxdbuser"
+     influxdb:
+       enabled: true
+       username: "influxdbuser"
 
 If this is set, ``influxdb-secret`` must be set in the Vault secret.
 See :ref:`influxdb` for more information.

docs/index.rst

@@ -5,7 +5,7 @@ Gafaelfawr
 Gafaelfawr is the authentication and authorization front-end for the Vera C. Rubin Observatory Science Platform.
 It's primary purpose is to serve as an NGINX ``auth_request`` backend.
 It also provides a web page where people can create and manage long-lived tokens for use outside of a web browser, and can serve as a simple OpenID Connect server.
-As a component of the Science Platform, it is designed for deployment with Kubernetes.
+As a component of the Science Platform, it is designed for deployment with Kubernetes using the `Phalanx infrastructure <https://github.com/lsst-sqre/phalanx>`__.
 Gafaelfawr requires the Kubernetes `NGINX ingress controller <https://github.com/kubernetes/ingress-nginx>`__.
 
 Gafaelfawr is developed on `GitHub <https://github.com/lsst-sqre/gafaelfawr>`__.

docs/logging.rst

@@ -50,7 +50,7 @@ All authenticated routes add the following attributes once the user's token has
     Chosen from ``cookie`` (found in the session cookie), ``bearer`` (provided as a bearer token in an ``Authorization`` header), or ``basic-username`` or ``basic-password`` (provided as the username or password in an HTTP Basic ``Authorization`` header).
 
 ``user``
-    The username claim of the token (configured via the ``username_claim`` configuration parameter).
+    The username of the token.
 
 The ``/auth`` route adds the following attributes:
 

examples/docker/gafaelfawr.yaml

@@ -20,26 +20,17 @@ database_url: "postgresql://gafaelfawr:INSECURE@postgresql/gafaelfawr"
 # development not accessible to the Internet.
 bootstrap_token: "gt-OwYUKbFrIRfcTnKcX9bEGA.9SPTVsVv3fxUzXqogQpCXg"
 
-# Replace this user with your GitHub username.
-initial_admins:
-  - "example"
-
 # This Redis will be started by the docker-compose.yaml file at the top
 # level of the repository.
 redis_url: "redis://redis:6379/0"
 redis_password_file: "/run/secrets/redis-password"
 
+# How long Gafaelfawr-issued tokens should live by default.
+token_lifetime_minutes: 1440 # 1 day
+
 # Where to send the user after logging out.
 after_logout_url: "http://localhost:8080"
 
-# Configuration for the JWTs issued by Gafaelfawr.
-issuer:
-  iss: "http://localhost:8080"
-  key_id: "localhost-key-id"
-  aud: "http://localhost"
-  key_file: "/run/secrets/issuer-key"
-  exp_minutes: 1440 # 1 day
-
 # To get these values, go to Settings > Developer Settings for either a
 # GitHub user or an organization, go into OAuth Apps, and create a new
 # application.  Set the homepage URL to http://localhost:8080/ and the
@@ -48,6 +39,18 @@ github:
   client_id: "<github-client-id>"
   client_secret_file: "/run/secrets/github-client-secret"
 
+# Configuration for the Gafaelfawr OpenID Connect server.
+oidc_server:
+  issuer: "http://localhost:8080"
+  key_id: "localhost-key-id"
+  audience: "http://localhost"
+  key_file: "/run/secrets/issuer-key"
+  secrets_file: "/run/secrets/oidc-clients.json"
+
+# Replace this user with your GitHub username.
+initial_admins:
+  - "example"
+
 # Sample values.  You can replace these with anything you want to use
 # for a scope.
 known_scopes:

examples/gafaelfawr-dev.yaml

@@ -20,26 +20,17 @@ database_url: "postgresql://gafaelfawr:INSECURE@localhost/gafaelfawr"
 # development not accessible to the Internet.
 bootstrap_token: "gt-OwYUKbFrIRfcTnKcX9bEGA.9SPTVsVv3fxUzXqogQpCXg"
 
-# Replace this user with your GitHub username.
-initial_admins:
-  - example
-
 # This Redis will be started by the docker-compose.yaml file at the top
 # level of the repository.
 redis_url: "redis://localhost:6379/0"
 redis_password_file: "examples/secrets/redis-password"
 
+# How long Gafaelfawr-issued tokens should live by default.
+token_lifetime_minutes: 1440 # 1 day
+
 # Where to send the user after logging out.
 after_logout_url: "http://localhost:8080"
 
-# Configuration for the JWTs issued by Gafaelfawr.
-issuer:
-  iss: "http://localhost:8080"
-  key_id: "localhost-key-id"
-  aud: "http://localhost"
-  key_file: "examples/secrets/issuer-key"
-  exp_minutes: 1440 # 1 day
-
 # To get these values, go to Settings > Developer Settings for either a
 # GitHub user or an organization, go into OAuth Apps, and create a new
 # application.  Set the homepage URL to http://localhost:8080/ and the
@@ -48,6 +39,18 @@ github:
   client_id: "<github-client-id>"
   client_secret_file: "examples/secrets/github-client-secret"
 
+# Configuration for the Gafaelfawr OpenID Connect server.
+oidc_server:
+  issuer: "http://localhost:8080"
+  key_id: "localhost-key-id"
+  audience: "http://localhost"
+  key_file: "examples/secrets/issuer-key"
+  secrets_file: "examples/secrets/oidc-clients.json"
+
+# Replace this user with your GitHub username.
+initial_admins:
+  - "example"
+
 # Sample values.  You can replace these with anything you want to use
 # for a scope.
 known_scopes: