From 5ba44de41fc8ebf9958ae87e7ae70298337eb614 Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Thu, 4 Apr 2024 08:20:49 +0200 Subject: [PATCH] Update docs about net-certmanager --- config/nav.yml | 2 +- .../encryption/cluster-local-domain-tls.md | 19 ++++- ...d => configure-certmanager-integration.md} | 77 +++++++++++-------- ...encryption-cluster-local-domain.drawio.svg | 2 +- .../encryption-external-domain.drawio.svg | 2 +- .../serving/encryption/encryption-overview.md | 4 +- .../encryption-system-internal.drawio.svg | 2 +- .../serving/encryption/external-domain-tls.md | 22 ++++-- .../serving/encryption/system-internal-tls.md | 20 ++--- 9 files changed, 96 insertions(+), 54 deletions(-) rename docs/serving/encryption/{install-and-configure-net-certmanager.md => configure-certmanager-integration.md} (84%) diff --git a/config/nav.yml b/config/nav.yml index 9209bd8652e..6d0be562761 100644 --- a/config/nav.yml +++ b/config/nav.yml @@ -159,7 +159,7 @@ nav: - Configure the Defaults ConfigMap: serving/configuration/config-defaults.md - Serving encryption configuration: - Overview: serving/encryption/encryption-overview.md - - Install and configure net-certmanager: serving/encryption/install-and-configure-net-certmanager.md + - Configure cert-manager integration: serving/encryption/configure-certmanager-integration.md - Configure external domain encryption: serving/encryption/external-domain-tls.md - Configure cluster-local domain encryption: serving/encryption/cluster-local-domain-tls.md - Configure Knative system-internal encryption: serving/encryption/system-internal-tls.md diff --git a/docs/serving/encryption/cluster-local-domain-tls.md b/docs/serving/encryption/cluster-local-domain-tls.md index f982db33164..6564fbf77cf 100644 --- a/docs/serving/encryption/cluster-local-domain-tls.md +++ b/docs/serving/encryption/cluster-local-domain-tls.md @@ -12,11 +12,16 @@ You must meet the following requirements to enable secure HTTPS connections: !!! warning This feature is currently only supported with Kourier and Istio as a networking layer. -## Enabling cluster-local-domain-tls -First, you need to install and configure `cert-manager` and `net-certmanager`. Please refer to [Installing and configuring net-certmanager](./install-and-configure-net-certmanager.md) for details. +## Installing and configuring cert-manager and integration + +First, you need to install and configure `cert-manager` and the Knative cert-manager integration. +Please refer to [Configuring Knative cert-manager integration](./configure-certmanager-integration.md) for details. + + +## Enabling cluster-local-domain-tls -Then, update the [`config-network` ConfigMap](https://github.com/knative/serving/blob/main/config/core/configmaps/network.yaml) in the `knative-serving` namespace to enable `cluster-local-domain-tls`: +To enable `cluster-local-domain-tls` update the [`config-network` ConfigMap](https://github.com/knative/serving/blob/main/config/core/configmaps/network.yaml) in the `knative-serving` namespace: 1. Run the following command to edit your `config-network` ConfigMap: @@ -38,6 +43,12 @@ Then, update the [`config-network` ConfigMap](https://github.com/knative/serving ... ``` +1. Restart the Knative Serving controller to start the Knative cert-manager integration: + + ```bash + kubectl rollout restart deploy/controller -n knative-serving + ``` + Congratulations! Knative is now configured to obtain and renew TLS certificates for cluster-local domains. @@ -73,5 +84,5 @@ status: A quick note on trust, all clients that call the cluster-local domain of a Knative Service need to trust the Certificate Authority that signed the certificates. This is out of scope of Knative, but needs to be addressed to ensure a working system. Especially, when a Certificate Authority performs a rotation of the CA or the intermediate certificates. Find more information on - [Install and configure net-certmanager](./install-and-configure-net-certmanager.md#managing-trust-and-rotation-without-downtime). + [Configuring Knative cert-manager integration](./configure-certmanager-integration.md#managing-trust-and-rotation-without-downtime). diff --git a/docs/serving/encryption/install-and-configure-net-certmanager.md b/docs/serving/encryption/configure-certmanager-integration.md similarity index 84% rename from docs/serving/encryption/install-and-configure-net-certmanager.md rename to docs/serving/encryption/configure-certmanager-integration.md index 62b9258f9d5..8d5198b2f97 100644 --- a/docs/serving/encryption/install-and-configure-net-certmanager.md +++ b/docs/serving/encryption/configure-certmanager-integration.md @@ -1,7 +1,7 @@ -# Installing and configuring net-certmanager +# Configuring Knative cert-manager integration Knative Serving relies on a bridging-component to use cert-manager for automated certificate provisioning. -If you intend to use that feature, you need to install net-certmanager. +If you intend to use that feature, you need to enable the Knative cert-manager integration. ## Prerequisites @@ -10,35 +10,20 @@ The following must be installed on your Knative cluster: - [Knative Serving](../../install/yaml-install/serving/install-serving-with-yaml.md). - [`cert-manager`](../../install/installing-cert-manager.md) version `1.0.0` or higher. - -## Installing net-certmanager - -net-certmanager can be installed using the YAML installation method: - -```bash -kubectl apply -f {{ artifact( repo="net-certmanager", file="release.yaml") }} -``` - !!! warning - Per default, this installs a self-signed cert-manager `ClusterIssuer` for an easy quick-start. - This issuer should **NOT BE USED IN PRODUCTION**! You can add a label filter to the command above - to remove the issuer from the installation and configure your issuer manually: `--selector knative.dev/issuer-install!="true"` - -Installing net-certmanager using the Knative operator is not yet supported. -But the operator can be configured to install additional manifests from YAML, -see [issue-950](https://github.com/knative/operator/issues/950#issuecomment-1032769274) for more info. + Make sure you have installed cert-manager. Otherwise, the Serving controller will not start up correctly. ## Issuer configuration -net-certmanager defines three references to [cert-manager issuers](https://cert-manager.io/docs/configuration/issuers/) to configure different CAs +The Knative cert-manager integration defines three references to [cert-manager issuers](https://cert-manager.io/docs/configuration/issuers/) to configure different CAs for the [three Knative Serving encryption features](./encryption-overview.md): * `issuerRef`: issuer for external-domain certificates used for ingress. * `clusterLocalIssuerRef`: issuer for cluster-local-domain certificates used for ingress. * `systemInternalIssuerRef`: issuer for certificates for system-internal-tls certificates used by Knative internal components. -Per default, Knative uses a self-signed `ClusterIssuer` and net-certmanager references that `ClusterIssuer` for all three configurations. +The following example uses a self-signed `ClusterIssuer` and the Knative cert-manager integration references that `ClusterIssuer` for all three configurations. As this **should not be used in production** (and does not support rotating the CA without downtime), you should think about which CA should be used for each use-case and how trust will be distributed to the clients calling the encrypted services. For the Knative system components, Knative provides a way to specify a bundle of CAs that should be trusted (more on this below). @@ -78,20 +63,52 @@ As this is also not possible via ACME protocol (DNS01/HTTP01), you need to confi ### Configuring issuers -1. Create your `Issuer` and/or `ClusterIssuer` +!!! warning + The self-signed cluster issuer should not be used in production, please see [Issuer configuration](#issuer-configuration) above for more information. -1. Ensure that your namespace `Issuer` or `ClusterIssuer` exists: +1. Create and apply the following self-signed `ClusterIssuer` to your cluster: - ```bash - kubectl get clusterissuer -o yaml + ```yaml + apiVersion: cert-manager.io/v1 + kind: ClusterIssuer + metadata: + name: selfsigned-cluster-issuer + spec: + selfSigned: {} + --- + apiVersion: cert-manager.io/v1 + kind: ClusterIssuer + metadata: + name: knative-selfsigned-issuer + spec: + ca: + secretName: knative-selfsigned-ca + --- + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: knative-selfsigned-ca + namespace: cert-manager # If you want to use it as a ClusterIssuer the secret must be in the cert-manager namespace. + spec: + secretName: knative-selfsigned-ca + commonName: knative.dev + usages: + - server auth + isCA: true + issuerRef: + kind: ClusterIssuer + name: selfsigned-cluster-issuer ``` + +1. Ensure that the `ClusterIssuer` is ready: + ```bash - kubectl get issuer -n -o yaml + kubectl get clusterissuer selfsigned-cluster-issuer -o yaml + kubectl get clusterissuer knative-selfsigned-issuer -o yaml ``` - Result: The `Status.Conditions` should include `Ready=True`. -1. Then reference it in the `config-certmanager` ConfigMap: +1. Then reference the `ClusterIssuer` in the `config-certmanager` ConfigMap: ```bash kubectl edit configmap config-certmanager -n knative-serving @@ -110,13 +127,13 @@ As this is also not possible via ACME protocol (DNS01/HTTP01), you need to confi data: issuerRef: | kind: ClusterIssuer - name: letsencrypt-http01-issuer + name: knative-selfsigned-issuer clusterLocalIssuerRef: | kind: ClusterIssuer - name: cluster-local-ca-issuer + name: knative-selfsigned-issuer systemInternalIssuerRef: | kind: ClusterIssuer - name: self-signed-issuer + name: knative-selfsigned-issuer ``` Ensure that the file was updated successfully: diff --git a/docs/serving/encryption/encryption-cluster-local-domain.drawio.svg b/docs/serving/encryption/encryption-cluster-local-domain.drawio.svg index cf3ddf17f52..a88267b654f 100644 --- a/docs/serving/encryption/encryption-cluster-local-domain.drawio.svg +++ b/docs/serving/encryption/encryption-cluster-local-domain.drawio.svg @@ -1,4 +1,4 @@ -



































Cluster
Cluster...






Namespace of ingress-controller
Namespace of ingress-controller...




















Namespace: user-2
Namespace: user-2...













Namespace: knative-serving
Namespace: knative-serving...
Kservice
with Queue-Proxy
Kservice...
Ingress Controller
(Kourier, Contour, Istio)
Ingress Controller...
trusts the CA
trusts the CA
create
create
Serving Controller
Serving Controller
Get Certificate
Get Certificate
populate
populate
net-certmanager
(via installed cert-manager)
net-certmanager...
read
read
Knative Certificate
visibility: cluster-local
dnsName: myapp.user-2.svc.cluster.local
Knative Certificate...
read
read
Secret
name: <id>
Secret...
Uses SNI to return the matching certificate
Uses SNI to r...
External CA
(e.g. Company CA, Vault, allowed to sign *.cluster.local)
External CA...
Namespace: user-1






Namespace: user-1...
User application
(Knative or non-Knative workload)
User application...
myapp.user-2.svc.cluster.local
myapp.user-2.svc.cluster.local
Knative Component
Knative Component
Kubernetes Secret
Kubernetes Secret
Knative Certificate CR
Knative Certificate CR
Traffic flow
Traffic flow
Text is not SVG - cannot display
\ No newline at end of file +



































Cluster
Cluster...






Namespace of ingress-controller
Namespace of ingress-controller...




















Namespace: user-2
Namespace: user-2...













Namespace: knative-serving
Namespace: knative-serving...
Kservice
with Queue-Proxy
Kservice...
Ingress Controller
(Kourier, Contour, Istio)
Ingress Controller...
trusts the CA
trusts the CA
create
create
Serving Controller
Serving Controller
Get Certificate
Get Certificate
populate
populate
via cert-manager
via cert-manager
read
read
Knative Certificate
visibility: cluster-local
dnsName: myapp.user-2.svc.cluster.local
Knative Certificate...
read
read
Secret
name: <id>
Secret...
Uses SNI to return the matching certificate
Uses SNI to r...
External CA
(e.g. Company CA, Vault, allowed to sign *.cluster.local)
External CA...
Namespace: user-1






Namespace: user-1...
User application
(Knative or non-Knative workload)
User application...
myapp.user-2.svc.cluster.local
myapp.user-2.svc.cluster.local
Knative Component
Knative Component
Kubernetes Secret
Kubernetes Secret
Knative Certificate CR
Knative Certificate CR
Traffic flow
Traffic flow
Text is not SVG - cannot display
\ No newline at end of file diff --git a/docs/serving/encryption/encryption-external-domain.drawio.svg b/docs/serving/encryption/encryption-external-domain.drawio.svg index ce9d2f726e1..9c7f2c7715c 100644 --- a/docs/serving/encryption/encryption-external-domain.drawio.svg +++ b/docs/serving/encryption/encryption-external-domain.drawio.svg @@ -1,4 +1,4 @@ -























Cluster
Cluster...






Namespace of ingress-controller
Namespace of ingress-controller...




















Namespace: user-2
Namespace: user-2...











Namespace: knative-serving
Namespace: knative-serving...
Kservice
with Queue-Proxy
Kservice...
Ingress Controller
(Kourier, Contour, Istio)
Ingress Controller...
myapp-user-2.example.com
myapp-user-2.example.com
trusts the
CA
trusts the...
Application/User
external to the cluster
Applic...
create
create
Serving Controller
Serving Controller
Get Certificate
Get Certificate
populate
populate
net-certmanager
(via installed cert-manager)
net-certmanager...
read
read
Knative Certificate
dnsName: myapp-
user-2.example.com
Knative Certificate...
read
read
Secret
name: <id>
Secret...
Uses SNI to return the matching certificate
Uses SNI to r...
External ACME CA
(e.g. Let's encrypt)
External ACME CA...
Knative Component
Knative Component
Kubernetes Secret
Kubernetes Secret
Knative Certificate CR
Knative Certificate CR
Traffic flow + CN/SAN used
Traffic flow + CN/SAN...
Text is not SVG - cannot display
\ No newline at end of file +























Cluster
Cluster...






Namespace of ingress-controller
Namespace of ingress-controller...




















Namespace: user-2
Namespace: user-2...











Namespace: knative-serving
Namespace: knative-serving...
Kservice
with Queue-Proxy
Kservice...
Ingress Controller
(Kourier, Contour, Istio)
Ingress Controller...
myapp-user-2.example.com
myapp-user-2.example.com
trusts the
CA
trusts the...
Application/User
external to the cluster
Applic...
create
create
Serving Controller
Serving Controller
Get Certificate
Get Certificate
populate
populate
via cert-manager
via cert-manager
read
read
Knative Certificate
dnsName: myapp-
user-2.example.com
Knative Certificate...
read
read
Secret
name: <id>
Secret...
Uses SNI to return the matching certificate
Uses SNI to r...
External ACME CA
(e.g. Let's encrypt)
External ACME CA...
Knative Component
Knative Component
Kubernetes Secret
Kubernetes Secret
Knative Certificate CR
Knative Certificate CR
Traffic flow + CN/SAN used
Traffic flow + CN/SAN...
Text is not SVG - cannot display
\ No newline at end of file diff --git a/docs/serving/encryption/encryption-overview.md b/docs/serving/encryption/encryption-overview.md index f2c4e640c60..c02bf88a4a2 100644 --- a/docs/serving/encryption/encryption-overview.md +++ b/docs/serving/encryption/encryption-overview.md @@ -34,7 +34,7 @@ See [Configure external domain encryption](./external-domain-tls.md) for more in * Certificate CN/SAN contains the cluster-local domains of a Knative Service, e.g. `myapp.namespace.svc.cluster.local`, `myapp.namespace.svc`, `myapp.namespace`. * The certificates are hosted using SNI by the cluster-local endpoint of the ingress-controller. * The caller has to trust the CA that signed the certificates (this is out of the scope of Knative). One option to do this is using [trust-manager](https://cert-manager.io/docs/trust/trust-manager/) from cert-manager. -* To create the certificates, Knative relies on [cert-manager](https://cert-manager.io/) and our bridging component [net-certmanager](https://github.com/knative-extensions/net-certmanager/). They need to be installed and configured for the feature to work. +* To create the certificates, Knative relies on [cert-manager](https://cert-manager.io/) and the Knative cert-manager integration. They need to be installed and configured for the feature to work. See [Configure cluster-local domain encryption](./cluster-local-domain-tls.md) for more information on this feature. @@ -44,7 +44,7 @@ See [Configure cluster-local domain encryption](./cluster-local-domain-tls.md) f Knative system internal components (Ingress-Controller, Activator, Queue-Proxy) are hosting TLS endpoints when this configuration is enabled. -* To get the certificates, Knative relies on [cert-manager](https://cert-manager.io/) and our bridging component [net-certmanager](https://github.com/knative-extensions/net-certmanager/). They need to be installed and configured for the feature to work. +* To create the certificates, Knative relies on [cert-manager](https://cert-manager.io/) and the Knative cert-manager integration. They need to be installed and configured for the feature to work. * Specific SANs are used to verify each connection. Each component needs to trust the CA (possibly the full chain) that signed the certificates. For this, Knative system components will consume and trust a provided `CABundle`. The CA bundle needs to be provided by the cluster administrator, possibly using [trust-manager](https://cert-manager.io/docs/trust/trust-manager/) from cert-manager. See [Configure Knative system-internal encryption](./system-internal-tls.md) for more information on this feature. diff --git a/docs/serving/encryption/encryption-system-internal.drawio.svg b/docs/serving/encryption/encryption-system-internal.drawio.svg index 29fc69436c0..c49f129d736 100644 --- a/docs/serving/encryption/encryption-system-internal.drawio.svg +++ b/docs/serving/encryption/encryption-system-internal.drawio.svg @@ -1,4 +1,4 @@ -

































Cluster
Cluster...
Namespace of ingress-controller






Namespace of ingress-controller...












Namespace: user-2
Namespace: user-2...





















Namespace: knative-serving
Namespace: knative-serving...
Activator
Activator
Kservice
with Queue-Proxy
Kservice...
Ingress Controller
(Kourier, Contour, Istio)
Ingress Controller...
populate
populate
Serving Controller
Serving Controller
Secret
name: serving-certs
Secret...
use
use
Secret
name: routing-serving-certs
Secret...
populate
populate
create
create
use
use
trust
trust
Knative Component
Knative Component
Kubernetes Secret
Kubernetes Secret
Traffic flow
Traffic flow
net-certmanager
(via installed cert-manager)
net-certmanager...
External CA
(e.g. Company CA, Vault, allowed to sign *.cluster.local)
External CA...
Get Certificate
Get Certificate
Text is not SVG - cannot display
\ No newline at end of file +

































Cluster
Cluster...
Namespace of ingress-controller






Namespace of ingress-controller...












Namespace: user-2
Namespace: user-2...





















Namespace: knative-serving
Namespace: knative-serving...
Activator
Activator
Kservice
with Queue-Proxy
Kservice...
Ingress Controller
(Kourier, Contour, Istio)
Ingress Controller...
populate
populate
Serving Controller
Serving Controller
Secret
name: serving-certs
Secret...
use
use
Secret
name: routing-serving-certs
Secret...
populate
populate
create
create
use
use
trust
trust
Knative Component
Knative Component
Kubernetes Secret
Kubernetes Secret
Traffic flow
Traffic flow
via cert-manager
via cert-manager
External CA
(e.g. Company CA, Vault, allowed to sign *.cluster.local)
External CA...
Get Certificate
Get Certificate
Text is not SVG - cannot display
\ No newline at end of file diff --git a/docs/serving/encryption/external-domain-tls.md b/docs/serving/encryption/external-domain-tls.md index 9e6c557efe0..ca2ef3b8663 100644 --- a/docs/serving/encryption/external-domain-tls.md +++ b/docs/serving/encryption/external-domain-tls.md @@ -21,13 +21,14 @@ You must meet the following requirements to enable secure HTTPS connections: ## Automatically obtain and renew certificates -### Installing and configuring net-certmanager +### Installing and configuring cert-manager and integration !!! info If you want to use HTTP-01 challenge, you need to configure your custom domain to map to the IP of ingress. You can achieve this by adding a DNS A record to map the domain to the IP according to the instructions of your DNS provider. -First, you need to install and configure `cert-manager` and `net-certmanager`. Please refer to [Installing and configuring net-certmanager](./install-and-configure-net-certmanager.md) for details. +First, you need to install and configure `cert-manager` and the Knative cert-manager integration. +Please refer to [Configuring Knative cert-manager integration](./configure-certmanager-integration.md) for details. ### Configuring Knative Serving @@ -43,13 +44,13 @@ Only one of them can be active at the same time! Update the [`config-network` ConfigMap](https://github.com/knative/serving/blob/main/config/core/configmaps/network.yaml) in the `knative-serving` namespace to enable `external-domain-tls`: -1. Run the following command to edit your `config-network` ConfigMap: +1. Run the following command to edit your `config-network` ConfigMap: ```bash kubectl edit configmap config-network -n knative-serving ``` -1. Add the `external-domain-tls: Enabled` attribute under the `data` section: +1. Add the `external-domain-tls: Enabled` attribute under the `data` section: ```yaml apiVersion: v1 @@ -63,6 +64,12 @@ Update the [`config-network` ConfigMap](https://github.com/knative/serving/blob/ ... ``` +1. Restart the Knative Serving controller to start the Knative cert-manager integration: + + ```bash + kubectl rollout restart deploy/controller -n knative-serving + ``` + #### Using one wildcard certificate per namespace !!! warning @@ -105,6 +112,11 @@ kubectl patch --namespace knative-serving configmap config-network -p '{"data": For more details on namespace selectors, see [the Kubernetes documentation](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors). +Restart the Knative Serving controller to start the Knative cert-manager integration: + +```bash +kubectl rollout restart deploy/controller -n knative-serving +``` Congratulations! Knative is now configured to obtain and renew TLS certificates. When your TLS Certificate is issued and available on your cluster, your Knative services will @@ -277,7 +289,7 @@ Use the following steps in the relevant tab to add your certificate to your Knat A quick note on trust, all clients that call the external domain of a Knative Service need to trust the Certificate Authority that signed the certificates. This is out of scope of Knative, but needs to be addressed to ensure a working system. Especially, when a Certificate Authority performs a rotation of the CA or the intermediate certificates. Find more information on - [Install and configure net-certmanager](./install-and-configure-net-certmanager.md#managing-trust-and-rotation-without-downtime). + [Configuring Knative cert-manager integration](./configure-certmanager-integration.md#managing-trust-and-rotation-without-downtime). ## Additional configuration diff --git a/docs/serving/encryption/system-internal-tls.md b/docs/serving/encryption/system-internal-tls.md index 98b5bb35057..c3d01a8bc11 100644 --- a/docs/serving/encryption/system-internal-tls.md +++ b/docs/serving/encryption/system-internal-tls.md @@ -12,11 +12,15 @@ You must meet the following requirements to enable secure HTTPS connections: !!! warning This feature is currently only supported with Kourier as a networking layer. -## Enabling system-internal-tls +### Installing and configuring cert-manager and integration + +First, you need to install and configure `cert-manager` and the Knative cert-manager integration. +Please refer to [Configuring Knative cert-manager integration](./configure-certmanager-integration.md) for details. -First, you need to install and configure `cert-manager` and `net-certmanager`. Please refer to [Installing and configuring net-certmanager](./install-and-configure-net-certmanager.md) for details. -Then, update the [`config-network` ConfigMap](https://github.com/knative/serving/blob/main/config/core/configmaps/network.yaml) in the `knative-serving` namespace to enable `system-internal-tls`: +## Enabling system-internal-tls + +To enable `system-internal-tls` update the [`config-network` ConfigMap](https://github.com/knative/serving/blob/main/config/core/configmaps/network.yaml) in the `knative-serving` namespace: 1. Run the following command to edit your `config-network` ConfigMap: @@ -38,13 +42,11 @@ Then, update the [`config-network` ConfigMap](https://github.com/knative/serving ... ``` -1. Restart the Activator component - - !!! warning - When the `system-internal-tls` feature is activated, the Activator component needs to be restarted to reconfigure its internal web server, as this is not possible during runtime. +1. Restart the Knative activator and controller component to start the Knative cert-manager integration: ```bash - kubectl rollout restart deploy/activator -n knative-serving + kubectl rollout restart deploy/activator -n knative-serving + kubectl rollout restart deploy/controller -n knative-serving ``` Congratulations! Knative will now use TLS between its internal system components (Ingress-Controller, Activator and Queue-Proxy). @@ -76,7 +78,7 @@ Congratulations! Knative will now use TLS between its internal system components !!! warning A quick note on trust, Knative will automatically trust the CA that signed the Certificates, if the cert-manager issuer allows putting the CA directly in the field `ca.crt` of the certificates `Secret`. Regardless of that, Cluster admins **should always** - provide a trust-bundle, as described in [Install and configure net-certmanager](./install-and-configure-net-certmanager.md#managing-trust-and-rotation-without-downtime). + provide a trust-bundle, as described in [Configuring Knative cert-manager integration](./configure-certmanager-integration.md#managing-trust-and-rotation-without-downtime). This is also strongly recommended in the [cert-manager documentation](https://cert-manager.io/docs/trust/trust-manager/#cert-manager-integration-intentionally-copying-ca-certificates) to avoid issues with rotation.