forked from k3s-io/k3s
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add ADR for support for etcd s3 config secret (k3s-io#9364)
* Add ADR for etcd s3 config secret * Update ADR with changes from initial review Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
- Loading branch information
Showing
1 changed file
with
75 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,75 @@ | ||
# Support etcd Snapshot Configuration via Kubernetes Secret | ||
|
||
Date: 2024-02-06 | ||
|
||
## Status | ||
|
||
Accepted | ||
|
||
## Context | ||
|
||
### Current State | ||
|
||
K3s currently reads configuration for S3 storage of etcd snapshots from CLI flags and/or configuration files. | ||
|
||
Security-conscious users have raised issue with the current state. They want to store snapshots on S3, but do not want | ||
to have credentials visible in config files or systemd units. Users operating in highly secure environments have also | ||
asked for the ability to configure a proxy server to be used when creating/restoring snapshots stored on S3, without | ||
managing complicated `NO_PROXY` settings or affecting the rest of the K3s process environment. | ||
|
||
### Security Considerations | ||
|
||
Storing credentials on-disk is generally considered a bad idea, and is not allowed by security practices in many | ||
organizations. Use of static credentials in the config file also makes them difficult to rotate, as K3s only reloads the | ||
configuration on startup. | ||
|
||
### Existing Work | ||
|
||
Cloud-providers and other tools that need to auth to external systems frequently can be configured to retrieve secrets | ||
from an existing credential secret that is provisioned via an external process, such as a secrets management tool. This | ||
avoids embedding the credentials directly in the system configuration, chart values, and so on. | ||
|
||
## Decision | ||
|
||
* We will add a `--etcd-s3-proxy` flag that can be used to set the proxy used by the S3 client. This will override the | ||
settings that golang's default HTTP client reads from the `HTTP_PROXY/HTTPS_PROXY/NO_PROXY` environment varibles. | ||
* We will add support for reading etcd snapshot S3 configuration from a Secret. The secret name will be specified via a new | ||
`--etcd-s3-secret` flag, which accepts the name of the Secret in the `kube-system` namespace. | ||
* Presence of the `--etcd-s3-secret` flag does not imply `--etcd-s3`. If S3 is not enabled by use of the `--etcd-s3` flag, | ||
the Secret will not be used. | ||
* The Secret does not need to exist when K3s starts; it will be checked for every time a snapshot operation is performed. | ||
* Secret and CLI/config values will NOT be merged. The Secret will provide values to be used in absence of other | ||
configuration; if S3 configuration is passed via CLI flags or configuration file, ALL fields set by the Secret | ||
will be ignored. | ||
* The Secret will ONLY be used for on-demand and scheduled snapshot save operations; it will not be used by snapshot restore. | ||
Snapshot restore operations that want to retrieve a snapshot from S3 will need to pass the appropriate configuration | ||
via environment variables or CLI flags, as the Secret is not available during the restore process. | ||
|
||
Fields within the Secret will match `k3s server` CLI flags / config file keys. For the `etcd-s3-endpoint-ca`, which | ||
normally contains the path of a file on disk, the `etcd-s3-endpoint-ca` field can specify an inline PEM-encoded CA | ||
bundle, or the `etcd-s3-endpoint-ca-name` can be used to specify the name of a ConfigMap in the `kube-system` namespace | ||
containing one or more CA bundles. All valid CA bundles found in either field are loaded. | ||
|
||
```yaml | ||
apiVersion: v1 | ||
kind: Secret | ||
metadata: | ||
name: k3s-etcd-snapshot-s3-config | ||
namespace: kube-system | ||
stringData: | ||
etcd-s3-endpoint: "" | ||
etcd-s3-endpoint-ca: "" | ||
etcd-s3-endpoint-ca-name: "" | ||
etcd-s3-skip-ssl-verify: "false" | ||
etcd-s3-access-key: "AWS_ACCESS_KEY_ID" | ||
etcd-s3-secret-key: "AWS_SECRET_ACCESS_KEY" | ||
etcd-s3-bucket: "bucket" | ||
etcd-s3-region: "us-east-1" | ||
etcd-s3-insecure: "false" | ||
etcd-s3-timeout: "5m" | ||
etcd-s3-proxy: "" | ||
``` | ||
## Consequences | ||
This will require additional documentation, tests, and QA work to validate use of secrets for s3 snapshot configuration. |