Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: implements kms v2beta1 APIs #210

Merged
merged 1 commit into from
May 17, 2023
Merged

feat: implements kms v2beta1 APIs #210

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 74 additions & 24 deletions .pipelines/templates/e2e-kind-template.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,34 +5,34 @@ jobs:
workspace:
clean: all
variables:
- name: REGISTRY_NAME
value: kind-registry
- name: REGISTRY_PORT
value: 5000
- name: KUBERNETES_VERSION
value: v1.23.5
- name: KIND_CLUSTER_NAME
value: kms
- name: KIND_NETWORK
value: kind
# contains the following environment variables:
# - AZURE_CLIENT_ID
# - AZURE_CLIENT_SECRET
# - AZURE_TENANT_ID
# - KEYVAULT_NAME
# - KEY_NAME
# - KEY_VERSION
- group: kubernetes-kms
- name: REGISTRY_NAME
value: kind-registry
- name: REGISTRY_PORT
value: 5000
- name: KUBERNETES_VERSION
value: v1.27.1
- name: KIND_CLUSTER_NAME
value: kms
- name: KIND_NETWORK
value: kind
# contains the following environment variables:
# - AZURE_CLIENT_ID
# - AZURE_CLIENT_SECRET
# - AZURE_TENANT_ID
# - KEYVAULT_NAME
# - KEY_NAME
# - KEY_VERSION
- group: kubernetes-kms
strategy:
matrix:
kind_v1_23_13:
KUBERNETES_VERSION: v1.23.13
kind_v1_24_7:
kmsv1_kind_v1_24_7:
KUBERNETES_VERSION: v1.24.7
kind_v1_25_3:
kmsv1_kind_v1_25_3:
KUBERNETES_VERSION: v1.25.3
kind_v1_26_0:
kmsv1_kind_v1_26_0:
KUBERNETES_VERSION: v1.26.0
nilekhc marked this conversation as resolved.
Show resolved Hide resolved
kmsv1_kind_v1_27_1:
KUBERNETES_VERSION: v1.27.1
steps:
- task: GoTool@0
inputs:
Expand All @@ -53,5 +53,55 @@ jobs:
- template: cluster-health-template.yml
- template: kind-debug-template.yml
- script: make e2e-test
displayName: "Run e2e tests"
displayName: "Run e2e tests for KMS v1"
- template: cleanup-template.yml
- job:
timeoutInMinutes: 15
cancelTimeoutInMinutes: 5
workspace:
clean: all
variables:
- name: REGISTRY_NAME
value: kind-registry
- name: REGISTRY_PORT
value: 5000
- name: KUBERNETES_VERSION
value: v1.27.1
- name: KIND_CLUSTER_NAME
value: kms
- name: KIND_NETWORK
value: kind
# contains the following environment variables:
# - AZURE_CLIENT_ID
# - AZURE_CLIENT_SECRET
# - AZURE_TENANT_ID
# - KEYVAULT_NAME
# - KEY_NAME
# - KEY_VERSION
- group: kubernetes-kms
strategy:
matrix:
kmsv2_kind_v1_27_1:
KUBERNETES_VERSION: v1.27.1
steps:
- task: GoTool@0
inputs:
version: 1.19
- script: make e2e-install-prerequisites
displayName: "Install e2e test prerequisites"
- script: |
make e2e-kmsv2-setup-kind
displayName: "Setup kind cluster with azure kms plugin"
env:
REGISTRY_NAME: $(REGISTRY_NAME)
REGISTRY_PORT: $(REGISTRY_PORT)
KUBERNETES_VERSION: $(KUBERNETES_VERSION)
KIND_CLUSTER_NAME: $(KIND_CLUSTER_NAME)
KIND_NETWORK: $(KIND_NETWORK)
AZURE_CLIENT_ID: $(AZURE_CLIENT_ID)
AZURE_CLIENT_SECRET: $(AZURE_CLIENT_SECRET)
- template: cluster-health-template.yml
- template: kind-debug-template.yml
- script: make e2e-kmsv2-test
displayName: "Run e2e tests for KMS v2"
- template: cleanup-template.yml
13 changes: 11 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,8 @@ DOCKER_BUILDKIT = 1
export DOCKER_BUILDKIT

# Testing var
KIND_VERSION ?= 0.15.0
KUBERNETES_VERSION ?= v1.25.0
KIND_VERSION ?= 0.18.0
KUBERNETES_VERSION ?= v1.27.1
BATS_VERSION ?= 1.4.1

## --------------------------------------
Expand Down Expand Up @@ -132,6 +132,11 @@ e2e-setup-kind: setup-local-registry
./scripts/connect-registry.sh &
sleep 90s

e2e-kmsv2-setup-kind: setup-local-registry
./scripts/setup-kmsv2-kind-cluster.sh &
./scripts/connect-registry.sh &
sleep 90s
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible to watch/poll something instead of hardcoded sleep?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried with the wait but it didn't work. Though we do have a check in the CI for cluster health: https://github.com/Azure/kubernetes-kms/blob/master/.pipelines/templates/cluster-health-template.yml

Copy link
Member

@sozercan sozercan Apr 25, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we run both async and wait for pids to finish? similar to https://github.com/open-policy-agent/gatekeeper/blob/master/.github/workflows/release.yaml#L110-L120

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we run the process in the background ADO marks the task as completed

The error is:

The STDIO streams did not close within 10 seconds of the exit event from process '/bin/bash'. This may indicate a child process inherited the STDIO streams and has not yet exited.

 
I think this is the reason we have sleep, to hold on to the task for enough time so that the registry can be connected to the Kind cluster and created successfully. 

@aramase to add any additional context.


.PHONY: setup-local-registry
setup-local-registry:
./scripts/setup-local-registry.sh
Expand All @@ -148,3 +153,7 @@ e2e-delete-kind:
e2e-test:
# Run test suite with kind cluster
bats -t tests/e2e/test.bats

e2e-kmsv2-test:
# Run test suite with kind cluster
bats -t tests/e2e/testkmsv2.bats
5 changes: 4 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ Now that Azure KMS provider is running in your cluster and the encryption config
sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/certs/ca.crt --cert=/etc/kubernetes/certs/etcdclient.crt --key=/etc/kubernetes/certs/etcdclient.key get /registry/secrets/default/secret1
```

3. Check that the stored secret is prefixed with `k8s:enc:kms:v1:azurekmsprovider`. This indicates the Azure KMS provider has encrypted the data.
3. Check that the stored secret is prefixed with `k8s:enc:kms:v1:azurekmsprovider` when KMSv1 is used for encryption, or with `k8s:enc:kms:v2:azurekmsprovider` when KMSv2 is used. This prefix indicates that the data has been encrypted by the Azure KMS provider.

4. Verify the secret is decrypted correctly when retrieved via the Kubernetes API:

Expand All @@ -69,6 +69,9 @@ Now that Azure KMS provider is running in your cluster and the encryption config

Refer to [doc](docs/rotation.md) for steps to rotate the KMS Key on an existing cluster.

## Metrics
Refer to [doc](docs/metrics.md) for details on the metrics exposed by the KMS Key Vault plugin.

## Contributing

The KMS Plugin for Key Vault project welcomes contributions and suggestions. Please see [CONTRIBUTING](CONTRIBUTING.md) for details.
Expand Down
57 changes: 49 additions & 8 deletions cmd/server/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,18 @@ import (
"syscall"
"time"

"github.com/Azure/kubernetes-kms/pkg/config"
"github.com/Azure/kubernetes-kms/pkg/metrics"
"github.com/Azure/kubernetes-kms/pkg/plugin"
"github.com/Azure/kubernetes-kms/pkg/utils"
"github.com/Azure/kubernetes-kms/pkg/version"

"google.golang.org/grpc"
pb "k8s.io/apiserver/pkg/storage/value/encrypt/envelope/v1beta1"
nilekhc marked this conversation as resolved.
Show resolved Hide resolved
logsapi "k8s.io/component-base/logs/api/v1"
json "k8s.io/component-base/logs/json"
"k8s.io/klog/v2"
kmsv1 "k8s.io/kms/apis/v1beta1"
kmsv2 "k8s.io/kms/apis/v2"
)

var (
Expand Down Expand Up @@ -57,7 +59,12 @@ func main() {

if *logFormatJSON {
jsonFactory := json.Factory{}
logger, _ := jsonFactory.Create(logsapi.LoggingConfiguration{Format: "json"})
logger, _ := jsonFactory.Create(
nilekhc marked this conversation as resolved.
Show resolved Hide resolved
logsapi.LoggingConfiguration{
Format: "json",
},
logsapi.LoggingOptions{},
)
klog.SetLogger(logger)
}

Expand All @@ -80,7 +87,7 @@ func main() {

klog.InfoS("Starting KeyManagementServiceServer service", "version", version.BuildVersion, "buildDate", version.BuildDate)

pc := &plugin.Config{
pluginConfig := &plugin.Config{
KeyVaultName: *keyvaultName,
KeyName: *keyName,
KeyVersion: *keyVersion,
Expand All @@ -90,9 +97,25 @@ func main() {
ProxyPort: *proxyPort,
ConfigFilePath: *configFilePath,
}
kmsServer, err := plugin.New(pc)

azureConfig, err := config.GetAzureConfig(pluginConfig.ConfigFilePath)
if err != nil {
klog.ErrorS(err, "failed to create server")
klog.ErrorS(err, "failed to get azure config")
os.Exit(1)
}

kvClient, err := plugin.NewKeyVaultClient(
azureConfig,
pluginConfig.KeyVaultName,
pluginConfig.KeyName,
pluginConfig.KeyVersion,
pluginConfig.ProxyMode,
pluginConfig.ProxyAddress,
pluginConfig.ProxyPort,
pluginConfig.ManagedHSM,
)
if err != nil {
klog.ErrorS(err, "failed to create key vault client")
os.Exit(1)
}

Expand All @@ -112,23 +135,41 @@ func main() {
klog.ErrorS(err, "failed to listen", "addr", addr, "proto", proto)
os.Exit(1)
}

opts := []grpc.ServerOption{
grpc.UnaryInterceptor(utils.UnaryServerInterceptor),
}

s := grpc.NewServer(opts...)
aramase marked this conversation as resolved.
Show resolved Hide resolved
pb.RegisterKeyManagementServiceServer(s, kmsServer)

// register kms v1 server
kmsV1Server, err := plugin.NewKMSv1Server(kvClient)
if err != nil {
klog.ErrorS(err, "failed to create server")
os.Exit(1)
}
kmsv1.RegisterKeyManagementServiceServer(s, kmsV1Server)

// register kms v2 server
kmsV2Server, err := plugin.NewKMSv2Server(kvClient)
if err != nil {
klog.ErrorS(err, "failed to create kms V2 server")
os.Exit(1)
}
kmsv2.RegisterKeyManagementServiceServer(s, kmsV2Server)

klog.InfoS("Listening for connections", "addr", listener.Addr().String())
go func() {
if err := s.Serve(listener); err != nil {
klog.ErrorS(err, "failed to serve")
klog.ErrorS(err, "failed to serve kms server")
os.Exit(1)
}
}()

// Health check for kms v1 and v2
healthz := &plugin.HealthZ{
KMSServer: kmsServer,
KMSv1Server: kmsV1Server,
KMSv2Server: kmsV2Server,
HealthCheckURL: &url.URL{
Host: net.JoinHostPort("", strconv.FormatUint(uint64(*healthzPort), 10)),
Path: *healthzPath,
Expand Down
Loading