acm-certificate-agent

K8s operator that automatically imports TLS certificates (such as those generated by cert-manager) into AWS Certificate Manager (ACM)

Description

The operator performs the following functions:

• Automatically imports (and maintains) SSL certificates stored as K8s Secrets (core/Secret) into ACM.
• Automatically adds annotations to ALB-enabled K8s Ingresses (networking.k8s.io/Ingress) that use HTTPS to consume ACM certificates.

For more information about to configure resources to use the agent, see Using in Kubernetes, below.

Operator scope

acm-certificate-agent does not scope its activities to the namespace in which it is installed. You do not need to (and should not) install copies into different namespaces within a single cluster.

A single installation will manage ACM certificate import and Ingress linking across an entire cluster.

Cluster installation

acm-certificate-agent consists of two components:

• An image containing the operator
• A Helm chart containing the K8s deployment

Both must be installed to use the operator within a cluster.

Prerequisites

You will need:

• An AWS EKS cluster with at least one node group, and the ARN associated with this cluster.

• An AWS ECR repository into which to deploy the acm-certificate-agent operator image, and the URI associated with this repository. The URI can be obtained from the AWS web console or CLI. Note that the AWS ECR repository does not need to be in the same region as the cluster.

• Local installations of golang, kubectl, aws-cli and helm. On Windows, these should be installed within WSL.

• Local installation of script-runner, if intending to use scripted IAM role/policy creation.

• A Kubernetes namespace into which to install the operator. This namespace must exist and be specified using the {NAMESPACE} parameter below.

  NOTE: The .kubeconfig associated with the WSL kubectl is NOT the same as the one used in Windows. Verify cluster access within WSL using kubectl config get-contexts and, if necessary, add the required context using e.g. aws --region {aws.region} eks update-kubeconfig --name {cluster.name}.

Installation procedure

1. Create an IAM role and associated policy that grants permission for the relevant ACM operations. Note the ARN of the role that is created.

   • Script runner can automate this task:

       script-runner scripts\acmCertificateAgent-prepare-config -p "cluster.arn:{CLUSTER_ARN}"
   

   The value returned as acmCertificateAgent.iam.serviceRole.arn should be used as the ROLE_ARN below.

   • To do this manually, follow the instructions at https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html using the trust policy template scripts\_resources\acmCertificateAgent-iam-role-trust-policy.template.
   
   

2. Build and push the operator image to ECR.

   NOTE: This step is not required if the operator image has previously been deployed to your AWS account.

   NOTE: On Windows, run this command within WSL.

       make docker-build docker-push REPO_URI={REPOSITORY_URI}

   
   

3. Deploy the operator to the cluster.

   NOTE: On Windows, run this command within WSL.

       make deploy REPO_URI={REPOSITORY_URI} CLUSTER_ARN={CLUSTER_ARN} ROLE_ARN={SERVICE_ROLE_ARN} NAMESPACE={NAMESPACE}

   Where {Namespace} is the name of the k8s namespace into which the operator should be installed. This namespace must exist.

   NOTE: Run make --help for more information on all potential make targets.

   Existing worker nodes should be processed and their corresponding EC2 instance names updated automatically. You can view these names using e.g. the AWS EC2 web console or CLI.

Using in Kubernetes

Core function 1: Automating ACM certificate import

To enable ACM certificate import, you can place an annotation on either Certificate or Secret resources.

• Certificates (cert-manager.io/Certificate)

  To enable ACM import for a cert-manager certificate, add the following annotation to its definition:

  acm-certificate-agent.validitron.io/enabled: 'true'

  The Secret containing the actual SSL certificate associated with this Certificate resource will be automatically imported into ACM.

• Secrets (core/Secret)

  NOTE: If the Secret is being managed by a cert-manager Certificate resource, you should not configure the Secret directly but rather annotate the Certificate instead (see above). This will ensure that if the Secret is deleted/recreated by cert-manager (for example, when the certificate is re-issued), agent configuration persists and ACM sychronisation continues without interruption.

  To enable ACM import for a SSL certificate, add the following annotation to its definition:

  acm-certificate-agent.validitron.io/enabled: 'true'

  Set the value to false to disable ACM import. Any existing ACM certificates will not be removed.

Core function 2: Automating explicit ALB ingress ACM certificate assignment

NOTE: ALB can automatically detect the correct certificate(s) to assign to Load Balancer routes, independently of acm-certificate-agent. This may be preferable to explicitly setting the ARN using acm-certificate-agent. To use auto-configuration, simply set the certificate-arn annotation on the Ingress to an empty string (and ignore the acm agent instructions below.)

    alb.ingress.kubernetes.io/certificate-arn: ''

If you want to explicitly assign an SSL certificate to an ALB Ingress (networking.k8s.io/Ingress), add the following annotation to its definition:

acm-certificate-agent.validitron.io/enabled: 'true'

An alb.ingress.kubernetes.io/certificate-arn annotation corresponding to a compatible SSL certificate will be added to the definition provided that the following preconditions are satisfied:

• The Ingress is marked as using ALB with the annotation kubernetes.io/ingress.class: 'alb'.
• The Ingress is marked as using HTTPS with the annotation alb.ingress.kubernetes.io/listen-ports containing at least one entry marked 'HTTPS'.

For more information about ALB annotations/configuration see https://kubernetes-sigs.github.io/aws-load-balancer-controller/v1.1/guide/ingress/annotation

The agent will select the first certificate(s) that is/are capable of providing SSL to the host name(s) specified in the Ingress. If ACM contains multiple certificates that support a given domain (after discounting expired and invalid certificates), which certificate will be selected cannot be guaranteed.

If the Ingress contains multiple routes that need more than one certificate to serve them, the agent will try to find all the required certificates. If one or more certificates cannot be found, the ARNs of those that have been found will be added to the annotation, and the agent will keep retrying until all the certificates can be matched.

Configuration options

Either or both of certificate import and ingress configuration can be disabled by configuring the acm-certificate-agent configmap associated with the deployment.

Uninstallation

Remove the operator from the cluster using:

    make undeploy CLUSTER_ARN={CLUSTER_ARN}

Any existing ACM certificates and resource annotations will not be removed during uninstallation and must be cleaned up manually.

Remarks

• acm-certificate-agent will never delete ACM certificates, even if they have expired. If import is enabled and a new certificate-agent certificate is found, then this will be imported alongside any existing certificates. If you are using automatic binding with ALB (see Core function 2, above), ALB will always select a valid/in-date certificate over an invalid/expired one. However if there are multiple valid certificates in ACM (for example, if a new certificate is issued before the expiry date of the previous one), then the ACM certificate that is selected for load balancing may not match the current cert-manager certificate within K8s.
• If a user manually removes acm-certificate-agent annotations from a Secret but its managing cert-manager Certificate resource still has an 'acm-certificate-agent/enabled' = true annotation, then eventually the Secret will be reconfigured (via certificate_controller) as agent-managed (and decorated with the appropriate annotations.) This is by design and happens because operators periodically run even if there are no changes to the target manifests.

How it works

This project uses the Kubernetes Operator pattern

It was built from a kubebuilder project and modified to use Helm.

It uses Controllers which provides a reconcile function responsible for synchronizing resources until the desired state is reached on the cluster.

More information can be found via the Kubebuilder Documentation

To support internal book-keeping, the agent automatically adds annotations to managed Secret objects. These should not be modified.

• acm-certificate-agent.validitron.io/certificate-arn
• acm-certificate-agent.validitron.io/domains
• acm-certificate-agent.validitron.io/expires
• acm-certificate-agent.validitron.io/inherits-from
• acm-certificate-agent.validitron.io/serial-number

Debugging

To debug the Helm chart and inspect the intermediate yaml file that is created run:

make helm-debug

Output will be generated as debug.yaml