Kafka charm does not trust multi-certificate ca-chains (only the ca-cert is trusted)

[Thinking-Dragon]

ℹ️ TL ; DR

You can skip to Explanation and Solution down below if you want to go straight to what is happening.

Steps to reproduce

1. Download multi-cert-ca-chain.zip, it contains a simple Certificate Authority chain that I created using openssl

Specifically, you will find:

1. root.pem public root certificate
2. intermediate.pem public intermediate certificate
3. intermediate.key the private key of the intermediate certificate, you will use it to sign CSRs from Kafka
4. ca-chain-ir the ca-chain in standard order (intermediate > root)
5. ca-chain-ri the ca-chain in reverse order, the reason for this chain is that the manual-tls-certificates charm expects reverse order when calling the provide-certificate action. I have raised this with the TLS team and they have a fix that is waiting to be approved

2. Create kafka.yaml

applications:
  kafka:
    channel: 3/stable
    charm: kafka
    num_units: 3
    base: ubuntu@22.04
  manual-tls-certificates:
    charm: manual-tls-certificates
    channel: latest/stable
    num_units: 1
  external-ca:
    channel: latest/stable
    charm: tls-certificates-operator
    num_units: 1
    base: ubuntu@22.04
    options:
      generate-self-signed-certificates: True
  zookeeper:
    channel: 3/stable
    charm: zookeeper
    num_units: 3
    base: ubuntu@22.04
  data-integrator:
    channel: latest/stable
    charm: data-integrator
    num_units: 1
    base: ubuntu@22.04
    options:
      topic-name: "default"
      extra-user-roles: "admin"

base: ubuntu@22.04

relations:
- - zookeeper:certificates
  - manual-tls-certificates:certificates
- - kafka:certificates
  - manual-tls-certificates:certificates
- - kafka:zookeeper
  - zookeeper:zookeeper
- - kafka
  - data-integrator
- - external-ca:certificates
  - kafka:trusted-ca

3. Run juju deploy ./kafka.yaml and wait for the deployment to settle down. You should see six outstanding CSRs when you run juju status.
4. Run the following script to extract CSRs from the manual-tls-certificates charm:

⚠️ The following scripts expect the ~/deployment directory to exist, run mkdir ~/deployment or modify the scripts to point to another directory before running the scripts.

#!/bin/bash

if [[ "$#" -ne 1 ]]; then
  echo "Usage: extract-csrs.sh <deployment-juju-model>"
  exit 1
fi

deployment="$1"

mkdir -p ~/deployment/csrs/$deployment/to-sign
mkdir -p ~/deployment/csrs/$deployment/signed

csrs_json=$(juju run manual-tls-certificates/0 get-outstanding-certificate-requests --format=json | jq -r '.["manual-tls-certificates/0"].results.result')
echo "${csrs_json//\'/\"}" > ~/deployment/csrs/$deployment/csrs.json

for i in $(seq 0 $(jq 'length - 1' ~/deployment/csrs/$deployment/csrs.json));
do
  unit=$(jq -r ".[$i].unit_name" ~/deployment/csrs/$deployment/csrs.json | sed 's|/|_|g')
  yq -r ".[$i].csr" ~/deployment/csrs/$deployment/csrs.json > ~/deployment/csrs/$deployment/to-sign/$unit.csr
done

5. Run the following script to sign the CSRs using the ca-chain you downloaded on step 1:

⚠️ You will need to adapt to script to point to your ca-chain files on your computer (replace the [...] sections)

#!/bin/bash

if [[ "$#" -ne 1 ]]; then
  echo "Usage: sign-csrs.sh <deployment-juju-model>"
  exit 1
fi

deployment="$1"

for i in $(seq 0 $(jq 'length - 1' ~/deployment/csrs/$deployment/csrs.json));
do
  unit=$(jq -r ".[$i].unit_name" ~/deployment/csrs/$deployment/csrs.json | sed 's|/|_|g')
  relation_id=$(jq -r ".[$i].relation_id" ~/deployment/csrs/$deployment/csrs.json)

  openssl x509 -req \
    -in             ~/deployment/csrs/$deployment/to-sign/$unit.csr \
    -CA             [...]/intermediate.pem \
    -CAkey          [...]/intermediate.key \
    -out            ~/deployment/csrs/$deployment/signed/$unit.pem \
    -CAcreateserial \
    -sha256 \
    -days 365 \
    -copy_extensions copyall
done

6. Run the following script to provide the signed certificates back to Kafka and ZooKeeper

⚠️ You will need to adapt to script to point to your ca-chain files on your computer (replace the [...] sections)

#!/bin/bash

if [[ "$#" -ne 1 ]]; then
  echo "Usage: provide-signed-certificates.sh <deployment-juju-model>"
  exit 1
fi

deployment="$1"
tls_unit="0"

for i in $(seq 0 $(jq 'length - 1' ~/deployment/csrs/$deployment/csrs.json));
do
  unit=$(jq -r ".[$i].unit_name" ~/deployment/csrs/$deployment/csrs.json | sed 's|/|_|g')
  relation_id=$(jq -r ".[$i].relation_id" ~/deployment/csrs/$deployment/csrs.json)

  juju run -m $deployment manual-tls-certificates/$tls_unit provide-certificate \
    certificate-signing-request="$(base64 -w0 ~/deployment/csrs/$deployment/to-sign/$unit.csr)" \
    certificate="$(base64 -w0 ~/deployment/csrs/$deployment/signed/$unit.pem)" \
    ca-certificate="$(base64 -w0 [...]/ca-chain-ri.pem)"
done

7. Wait for the deployment to settle.

Expected behavior

When I run juju status, I should see all nodes active.

When I run nc $ZOOKEEPER_ID 2181, I should see statistics (latency, received, sent, connections, outstanding, etc.)
Indicating that Kafka nodes are up and running.

Actual behavior

When I run juju status, I see the ZooKeeper nodes go inactive and the Kafka machines waiting for ZooKeeper.

When I run nc $ZOOKEEPER_ID 2181, I see an error message saying that ZK is unavailable.

Explanation

The reason for this behavior is that when the charm sets the truststore here it trusts only the ca.pem file which comes from the ca-certificates argument through the :certificates relation. That works if the CSRs are signed with a root certificate, but as soon as you have intermediate certificates, the charm doesn't trust certificate other than the one you provide through ca-certificate.

Solution

When the charm sets the truststore here, it should not only trust ca.pem, it should also trust all of the certificates contained in the ca-chain argument from the :certificates relation.

Versions

Operating system: Ubuntu 22.04.5 LTS

Juju CLI: 3.5.4-genericlinux-amd64

Juju agent: 3.5.4

Charm revision: 149

LXD: 5.0.3

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-6260.

This message was autogenerated