Skip to content

Commit

Permalink
K8SPSMDB-780: Unsafe improvements (#1504)
Browse files Browse the repository at this point in the history 
* K8SPSMDB-780: Unsafe flags

These changes attempt to fix the overloaded `allowUnsafeConfigurations`
flag.

In previous implementation, `allowUnsafeConfigurations` wasn't just
allow unsafe configuration but make everything unsafe by disabling TLS,
allowing backups in unhealthy clusters, etc... without user's explicit
intent.

With these changes, we decouple those things from the unsafe flag and
remove all implicit behaviors. We introduce a new section called
`unsafeFlags`:

```
unsafeFlags:
  tls: false
  replsetSize: false
  mongosSize: false
  terminationGracePeriod: false
  backupIfUnhealthy: false
```

Starting from `v1.16.0`, `allowUnsafeConfigurations` is deprecated and
won't have any affect.

**TLS Mode**

This decoupling required a special attention to the TLS configuration.
Before these changes only way to disable TLS is setting
`allowUnsafeConfigurations` to true. Now, we introduce a new field:

```
spec:
  tls:
    mode: disabled
```

This field accepts the following values: `disabled`, `allowTLS`,
`preferTLS` and `requireTLS`.

If user sets mode to `disabled`, the operator will throw an error: `TLS
must be enabled. Set spec.unsafeFlags.tls to true to disable this
check.`

Since the use of TLS flags and reconciling TLS secrets depends on
`tls.mode` field, we need to block users to set `net.tls.mode` in custom
MongoDB configuration. If user sets a custom configuration like:

```
spec:
  replsets:
  - name: rs0
    size: 3
    configuration: |
      net:
        tls:
          mode: allowTLS
```

the operator will throw an error: `tlsMode must be set using spec.tls.mode`.

* fix tests

* fix tests

* fix tests

* fix tests

* fix tests

* fix pvc-resize

* fix custom-replset-name

* address review comments

* fix cluster deletion

* comment unsafe flags
  • Loading branch information
@egegunes
egegunes authored Apr 17, 2024
1 parent 5bc28b1 commit c325d88
Show file tree
Hide file tree
Showing 195 changed files with 980 additions and 486 deletions.
10 changes: 6 additions & 4 deletions build/pbm-entry.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,12 @@

PBM_MONGODB_URI="mongodb://${PBM_AGENT_MONGODB_USERNAME}:${PBM_AGENT_MONGODB_PASSWORD}@localhost:${PBM_MONGODB_PORT}/?replicaSet=${PBM_MONGODB_REPLSET}"

MONGO_SSL_DIR=/etc/mongodb-ssl
if [[ -f "${MONGO_SSL_DIR}/tls.crt" ]] && [[ -f "${MONGO_SSL_DIR}/tls.key" ]]; then
PBM_MONGODB_URI="${PBM_MONGODB_URI}&tls=true&tlsCertificateKeyFile=%2Ftmp%2Ftls.pem&tlsCAFile=${MONGO_SSL_DIR}%2Fca.crt&tlsInsecure=true"
cat "${MONGO_SSL_DIR}/tls.key" "${MONGO_SSL_DIR}/tls.crt" > /tmp/tls.pem
if [[ -z ${PBM_AGENT_TLS_ENABLED} ]] || [[ ${PBM_AGENT_TLS_ENABLED} == "true" ]]; then
MONGO_SSL_DIR=/etc/mongodb-ssl
if [[ -f "${MONGO_SSL_DIR}/tls.crt" ]] && [[ -f "${MONGO_SSL_DIR}/tls.key" ]]; then
PBM_MONGODB_URI="${PBM_MONGODB_URI}&tls=true&tlsCertificateKeyFile=%2Ftmp%2Ftls.pem&tlsCAFile=${MONGO_SSL_DIR}%2Fca.crt&tlsInsecure=true"
cat "${MONGO_SSL_DIR}/tls.key" "${MONGO_SSL_DIR}/tls.crt" >/tmp/tls.pem
fi
fi

export PBM_MONGODB_URI
Expand Down
177 changes: 76 additions & 101 deletions build/ps-entry.sh
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
#!/bin/bash
set -Eeuo pipefail
set -o xtrace

if [ "${1:0:1}" = '-' ]; then
set -- mongod "$@"
Expand Down Expand Up @@ -68,9 +69,9 @@ _mongod_hack_have_arg() {
local arg
for arg; do
case "$arg" in
"$checkArg" | "$checkArg"=*)
return 0
;;
"$checkArg" | "$checkArg"=*)
return 0
;;
esac
done
return 1
Expand All @@ -83,14 +84,14 @@ _mongod_hack_get_arg_val() {
local arg="$1"
shift
case "$arg" in
"$checkArg")
echo "$1"
return 0
;;
"$checkArg"=*)
echo "${arg#"$checkArg"=}"
return 0
;;
"$checkArg")
echo "$1"
return 0
;;
"$checkArg"=*)
echo "${arg#"$checkArg"=}"
return 0
;;
esac
done
return 1
Expand Down Expand Up @@ -131,14 +132,14 @@ _mongod_hack_ensure_no_arg_val() {
local arg="$1"
shift
case "$arg" in
"$ensureNoArg")
shift # also skip the value
continue
;;
"$ensureNoArg"=*)
# value is already included
continue
;;
"$ensureNoArg")
shift # also skip the value
continue
;;
"$ensureNoArg"=*)
# value is already included
continue
;;
esac
mongodHackedArgs+=("$arg")
done
Expand Down Expand Up @@ -282,10 +283,10 @@ if [ "$originalArgOne" = 'mongod' ]; then
# if we've got any /docker-entrypoint-initdb.d/* files to parse later, we should initdb
for f in /docker-entrypoint-initdb.d/*; do
case "$f" in
*.sh | *.js) # this should match the set of files we check for below
shouldPerformInitdb="$f"
break
;;
*.sh | *.js) # this should match the set of files we check for below
shouldPerformInitdb="$f"
break
;;
esac
done
fi
Expand Down Expand Up @@ -321,20 +322,6 @@ if [ "$originalArgOne" = 'mongod' ]; then
_mongod_hack_ensure_no_arg_val --replSet "${mongodHackedArgs[@]}"
fi

# "BadValue: need sslPEMKeyFile when SSL is enabled" vs "BadValue: need to enable SSL via the sslMode flag when using SSL configuration parameters"
tlsMode='disabled'
if _mongod_hack_have_arg '--tlsCertificateKeyFile' "${mongodHackedArgs[@]}"; then
tlsMode='preferTLS'
elif _mongod_hack_have_arg '--sslPEMKeyFile' "${mongodHackedArgs[@]}"; then
tlsMode='preferSSL'
fi
# 4.2 switched all configuration/flag names from "SSL" to "TLS"
if [ "$tlsMode" = 'preferTLS' ] || mongod --help 2>&1 | grep -q -- ' --tlsMode '; then
_mongod_hack_ensure_arg_val --tlsMode "$tlsMode" "${mongodHackedArgs[@]}"
else
_mongod_hack_ensure_arg_val --sslMode "$tlsMode" "${mongodHackedArgs[@]}"
fi

if stat "/proc/$$/fd/1" >/dev/null && [ -w "/proc/$$/fd/1" ]; then
# https://github.com/mongodb/mongo/blob/38c0eb538d0fd390c6cb9ce9ae9894153f6e8ef5/src/mongo/db/initialize_server_global_state.cpp#L237-L251
# https://github.com/docker-library/mongo/issues/164#issuecomment-293965668
Expand Down Expand Up @@ -396,17 +383,17 @@ if [ "$originalArgOne" = 'mongod' ]; then
echo
for f in /docker-entrypoint-initdb.d/*; do
case "$f" in
*.sh)
echo "$0: running $f"
# shellcheck source=/dev/null
. "$f"
;;
*.js)
echo "$0: running $f"
"${mongo[@]}" "$MONGO_INITDB_DATABASE" "$f"
echo
;;
*) echo "$0: ignoring $f" ;;
*.sh)
echo "$0: running $f"
# shellcheck source=/dev/null
. "$f"
;;
*.js)
echo "$0: running $f"
"${mongo[@]}" "$MONGO_INITDB_DATABASE" "$f"
echo
;;
*) echo "$0: ignoring $f" ;;
esac
echo
done
Expand All @@ -422,76 +409,64 @@ fi

if [[ $originalArgOne == mongo* ]]; then
mongodHackedArgs=("$@")
MONGO_SSL_DIR=${MONGO_SSL_DIR:-/etc/mongodb-ssl}
CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
if [ -f /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt ]; then
CA=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
fi
if [ -f "${MONGO_SSL_DIR}/ca.crt" ]; then
CA="${MONGO_SSL_DIR}/ca.crt"
fi
LDAP_SSL_DIR=${LDAP_SSL_DIR:-/etc/openldap/certs}
if [ -f "${LDAP_SSL_DIR}/ca.crt" ]; then
echo "TLS_CACERT ${LDAP_SSL_DIR}/ca.crt" >/etc/openldap/ldap.conf
fi
if [ -f "${MONGO_SSL_DIR}/tls.key" ] && [ -f "${MONGO_SSL_DIR}/tls.crt" ]; then
cat "${MONGO_SSL_DIR}/tls.key" "${MONGO_SSL_DIR}/tls.crt" >/tmp/tls.pem
_mongod_hack_ensure_arg_val --sslPEMKeyFile /tmp/tls.pem "${mongodHackedArgs[@]}"
if [ -f "${CA}" ]; then
_mongod_hack_ensure_arg_val --sslCAFile "${CA}" "${mongodHackedArgs[@]}"
fi

tlsMode=""
# if --tlsMode arg is present, get it
if _mongod_hack_have_arg --tlsMode "${mongodHackedArgs[@]}"; then
tlsMode="$(_mongod_hack_get_arg_val --tlsMode "${mongodHackedArgs[@]}")"
fi
MONGO_SSL_INTERNAL_DIR=${MONGO_SSL_INTERNAL_DIR:-/etc/mongodb-ssl-internal}
if [ -f "${MONGO_SSL_INTERNAL_DIR}/tls.key" ] && [ -f "${MONGO_SSL_INTERNAL_DIR}/tls.crt" ]; then
cat "${MONGO_SSL_INTERNAL_DIR}/tls.key" "${MONGO_SSL_INTERNAL_DIR}/tls.crt" >/tmp/tls-internal.pem
_mongod_hack_ensure_arg_val --sslClusterFile /tmp/tls-internal.pem "${mongodHackedArgs[@]}"
if [ -f "${MONGO_SSL_INTERNAL_DIR}/ca.crt" ]; then
_mongod_hack_ensure_arg_val --sslClusterCAFile "${MONGO_SSL_INTERNAL_DIR}/ca.crt" "${mongodHackedArgs[@]}"
fi

if [[ -z ${tlsMode} ]]; then
# if neither --tlsMode arg or net.tls.mode is present, set it to preferTLS
tlsMode="preferTLS"
fi

# don't add --tlsMode if allowUnsafeConfigurations is true
# don't add --tlsMode if TLS is disabled
if clusterAuthMode="$(_mongod_hack_get_arg_val --clusterAuthMode "${mongodHackedArgs[@]}")"; then
if [[ ${clusterAuthMode} != "keyFile" ]]; then
tlsMode="preferSSL"
# if --config arg is present, try to get tlsMode from it
if _parse_config "${mongodHackedArgs[@]}"; then
tlsMode=$(jq -r '.net.tls.mode // "preferSSL"' "${jsonConfigFile}")
fi
_mongod_hack_ensure_arg_val --sslMode "${tlsMode}" "${mongodHackedArgs[@]}"
_mongod_hack_ensure_arg_val --tlsMode "${tlsMode}" "${mongodHackedArgs[@]}"
else
_mongod_hack_ensure_no_arg --sslAllowInvalidCertificates "${mongodHackedArgs[@]}"
fi
fi

if [ "$MONGODB_VERSION" != 'v4.0' ]; then

_mongod_hack_rename_arg_save_val --sslMode --tlsMode "${mongodHackedArgs[@]}"

if _mongod_hack_have_arg '--tlsMode' "${mongodHackedArgs[@]}"; then
tlsMode="none"
if _mongod_hack_have_arg 'allowSSL' "${mongodHackedArgs[@]}"; then
tlsMode='allowTLS'
elif _mongod_hack_have_arg 'preferSSL' "${mongodHackedArgs[@]}"; then
tlsMode='preferTLS'
elif _mongod_hack_have_arg 'requireSSL' "${mongodHackedArgs[@]}"; then
tlsMode='requireTLS'
if [[ ${tlsMode} != "disabled" ]]; then
MONGO_SSL_DIR=${MONGO_SSL_DIR:-/etc/mongodb-ssl}
CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
if [ -f /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt ]; then
CA=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
fi
if [ -f "${MONGO_SSL_DIR}/ca.crt" ]; then
CA="${MONGO_SSL_DIR}/ca.crt"
fi
if [ -f "${MONGO_SSL_DIR}/tls.key" ] && [ -f "${MONGO_SSL_DIR}/tls.crt" ]; then
cat "${MONGO_SSL_DIR}/tls.key" "${MONGO_SSL_DIR}/tls.crt" >/tmp/tls.pem
_mongod_hack_ensure_arg_val --sslPEMKeyFile /tmp/tls.pem "${mongodHackedArgs[@]}"
if [ -f "${CA}" ]; then
_mongod_hack_ensure_arg_val --sslCAFile "${CA}" "${mongodHackedArgs[@]}"
fi

if [ "$tlsMode" != "none" ]; then
_mongod_hack_ensure_no_arg_val --tlsMode "${mongodHackedArgs[@]}"
_mongod_hack_ensure_arg_val --tlsMode "$tlsMode" "${mongodHackedArgs[@]}"
fi
MONGO_SSL_INTERNAL_DIR=${MONGO_SSL_INTERNAL_DIR:-/etc/mongodb-ssl-internal}
if [ -f "${MONGO_SSL_INTERNAL_DIR}/tls.key" ] && [ -f "${MONGO_SSL_INTERNAL_DIR}/tls.crt" ]; then
cat "${MONGO_SSL_INTERNAL_DIR}/tls.key" "${MONGO_SSL_INTERNAL_DIR}/tls.crt" >/tmp/tls-internal.pem
_mongod_hack_ensure_arg_val --sslClusterFile /tmp/tls-internal.pem "${mongodHackedArgs[@]}"
if [ -f "${MONGO_SSL_INTERNAL_DIR}/ca.crt" ]; then
_mongod_hack_ensure_arg_val --sslClusterCAFile "${MONGO_SSL_INTERNAL_DIR}/ca.crt" "${mongodHackedArgs[@]}"
fi
fi

_mongod_hack_rename_arg_save_val --sslPEMKeyFile --tlsCertificateKeyFile "${mongodHackedArgs[@]}"
if ! _mongod_hack_have_arg '--tlsMode' "${mongodHackedArgs[@]}"; then
if _mongod_hack_have_arg '--tlsCertificateKeyFile' "${mongodHackedArgs[@]}"; then
_mongod_hack_ensure_arg_val --tlsMode "preferTLS" "${mongodHackedArgs[@]}"
fi
LDAP_SSL_DIR=${LDAP_SSL_DIR:-/etc/openldap/certs}
if [ -f "${LDAP_SSL_DIR}/ca.crt" ]; then
echo "TLS_CACERT ${LDAP_SSL_DIR}/ca.crt" >/etc/openldap/ldap.conf
fi
fi

if [ "$MONGODB_VERSION" != 'v4.0' ]; then
_mongod_hack_rename_arg '--sslAllowInvalidCertificates' '--tlsAllowInvalidCertificates' "${mongodHackedArgs[@]}"
_mongod_hack_rename_arg '--sslAllowInvalidHostnames' '--tlsAllowInvalidHostnames' "${mongodHackedArgs[@]}"
_mongod_hack_rename_arg '--sslAllowConnectionsWithoutCertificates' '--tlsAllowConnectionsWithoutCertificates' "${mongodHackedArgs[@]}"
_mongod_hack_rename_arg '--sslFIPSMode' '--tlsFIPSMode' "${mongodHackedArgs[@]}"
_mongod_hack_rename_arg '--sslMode' '--tlsMode' "${mongodHackedArgs[@]}"

_mongod_hack_rename_arg_save_val --sslPEMKeyPassword --tlsCertificateKeyFilePassword "${mongodHackedArgs[@]}"
_mongod_hack_rename_arg_save_val --sslClusterFile --tlsClusterFile "${mongodHackedArgs[@]}"
Expand Down
15 changes: 15 additions & 0 deletions config/crd/bases/psmdb.percona.com_perconaservermongodbs.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17309,9 +17309,24 @@ spec:
required:
- name
type: object
mode:
type: string
type: object
unmanaged:
type: boolean
unsafeFlags:
properties:
backupIfUnhealthy:
type: boolean
mongosSize:
type: boolean
replsetSize:
type: boolean
terminationGracePeriod:
type: boolean
tls:
type: boolean
type: object
updateStrategy:
type: string
upgradeOptions:
Expand Down
15 changes: 15 additions & 0 deletions deploy/bundle.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17982,9 +17982,24 @@ spec:
required:
- name
type: object
mode:
type: string
type: object
unmanaged:
type: boolean
unsafeFlags:
properties:
backupIfUnhealthy:
type: boolean
mongosSize:
type: boolean
replsetSize:
type: boolean
terminationGracePeriod:
type: boolean
tls:
type: boolean
type: object
updateStrategy:
type: string
upgradeOptions:
Expand Down
4 changes: 3 additions & 1 deletion deploy/cr-minimal.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@ metadata:
spec:
crVersion: 1.16.0
image: perconalab/percona-server-mongodb-operator:main-mongod6.0
allowUnsafeConfigurations: true
unsafeFlags:
replsetSize: true
mongosSize: true
upgradeOptions:
apply: disabled
schedule: "0 2 * * *"
Expand Down
11 changes: 7 additions & 4 deletions deploy/cr.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ spec:
image: perconalab/percona-server-mongodb-operator:main-mongod7.0
imagePullPolicy: Always
# tls:
# mode: preferTLS
# # 90 days in hours
# certValidityDuration: 2160h
# issuerConf:
Expand All @@ -25,7 +26,12 @@ spec:
# - name: private-registry-credentials
# initImage: perconalab/percona-server-mongodb-operator:main
# initContainerSecurityContext: {}
allowUnsafeConfigurations: false
# unsafeFlags:
# tls: false
# replsetSize: false
# mongosSize: false
# terminationGracePeriod: false
# backupIfUnhealthy: false
updateStrategy: SmartUpdate
# ignoreAnnotations:
# - service.beta.kubernetes.io/aws-load-balancer-backend-protocol
Expand Down Expand Up @@ -71,9 +77,6 @@ spec:
# - host: 34.124.76.92
# # for more configuration fields refer to https://docs.mongodb.com/manual/reference/configuration-options/
# configuration: |
# net:
# tls:
# mode: preferTLS
# operationProfiling:
# mode: slowOp
# systemLog:
Expand Down
15 changes: 15 additions & 0 deletions deploy/crd.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17982,9 +17982,24 @@ spec:
required:
- name
type: object
mode:
type: string
type: object
unmanaged:
type: boolean
unsafeFlags:
properties:
backupIfUnhealthy:
type: boolean
mongosSize:
type: boolean
replsetSize:
type: boolean
terminationGracePeriod:
type: boolean
tls:
type: boolean
type: object
updateStrategy:
type: string
upgradeOptions:
Expand Down
Loading

0 comments on commit c325d88

Please sign in to comment.