Problem with docker container mongo

[janusn]

First of all, Let me thanks Jip-Hop's great effort for creating this magnificent script. Its tremendous usefulness cannot be overstated!

There are many other containers running in a jail. They all run happy besides occasional cifs docker volumes mount after the containers started.

Unfortunately, I run into a problem with running mongo docker container inside the jail. It always corrupts when a user is created in the database.

With a simple compose.yaml mentioned on the overview page:

---
services:

  mongo:
    image: mongo
    restart: always
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: example

Connecting to the mongodb always fails with authentication error:

docker exec -it mongo-mongo1 /bin/bash
root@aaa4517b8cc4:/# mongosh -u "root" -p "example" --authenticationDatabase "admin"
Current Mongosh Log ID:	667b290c77b4435ed2a26a12
Connecting to:		mongodb://<credentials>@127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&authSource=unifidb&appName=mongosh+2.2.6
MongoServerError: Authentication failed.

For the docker log, it seems the user root does not exist.

t={"$date":"2024-06-25T14:01:30.203+00:00"} s=I c=NETWORK id=51800 ctx=conn2 msg=client metadata attr={"remote":"127.0.0.1:41124","client":"conn2","negotiatedCompressors":[],"doc":{"application":{"name":"mongosh 2.2.6"},"driver":{"name":"nodejs|mongosh","version":"6.6.2|2.2.6"},"platform":"Node.js v20.12.2, LE","os":{"name":"linux","architecture":"x64","version":"3.10.0-327.22.2.el7.x86_64","type":"Linux"},"env":{"container":{"runtime":"docker"}}}}
t={"$date":"2024-06-25T14:01:30.204+00:00"} s=I c=ACCESS id=20251 ctx=conn2 msg=Supported SASL mechanisms requested for unknown user attr={"user":{"user":"root","db":"admin"}}
t={"$date":"2024-06-25T14:01:30.204+00:00"} s=I c=ACCESS id=6788604 ctx=conn2 msg=Auth metrics report attr={"metric":"acquireUser","micros":0}
t={"$date":"2024-06-25T14:01:30.204+00:00"} s=I c=ACCESS id=5286307 ctx=conn2 msg=Failed to authenticate attr={"client":"127.0.0.1:41124","isSpeculative":true,"isClusterMember":false,"mechanism":"SCRAM-SHA-256","user":"root","db":"admin","error":"UserNotFound: Could not find user \"root\" for db \"admin\"","result":11,"metrics":{"conversation_duration":{"micros":271,"summary":{"0":{"step":1,"step_total":2,"duration_micros":237}}}},"extraInfo":{}}
t={"$date":"2024-06-25T14:01:30.206+00:00"} s=I c=ACCESS id=6788604 ctx=conn2 msg=Auth metrics report attr={"metric":"acquireUser","micros":0}
t={"$date":"2024-06-25T14:01:30.206+00:00"} s=I c=ACCESS id=5286307 ctx=conn2 msg=Failed to authenticate attr={"client":"127.0.0.1:41124","isSpeculative":false,"isClusterMember":false,"mechanism":"SCRAM-SHA-1","user":"root","db":"admin","error":"UserNotFound: Could not find user \"root\" for db \"admin\"","result":11,"metrics":{"conversation_duration":{"micros":199,"summary":{"0":{"step":1,"step_total":2,"duration_micros":176}}}},"extraInfo":{}}

I have tried the same procedures on another ubuntu 24.04 machine. It worked faultlessly as expected.

Environment:
Host: Dragonfish-24.04.1.1
CPU: 2 x ntel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz
jail config:

$ sudo cat config
startup=1
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
# The docker_compatible option is deprecated and will be removed in a future release
docker_compatible=0
# Turning off seccomp filtering improves performance at the expense of security
seccomp=1

# Add additional systemd-nspawn flags
# E.g. to mount host storage in the jail (--bind-ro for readonly):
# --bind='/mnt/pool/dataset:/home' --bind-ro=/etc/certificates
# E.g. macvlan networking:
# --network-macvlan=eno1 --resolv-conf=bind-host
# E.g. bridge networking:
# --network-bridge=br1 --resolv-conf=bind-host
# E.g. allow syscalls required by docker:
# --system-call-filter='add_key keyctl bpf'
systemd_nspawn_user_args=--system-call-filter='add_key keyctl bpf'
	--network-bridge=br1
	--resolv-conf=bind-host
        --bind=/mnt/tank/cctv:/mnt/cctv
	--bind=/mnt/tank/docker:/docker
        --bind=/mnt/tank/infinityloop:/mnt/infinityloop
	--bind=/mnt/tank/logs:/mnt/logs
	--bind=/mnt/tank/media:/mnt/media
	--bind=/mnt/tank/nextcloud:/mnt/nextcloud
	--bind=/mnt/tank/private:/mnt/private
# Specify command/script to run on the HOST before starting the jail
# For example to load kernel modules and config kernel settings
pre_start_hook=
# pre_start_hook=#!/usr/bin/bash
#     set -euo pipefail
#     echo 'PRE_START_HOOK_EXAMPLE'
#     echo 1 > /proc/sys/net/ipv4/ip_forward
#     modprobe br_netfilter
#     echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
#     echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

# Specify a command/script to run on the HOST after stopping the jail
post_stop_hook=
# post_stop_hook=echo 'POST_STOP_HOOK_EXAMPLE'

# Only used while creating the jail
distro=debian
release=bookworm

# Specify command/script to run IN THE JAIL before the first start
# Useful to install packages on top of the base rootfs
# NOTE: this script will run in the host networking namespace and
# ignores all systemd_nspawn_user_args such as bind mounts
initial_setup=
# initial_setup=bash -c 'apt-get update && apt-get -y upgrade'

# Usually no need to change systemd_run_default_args
systemd_run_default_args=--collect
	--property=Delegate=yes
	--property=RestartForceExitStatus=133
	--property=SuccessExitStatus=133
	--property=TasksMax=infinity
	--property=Type=notify
	--setenv=SYSTEMD_NSPAWN_LOCK=0
	--property=KillMode=mixed

# Usually no need to change systemd_nspawn_default_args
systemd_nspawn_default_args=--bind-ro=/sys/module
	--boot
	--inaccessible=/sys/module/apparmor
	--quiet
	--keep-unit

[Jip-Hop]

Strange. I'd be surprised if this is caused by running inside a jail. Please try with a Ubuntu 24.04 rootfs image instead of the default debian one to make both your test cases more similar.

[janusn]

Yes, with seccomp=0 alone, a user can be created. It failed while I have enabled it again.

Regarding strace command, I am not very sure about how to use it.

I tried
# strace /usr/bin/mongosh 2>&1 |grep Operation\ not\ permitted
and
# strace /usr/bin/mongod 2>&1 |grep Operation\ not\ permitted
on both inside the jail and inside container.

Nothing was printed on all 4 combinations.

[Jip-Hop]

Maybe this helps? https://stackoverflow.com/questions/67782170/mongo-cli-drops-with-failed-to-mlock-cannot-allocate-locked-memory

[janusn]

Thank a ton! Your generous help is greatly appreciated!

I can finally add a user to mongodb successfully with "--capability=CAP_IPC_LOCK" added to systemd_nspawn_user_args.

Let's hope it works well for the remaining tasks.

[Jip-Hop]

Glad you got it working. According to the last link you may also need to add @memlock like this: systemd_nspawn_user_args=--system-call-filter='add_key keyctl bpf @memlock'. But at least you have an idea where to look now in case you run into errors. :)

[jm1tchell]

Thank you @janusn and @Jip-Hop! I've been beating my head against a wall for a week trying to figure out why mongo wasn't working.

[Lockszmith-GH]

It's been a couple of years since I dealt with mongodb. So I might be just throwing a wrench here in the conversation...

I recall having issues with running mongodb inside a docker container on a VM.
The root cause had to do with the underlying file-system.

May I suggest you use create a zvol, and have it formatted as ext4 inside your jail - I'm curious whether that will resolve the issue.

[janusn]

Thanks for your suggestion.
It might work but it is kinda complicated the situation. I rather keep it simple. And the pointer @Jip-Hop given me solved the issue so far.