notify-mainpid

command to notify MAINPID to systemd from ExecStartPost=

Status: Currently work-in-progress. An experiment to see if a technical solution works. This idea is currently half-baked.

This GitHub project was created to test an architecture idea for containers/podman#12778

Problem description

If you run a systemd system service with User= and type=notify, systemd will not allow

  std::string msg = std::format("MAINPID={}\n", mainpid);
  sd_pid_notify(systemd_exec_pid, 0, msg.c_str());

when mainpid is not in the expected cgroup.

systemd only allows such a MAINPID to be sent from root.

For more details, see the systemd Git commit message.

Suggested solution

Sketch:

1. systemd starts ExecStart=myapp /run/myservice.resultfile
2. myapp has an inherited file descriptor that originates from

   OpenFile=/run/no_user_access/myapp.service.mainpid:mainpidfile:truncate
   

3. myapp starts a child process to run the workload (in this example we use sleep 3600)
4. myapp writes the PID of the child process to the file descriptor named mainpidfile
5. myapp calls the function sd_notify(0, "READY=1");
6. systemd starts ExecStartPost=+notify-mainpid /run/no_user_access/myapp.service.mainpid /run/myapp.service.result. The process will be running as root because the first character of the ExecStartPost= value is a +
7. notify-mainpid reads the mainpid from the filepath that was passed as the first command-line argument.
8. notify-mainpid notifies the mainpid to systemd
9. notify-mainpid writes ok to the filepath that was passed as the second command-line argument.
10. myapp is watching the file /run/myapp.service.result with inotify and notices when it has been written to.

Demo

build and install software

g++ -std=c++20 notify-mainpid.cpp -lsystemd -o notify-mainpid
gcc myapp.c -lsystemd -o myapp
sudo cp notify-mainpid /usr/bin/notify-mainpid
sudo chcon --reference /usr/bin/cp /usr/bin/notify-mainpid
sudo chown root:root /usr/bin/notify-mainpid
sudo chmod 755 /usr/bin/notify-mainpid
sudo cp myapp /usr/bin/myapp
sudo chcon --reference /usr/bin/cp /usr/bin/myapp
sudo chown root:root /usr/bin/myapp
sudo chmod 755 /usr/bin/myapp

sudo useradd test
sudo cp myapp.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo mkdir /run/no_user_access
sudo chmod 700 /run/no_user_access

enable debug logging in systemd

kill -signal=SIGRTMIN+23 1

maybe it's required to turn off SELinux (TODO: check this)

sudo setenforce 0

(remember to turn on SELinux after testing this)

start the service

sudo systemctl start myapp.service

Result

The service is active but you will see a warning message:

Supervising process 6329 which is not our child. We'll most likely not notice when it exits.

[root@linux ~]# journalctl -xeu myapp.service | grep -A3  "MAINPID"
Aug 21 18:20:56 linux systemd[1]: myapp.service: Got notification message from PID 6330 (MAINPID=6329)
Aug 21 18:20:56 linux systemd[1]: myapp.service: New main PID 6329 belongs to service, we are happy.
Aug 21 18:20:56 linux systemd[1]: myapp.service: Supervising process 6329 which is not our child. We'll most likely not notice when it exits.
Aug 21 18:20:56 linux systemd[1]: myapp.service: Child 6330 belongs to myapp.service.

I now realize that myapp also starts up fine without notify-mainpid because the workload (sleep 3600) is not in an out-of-tree cgroup. Currently myapp is not a functional demo because of this. (I need to learn more about cgroups to be able to put the workload in an out-of-tree cgroup)

Maybe notify-mainpid is still useful for podman/conmon? I haven't tried it out yet. The OpenFile= file descriptor would have to be handled by Podman for example.

The only test I did with Podman was with some code that is pretty similar to notify-mainpid: containers/podman#12778 (comment) (but without OpenFile= and inotify)