rsyslog 8.33.0: logrotate config still tries to reference PID file, but rsyslog is started with the option to not create a PID file

[atc0005]

Short version:

1. The PID file doesn't exist
2. rsyslog was told not to create one
3. The logrotate config file expects to use one

Details:

From the /etc/logrotate.d/syslog file on a CentOS 7 system:

/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    missingok
    sharedscripts
    postrotate
	/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

[root@centos7-test init.d]# ls -l /var/run/syslogd.pid

ls: cannot access /var/run/syslogd.pid: No such file or directory

[root@centos7-test init.d]# systemctl status rsyslog

● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-02-22 12:36:09 CST; 43s ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
 Main PID: 698 (rsyslogd)
   CGroup: /system.slice/rsyslog.service
           └─698 /usr/sbin/rsyslogd -n -iNONE

Feb 22 12:36:08 centos7-test systemd[1]: Starting System Logging Service...
Feb 22 12:36:09 centos7-test rsyslogd[698]: environment variable TZ is not set, auto correcting this to TZ=/etc/localtime  [v8.33.0 try http://www.rsyslog.com/e/2442 ]
Feb 22 12:36:09 centos7-test rsyslogd[698]:  [origin software="rsyslogd" swVersion="8.33.0" x-pid="698" x-info="http://www.rsyslog.com"] start
Feb 22 12:36:09 centos7-test systemd[1]: Started System Logging Service.

According to rpm, the faulty /etc/logrotate.d/syslog file is owned by the latest 8.33.0 package:

[root@centos7-test init.d]# rpm -qf /etc/logrotate.d/syslog

rsyslog-8.33.0-1.el7.centos.x86_64

See also rsyslog/rsyslog-pkg-ubuntu#74 and rsyslog/rsyslog#2143

[atc0005]

Workaround for anyone landing here affected by the change in the current 8.33.0 package:

1. sudo mkdir -p /etc/systemd/system/rsyslog.service.d
2. sudo nano /etc/systemd/system/rsyslog.service.d/10-enable-creation-of-pid-file.conf
3. sudo systemctl daemon-reload
4. sudo systemctl restart rsyslog

Here is the content to put into the /etc/systemd/system/rsyslog.service.d/10-enable-creation-of-pid-file.conf file:

# /etc/systemd/system/rsyslog.service.d/10-enable-creation-of-pid-file.conf
[Service]

# Upstream package choice:
#ExecStart=/usr/sbin/rsyslogd -n -iNONE

# Our override (options prior to 8.33.0 release):
ExecStart=
ExecStart=/usr/sbin/rsyslogd -n

[friedl]

A little late to this, but yesterday I added a change to the logrotate script, so the HUP will now be triggered by:

systemctl kill -s HUP rsyslog.service

as @mbiebl described here: rsyslog/rsyslog@6fafe7c. That should fix it, right?

[mielas]

systemctl kill -s HUP rsyslog.service

fails with current rhel7 selinux policies:

type=USER_AVC msg=audit(1527989823.315:169170): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc:  denied  { stop } for auid=0 uid=0 gid=0 path="/usr/lib/systemd/system/rsyslog.service" cmdline="systemctl kill -s HUP rsyslog.service" scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=service  exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'

so workaround given by atc0005 still the best (?) solution here for RHEL SELinux folks

[mbrownnycnyc]

[root@RHEL7 ~]# cat test.log | audit2why
type=USER_AVC msg=audit(1527989823.315:169170): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc:  denied  { stop } for auid=0 uid=0 gid=0 path="/usr/lib/systemd/system/rsyslog.service" cmdline="systemctl kill -s HUP rsyslog.service" scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=service  exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'

        Was caused by:
                Unknown - would be allowed by active policy
                Possible mismatch between this policy and the one under which the audit message was generated.

                Possible mismatch between current in-memory boolean settings vs. permanent ones.

And to allow:

[root@RHEL7 ~]# cat test.log | audit2allow


#============= logrotate_t ==============

#!!!! This avc is allowed in the current policy
allow logrotate_t systemd_unit_file_t:service stop;

[mielas]

regarding systemd_unit_file_t it looks like logrotate_t is almost in "god mode":

sesearch -AC -s logrotate_t -t systemd_unit_file_t -c service
Found 2 semantic av rules:
   allow logrotate_t systemd_unit_file_t : service { start stop status reload enable disable } ; 
   allow logrotate_t systemd_unit_file_type : service { start status reload } ; 

so it kinda strange stop isn't already there. but i will always choose workaround with pid file over selinux module because it affects only rsyslog. giving logrotate_t power to potentially shutdown any systemd service (while it lacks of that power currently -- and yes, it is already almost "god" now) is totally another level of workaround. personally -- i'll stick with pid file in such case

[davidelang]

if logrotate has the ability to send rsyslog the HUP signal, it also has the ability to send it TERM or KILL David Lang

[mielas]

well, in current selinux policies stop is right there and it wasn't there before...

all i was trying to say is if i had to choose between those two workarounds i wouldn't choose selinux module as it changes not only rsyslog stuff. it broadens capabilities beyond rsyslog case

nowadays rsyslog creates pid file and policies have "stop" ability. no more problems, life is good again...

[davidelang]

no disagreement about not wanting to rely on selinux and systemd.