Don't write a pid file when in socket activation mode #2143

mbiebl · 2017-11-30T15:11:07Z

When rsyslog is used under systemd, it uses socket activation and the sd_notify API to signal service readiness. In that case, writing (and cleaning up) a pid file is not necessary. It's actually harmful if you want to lock down rsyslog.service with the systemd builtin mechanisms, as you need one more writable directory.

Please consider not writing a pid file when using socket activation mode or provide a command line switch which allows to turn off the pid file creation and can be used in rsyslog.service.

Thanks for considering

rgerhards · 2017-11-30T15:56:52Z

very good suggestion, thx!

Would it be appropriate to do not write (nor check) pid files if -n option is given. This means no backgrounding, and so the need for a pid file is more than questionable. That's also used in rsyslog.service file. On the other hand, it's a (slight) change in behaviour. But I would say that's fine.

Comments? @davidelang ?

mbiebl · 2017-11-30T16:41:30Z

Would it be appropriate to do not write (nor check) pid files if -n option is given.

Sounds ok to me. The altenative would be to check for getenv("LISTEN_PID") resp. if nFds > 0

rgerhards · 2017-11-30T16:44:58Z

@mbiebl What would you find to be more appropriate. I think that "fixing" -n might even be appropriate in cases without systemd, in the (unusual) case that you run two instances in non-background in parallel... It kind of feels more like the right thing to me, but I am not sure if going overboard...

mbiebl · 2017-11-30T16:46:26Z

I don't have a strong preference. Both would be ok to me.

rgerhards · 2017-11-30T16:47:37Z

thx -- I'll keep it open a bit for further comments. Will change it in 8.32 timeframe for sure.

jay7x · 2017-12-01T10:54:34Z

I'd say it's better to keep current behaviour for backward compatibility but provide specific cmdline switch to disable pidfile creation (--no-pidfile or -I e.g.). So systemd unit's options may be adjusted later.

Just remember there are old logrotate scripts which are not aware of systemd still (check /etc/logrotate.d/syslog in CentOS RPM for example).

mbiebl · 2017-12-01T10:56:58Z

Just remember there are old logrotate scripts which are not aware of systemd still (check /etc/logrotate.d/syslog in CentOS RPM for example).

I assume those old logrotate scripts check for the existence of a pid file and would be broken as well once the .service file start using --no-pidfile ?

rgerhards · 2017-12-01T11:13:48Z

I thought that is a good point. Just curios: how will logrotate work when no pidfile is given?

mbiebl · 2017-12-01T11:15:51Z

I would hope that the /etc/logrotate.d/syslog script is provided by the rsyslog package and updated along with the change to not write a pid file. Ideally it wouldn't check for the pid file at all

rgerhards · 2017-12-01T11:24:48Z

ok, let me re-phrase my question: how can the script learn rsyslog's pid from systemd? Sorry for the basic question, but you probably know out of your head and I need to search around ;-)

jay7x · 2017-12-01T11:25:36Z

It's easy to fix rsyslog rpm to call systemctl instead of kill cat pidfile. But there are lots of legacy scripts exists. I've spent some time migrating our production to use sytemctl kill -s HUP rsyslog but still not sure I've found all occurrences. Because every time you're rotating rsyslog-managed log file you must signal rsyslog to reopen those files.

mbiebl · 2017-12-01T11:25:53Z

ok, let me re-phrase my question: how can the script learn rsyslog's pid from systemd? Sorry for the basic question, but you probably know out of your head and I need to search around ;-)

Why would the script need to know that in the first place?

rgerhards · 2017-12-01T11:26:29Z

consider this answered ;-)

rgerhards · 2017-12-01T11:27:28Z

And I'd say @jay7x actually provided a good reason for a separate option, especially as we weren't strong in favor of one of those...

mbiebl · 2017-12-01T11:28:13Z

It's easy to fix rsyslog rpm to call systemctl instead of kill cat pidfile. But there are lots of legacy scripts exists.

Are there really many scripts which need to know the rsyslog pid? the logrotate one is the only one that comes to mind here and in case of Debian, the logrotate script is provided by the rsyslog package itself, so would obviously be updated as well (in Debian the logrotate script does not use the pid file anymore, so wouldn't be affected anyway)

And I'd say @jay7x actually provided a good reason for a separate option, especially as we weren't strong in favor of one of those...

well, I don't see how this would magically fix those scripts once the rsyslog.service file uses that new option?

jay7x · 2017-12-01T11:28:45Z

BTW do we still support CentOS 6 packaging? There is no systemd. I guess that's why we still have that scary kill `cat pidfile` >/dev/null 2>/dev/null

rgerhards · 2017-12-01T11:29:51Z

@jay7x To the best of my knowledge, it doesn't use -n on Centos, it backgrounds there. But you never know, right...

jay7x · 2017-12-01T11:29:52Z

well, I don't see how this would magically fix those scripts once the rsyslog.service file uses that new option?

@mbiebl Actually you're right.. but if we will change default behaviour then we may break every installation.

rgerhards · 2017-12-01T11:30:05Z

well, I don't see how this would magically fix those scripts once the rsyslog.service file uses that new option?

you win ;-)

mbiebl · 2017-12-01T11:31:00Z

well, I don't see how this would magically fix those scripts once the rsyslog.service file uses that new option?
@mbiebl Actually you're right.. but if we will change default behaviour then we may break every installation.

which package does provide /etc/logrotate.d/syslog in CentOS?

jay7x · 2017-12-01T11:32:19Z

Just to be clean. I'm ok with any decision. Just raised my concerns here :)

mbiebl · 2017-12-01T11:34:38Z

So, there is a problem indeed if there exist scripts which need to know the pid of the rsyslog process and use the pid file for that. What I'm trying to get a better overview is, if there might actually exist such scripts aside from the logrotate case. And assuming the logrotate script is shipped by the rsyslog package itself, I don't see a problem in this specific case.

rgerhards · 2017-12-01T11:39:44Z

IMHO I think this can actually exist, for example in custom log rotation scripts/packages (besides logrotate). I am sure there are some solutions in e.g. Java. I barely remember having heared about that once or twice during the past couple of years. I don't think, however, that this is a reason NOT to implement the proposed change. It's more a good visible warning in ChangeLog IMHO.

jay7x · 2017-12-01T11:41:48Z

which package does provide /etc/logrotate.d/syslog in CentOS

Just found it finally here:
https://github.com/rsyslog/rsyslog-pkg-rhel-centos/blob/master/rpmbuild/SOURCES/rsyslog.log.epel6

Installed by rsyslog package. v8 spec is here:
https://github.com/rsyslog/rsyslog-pkg-rhel-centos/blob/master/rpmbuild/SPECS/v8-stable.spec

jay7x · 2017-12-01T11:45:49Z

Well.. we may not create pid file when 1) there is '-n' cmdline option AND 2) no '-i' cmdline option is specified.

But we must fix supplied logrotate rule then. And man pages BTW.

davidelang · 2017-12-08T19:49:42Z

Sorry for the late reply I think there should ba a way to not write a pid file, but I don't like changing the behavior and not writing it automatically. There are too many cases where -n is used for testing, and without a PID file, log rotation scripts will get confused.

davidelang · 2017-12-08T19:52:25Z

On Fri, 1 Dec 2017, Michael Biebl wrote: well, I don't see how this would magically fix those scripts once the rsyslog.service file uses that new option?

it won't, but that's better than silently changing behavior just because the -n flag is used.

davidelang · 2017-12-08T19:53:25Z

On Fri, 1 Dec 2017, Michael Biebl wrote: So, there is a problem indeed if there exists scripts which need to know the pid of the rsyslog process and use the pid file for that. What I'm trying to getter overview is, if there might actually exist such scripts aside from the logrotate case. And assuming the logrotate script is shipped by the rsyslog package itself, I don't see a problem in this specific case.

logrotate is not the only way to manage log rotation, so just fixing logrotate doesn't mean that we won't break other people's scripts.

mbiebl · 2017-12-08T20:29:43Z

@davidelang so, assuming we add this new switch, and once it's available we use it in the systemd rsyslog.service file, what exactly would be different to having rsyslog not write a pid file automatically when being socket activated?

davidelang · 2017-12-08T20:35:22Z

On Fri, 8 Dec 2017, Michael Biebl wrote: @davidelang so, assuming we add this new switch, and once it's available we use it in the systemd rsyslog.service file, what exactly would be different to having rsyslog not write a pid file automatically when being socket activated?

Other distros that don't change the flags won't get different behavior. Not everything is systemd

mbiebl · 2017-12-08T20:37:44Z

Other distros that don't change the flags won't get different behavior.

Not everything is systemd

I don't understand your reply. The behaviour in the non-socket activated case would not change.

davidelang · 2017-12-08T20:42:21Z

I'm catching up over a bunch of posts, and the early part of this discussion was talking about not writing the PID file if the -n flag was provided. That is the main thing I was objecting to (it changes behavior on people) I understood you to be asking what the difference was between not writing a pid file if an explicit flag is given that's made part of the standard systemd config vs changing the behavior of an existing flag. If someone add a flag to explicitly disable the PID file, and it doesn't work on their system, that's their problem. I think it's probably a mistake to not write the PID file (it's very useful for log rotation scripts, and not everything is logrotate), but that's up to the distro to decide :-)

mbiebl · 2017-12-08T21:08:51Z

Thanks for the clarification, @davidelang
So if you got you right, your main objection is to change the behaviour of -n. I guess I can sympathize with that.
So I guess we have two options left then:
a/ do not write a pidfile if explicitly asked via a command line switch, like --no-pidfile.
The upstream rsyslog.service file would be updated to pass --no-pidfile by default I assume.

b/ do not write a pid file if rsyslog has been socket activated by systemd, i.e. if nFds > 0. This would require no changes to rsyslog.service

davidelang · 2017-12-08T21:11:51Z

On Fri, 8 Dec 2017, Michael Biebl wrote: Thanks for the clarification, @davidelang So if you got you right, your main objection is to change the behaviour of -n. I guess I can sympathize with that. So I guess we have two options left then: a/ do not write a pidfile if explicitly asked via a command line switch, like --no-pidfile. The upstream rsyslog.service file would be updated to pass --no-pidfile by default I assume.

This would be my choice.

b/ do not write a pid file if rsyslog has been socket activated by systemd, i.e. if nFds > 0

I hate heuristics like this, are there other ways that this condition could be triggered? David Lang

Command line option -iNONE provides this capability. This utilizes the pre-existing -i option, but uses the special name "NONE" to turn of the pid file check feature. Turning off is useful for systems where this no longer is needed (e.g. systemd based). closes rsyslog#2143

rgerhards · 2017-12-28T16:56:35Z

I have extended the -i option to support the special file name NONE (so -iNONE). The looked more natural and consistent instead of introducing yet another option.

atc0005 · 2018-02-22T18:30:55Z

Still digging into it, but it looks like installing 8.33.0 on a stock Ubuntu 16.04 system results in /etc/init.d/rsyslog rotate failing. Haven't tested CentOS 7 adiscon packages yet.

I reported this in rsyslog/rsyslog-pkg-ubuntu#74

atc0005 · 2018-02-22T18:31:53Z

Still digging into it, but it looks like installing 8.33.0 on a stock Ubuntu 16.04 system results in /etc/init.d/rsyslog rotate failing. Haven't tested CentOS 7 adiscon packages yet.

I reported this in rsyslog/rsyslog-pkg-ubuntu#74

Sorry, I didn't qualify my remark: I meant that after installing 8.33.0 from the PPA that I observed that behavior.

atc0005 · 2018-02-22T18:47:15Z

Confirmed that CentOS 7 has the same problem (not the same file mind you): the logrotate configuration expects to reference a PID file, but the rsyslog systemd unit file was configured to start rsyslog with the option to skip creating a PID file.

Created a separate issue for that: rsyslog/rsyslog-pkg-rhel-centos#42

deoren · 2018-02-22T18:52:45Z

One workaround is to override the startup options using a systemd unit file drop-in. That has worked well enough in the past for us.

atc0005 · 2018-02-22T19:31:08Z

@deoren

That doesn't work in this case.

root@ubuntu-1604-virtual-machine:~# systemctl status rsyslog

● rsyslog.service - System Logging Service
   Loaded: error (Reason: Invalid argument)
  Drop-In: /etc/systemd/system/rsyslog.service.d
           └─10-enable-creation-of-pid-file.conf
   Active: active (running) since Thu 2018-02-22 11:28:59 CST; 2h 0min ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
 Main PID: 845 (rsyslogd)
   CGroup: /system.slice/rsyslog.service
           └─845 /usr/sbin/rsyslogd -n -iNONE

Feb 22 11:28:56 ubuntu-1604-virtual-machine systemd[1]: Starting System Logging Service...
Feb 22 11:28:59 ubuntu-1604-virtual-machine rsyslogd[845]: environment variable TZ is not set, auto correcting this to TZ=/etc/localtime  [v8.33.0 try http://www.rsyslog.com/e/2442 ]
Feb 22 11:28:59 ubuntu-1604-virtual-machine rsyslogd[845]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.33.0]
Feb 22 11:28:59 ubuntu-1604-virtual-machine rsyslogd[845]: rsyslogd's groupid changed to 108
Feb 22 11:28:59 ubuntu-1604-virtual-machine rsyslogd[845]: rsyslogd's userid changed to 104
Feb 22 11:28:59 ubuntu-1604-virtual-machine rsyslogd[845]:  [origin software="rsyslogd" swVersion="8.33.0" x-pid="845" x-info="http://www.rsyslog.com"] start
Feb 22 11:28:59 ubuntu-1604-virtual-machine systemd[1]: Started System Logging Service.
Feb 22 13:29:12 ubuntu-1604-virtual-machine systemd[1]: rsyslog.service: Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services. Refusing.

atc0005 · 2018-02-22T19:40:50Z

Nevermind:

https://askubuntu.com/questions/659267/how-do-i-override-or-configure-systemd-services

This seems to work:

[Service]

# Upstream package choice:
#ExecStart=/usr/sbin/rsyslogd -n -iNONE

# Our override (options prior to 8.33.0 release):
ExecStart=
ExecStart=/usr/sbin/rsyslogd -n

systemctl daemon-reload
systemctl restart rsyslog

root@ubuntu-1604-virtual-machine:~# systemctl status rsyslog

● rsyslog.service - System Logging Service
   Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/rsyslog.service.d
           └─10-enable-creation-of-pid-file.conf
   Active: active (running) since Thu 2018-02-22 13:36:09 CST; 1min 19s ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
 Main PID: 9371 (rsyslogd)
   CGroup: /system.slice/rsyslog.service
           └─9371 /usr/sbin/rsyslogd -n

Feb 22 13:36:09 ubuntu-1604-virtual-machine systemd[1]: Starting System Logging Service...
Feb 22 13:36:09 ubuntu-1604-virtual-machine systemd[1]: Started System Logging Service.
Feb 22 13:36:09 ubuntu-1604-virtual-machine rsyslogd[9371]: environment variable TZ is not set, auto correcting this to TZ=/etc/localtime  [v8.33.0 try http://www.rsyslog.com/e/2442 ]
Feb 22 13:36:09 ubuntu-1604-virtual-machine rsyslogd[9371]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.33.0]
Feb 22 13:36:09 ubuntu-1604-virtual-machine rsyslogd[9371]: rsyslogd's groupid changed to 108
Feb 22 13:36:09 ubuntu-1604-virtual-machine rsyslogd[9371]: rsyslogd's userid changed to 104
Feb 22 13:36:09 ubuntu-1604-virtual-machine rsyslogd[9371]:  [origin software="rsyslogd" swVersion="8.33.0" x-pid="9371" x-info="http://www.rsyslog.com"] start

root@ubuntu-1604-virtual-machine:~# /etc/init.d/rsyslog rotate

 * Closing open files rsyslogd        [ OK ]

root@ubuntu-1604-virtual-machine:~# ls -l /var/run/rsyslogd.pid

-rw-r--r-- 1 root root 4 Feb 22 13:36 /var/run/rsyslogd.pid

mbiebl · 2018-02-22T19:43:33Z

Confirmed that CentOS 7 has the same problem (not the same file mind you): the logrotate configuration expects to reference a PID file, but the rsyslog systemd unit file was configured to start rsyslog with the option to skip creating a PID file.

At least for the Debian package, that is not true:

$ cat /etc/logrotate.d/rsyslog 
/var/log/syslog
{
	rotate 7
	daily
	missingok
	notifempty
	delaycompress
	compress
	postrotate
		/usr/lib/rsyslog/rsyslog-rotate
	endscript
}

$ cat /usr/lib/rsyslog/rsyslog-rotate 
#!/bin/sh

if [ -d /run/systemd/system ]; then
    systemctl kill -s HUP rsyslog.service
else
    invoke-rc.d rsyslog rotate > /dev/null
fi

mbiebl · 2018-02-22T19:44:18Z

See, no pid file needed when running under systemd

atc0005 · 2018-02-22T20:02:05Z

See, no pid file needed when running under systemd

A pid file may not strictly be needed, but the files provided by the related packages on Ubuntu 16.04 and CentOS 7 are setup to use a pid.

CentOS 7: /etc/logrotate.d/syslog
Ubuntu 16.04: /etc/logrotate.d/rsyslog

root@ubuntu-1604-virtual-machine:/etc/logrotate.d# cat rsyslog

/var/log/syslog
{
	rotate 7
	daily
	missingok
	notifempty
	delaycompress
	compress
	postrotate
		invoke-rc.d rsyslog rotate >/dev/null
	endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/rsyslog.log
/var/log/debug
/var/log/messages
{
	rotate 4
	weekly
	missingok
	notifempty
	compress
	delaycompress
	sharedscripts
	postrotate
		invoke-rc.d rsyslog rotate >/dev/null
	endscript
}

root@ubuntu-1604-virtual-machine:/etc/systemd/system/rsyslog.service.d# invoke-rc.d rsyslog rotate

 * Closing open files rsyslogd                   [fail]

From the /etc/logrotate.d/syslog file on a CentOS 7 system:

/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    missingok
    sharedscripts
    postrotate
	/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

No pid is created for the rsyslog 8.33.0 package for CentOS 7 (rpms.adiscon.com/v8-stable) nor Ubuntu 16.04 (ppa:adiscon/v8-stable).

@mbiebl My guess would be that at some point you updated the Debian packages to use the settings you specified in a prior post, but it doesn't appear that the same settings are used in the "upstream" packages.

mbiebl · 2018-02-22T20:03:09Z

Then fix the Ubuntu packages to not use a pid file in the logrotate config. Easy, right?

mbiebl · 2018-02-22T20:04:58Z

I mean, that's part of maintaing packages. If the adiscon provided packages do not handle this correctly, then they should be fixed indeed.

atc0005 · 2018-02-22T20:05:12Z

Then fix the Ubuntu packages to not use a pid file in the logrotate config. Easy, right?

That would be logical, yes, but the net effect is that the 8.33.0 packages (as-is) break existing configurations when deployed. The related conf files provided by those packages will need to be updated to no longer look for the pid file.

Until that happens, I will use a systemd drop-in to restore the old behavior.

lock · 2019-12-26T13:17:54Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

rgerhards added this to the v8.32 milestone Nov 30, 2017

rgerhards mentioned this issue Dec 28, 2017

rsyslogd: add capability to specify that no pid file shall be written #2309

Merged

rgerhards added the enhancement label Dec 28, 2017

rgerhards self-assigned this Dec 28, 2017

rgerhards added the worked-on label Dec 28, 2017

rgerhards closed this as completed in #2309 Dec 29, 2017

atc0005 mentioned this issue Feb 22, 2018

rsyslog 8.33.0: "/etc/init.d/rsyslog rotate" fails rsyslog/rsyslog-pkg-ubuntu#74

Closed

atc0005 mentioned this issue Feb 22, 2018

rsyslog 8.33.0: logrotate config still tries to reference PID file, but rsyslog is started with the option to not create a PID file rsyslog/rsyslog-pkg-rhel-centos#42

Open

waster mentioned this issue Jun 14, 2018

rsyslog rotate failed on Ubuntu 16.04 #2780

Closed

lock bot locked as resolved and limited conversation to collaborators Dec 26, 2019

Don't write a pid file when in socket activation mode #2143

Don't write a pid file when in socket activation mode #2143

Comments

mbiebl commented Nov 30, 2017

rgerhards commented Nov 30, 2017

mbiebl commented Nov 30, 2017

rgerhards commented Nov 30, 2017

mbiebl commented Nov 30, 2017

rgerhards commented Nov 30, 2017

jay7x commented Dec 1, 2017

mbiebl commented Dec 1, 2017

rgerhards commented Dec 1, 2017

mbiebl commented Dec 1, 2017 • edited Loading

rgerhards commented Dec 1, 2017

jay7x commented Dec 1, 2017

mbiebl commented Dec 1, 2017

rgerhards commented Dec 1, 2017

rgerhards commented Dec 1, 2017

mbiebl commented Dec 1, 2017 • edited Loading

jay7x commented Dec 1, 2017

rgerhards commented Dec 1, 2017

jay7x commented Dec 1, 2017

rgerhards commented Dec 1, 2017

mbiebl commented Dec 1, 2017 • edited Loading

jay7x commented Dec 1, 2017

mbiebl commented Dec 1, 2017 • edited Loading

rgerhards commented Dec 1, 2017

jay7x commented Dec 1, 2017

jay7x commented Dec 1, 2017 • edited Loading

davidelang commented Dec 8, 2017 via email

davidelang commented Dec 8, 2017 via email

davidelang commented Dec 8, 2017 via email

mbiebl commented Dec 8, 2017

davidelang commented Dec 8, 2017 via email

mbiebl commented Dec 8, 2017

davidelang commented Dec 8, 2017 via email

mbiebl commented Dec 8, 2017 • edited Loading

davidelang commented Dec 8, 2017 via email

rgerhards commented Dec 28, 2017

atc0005 commented Feb 22, 2018

atc0005 commented Feb 22, 2018

atc0005 commented Feb 22, 2018

deoren commented Feb 22, 2018

atc0005 commented Feb 22, 2018

atc0005 commented Feb 22, 2018

mbiebl commented Feb 22, 2018 • edited Loading

mbiebl commented Feb 22, 2018

atc0005 commented Feb 22, 2018

mbiebl commented Feb 22, 2018

mbiebl commented Feb 22, 2018

atc0005 commented Feb 22, 2018

lock bot commented Dec 26, 2019

mbiebl commented Dec 1, 2017 •

edited

Loading

mbiebl commented Dec 1, 2017 •

edited

Loading

mbiebl commented Dec 1, 2017 •

edited

Loading

mbiebl commented Dec 1, 2017 •

edited

Loading

jay7x commented Dec 1, 2017 •

edited

Loading

mbiebl commented Dec 8, 2017 •

edited

Loading

mbiebl commented Feb 22, 2018 •

edited

Loading