Monday, October 27, 2008

New rsyslog HUP processing

There has been some discussions about rsyslog HUP processing. Traditionally, SIGHUP is used to signal the syslogd to a) close its files and b) reload its config. Rsyslog carried over this behavior from sysklogd.

However, rsyslog is much more capable than sysklogd. Among others, it is able to buffer messages that were received, but could not yet be processed. To remain compatible to the sysklogd of doing HUP, rsyslogd does a full daemon restart when it is HUPed. Among others, that means that messages from the queue are discarded, at least if the queue is configured with default settings. David Lang correctly stated that this may surprise some, if not most users. While I am still of the view that discarding the queue, under these circumstances, is the right thing to do, I agree it may be surprising (I added a hint to the man pages recently to reduce the level of surprise).

Still, there is no real need to do a full daemon restart in most cases. The typical HUP case is when logrotation wants to rotate files away and it needs to tell rsyslogd to close them. Actually, I asked if anybody knew any script that HUPs rsyslog to do a full config reload. The outcome was that nobody knew. However, some people liked to stick with the old semantics, and there may be reason to do so.

I have now implemented a lightweight HUP to address this issue. It is triggered via a new configuration directive, $HUPisRestart. If set to "on", rsyslogd will work as usual and do a (very, very expensive) full restart. This is the default to keep folks happy that want to keep things as backwards-compatible as possible. Still, I guess most folks will set it to "off", which is the new non-restart mode. In it, only output files are closed. Actually, the output plugin receive a HUP notification and can do whatever it likes. Currently, onle omfile acts on that and closes any open files. I can envision that other outputs, e.g. omfwd, can also be configured to do some light HUP action (for example close outbound connections).

The administrator needs to select either mode for the system. I think this is no issue at all and it safes me the trouble to define multiple signals just to do different types of HUP. My suggestion obviously is to use the new lightweight HUP for file closing, which means you need not to change anything for logrotate et al. Then, when you need to do a config reload, do a "real" restart by issuing a command like "/etc/init.d/rsyslogd restart". And if there really exists a script that requires a config-reload HUP, that should be changed accordingly.


flaco said...

I'm in the typical HUP case: when logrotation wants to rotate files away and it needs to tell rsyslogd to close them.
(btw I have few logrotate configurations. And I'm little confusing on how many hup I have to do...)

From which rsyslog version is configurable "$HUPisRestart off" ?

I appreciate rsyslog, it is the best syslog deamon could I find.

Rainer said...

Hi Falco,

thanks for your kind words. Please keep in mind that my blog is always at the (b)leading edge of development. So the new HUP processing is currently available as part of the git tree, only. There, it is in nextmaster.


Christopher Cashell said...

Just my opinion, but I hope a HUP signal will always default to processing config changes. ;-)

It's a pretty commonly accepted Unix practice that sending a HUP signal to most "server" processes (among them syslogd) will cause it to reread it's configuration file. There may not be many scripts that use this functionality, but there are a *lot* of administrators that do. This behavior is documented thoroughly in lots of places, and lots of Unix books.

In fact, a book sitting in front of me (Unix System Administration Handbook, Third Edition) explicitly directs you to always send a HUP signal to the syslogd process after making configuration changes.

I expect that many people will want to retain this default behavior to improve cross-platform compatibility and familiarity.

Rainer said...

Hi Christopher,

books and other docs is a good point for keeping the default at where it is. Still, it is a horrible default, if one looks at what this requires. I am not sure what packagers will do, if I had to package, I'd probably include the NON-restart HUP as the default for my package. But... not really of my concern ;)


Matt said...


what is the "best practice" for rotating logs with the current release?

Rainer said...

Hi Matt,

well, "best practice"... it depends. A log of folks use logrotate (which works well with the new HUP processing) while others prefer to use dynafiles, which include a date part (and thus no log rotation is necessary).


Al Whipp said...

In response to Flaco's query regarding multiple daemons of rsyslog running, it depends on whether they all get rotated at the same time. If not, of course, you can just rotate and HUP each independent.
I'm in the pickle of 3 processes all logging to the same spot, so all rotate together, so changed my HUP line to;
POSTROTATE="/usr/bin/killall -HUP /sbin/rsyslogd 2> /dev/null` 2> /dev/null"
Not as nice, but collects all rsyslogds nicely

simplifying rsyslog JSON generation

With RESTful APIs, like for example ElasticSearch, you need to generate JSON strings. Rsyslog will soon do this in a very easy to use way. ...