Wednesday, February 27, 2013

Should I use rsyslog's new or old config style?

I got a very interesting question on the rsyslog support forums, and I thought I share it, together with the answer, here at a more prominent spot:

After over a decade of using stock bsd syslog, I finally have a need to do some more complicated processing of logs (splitting off Postgres query logs from general Postgres logs), and after looking at other options (basically syslog-ng), I think rsyslog looks like a better fit. I'm mainly in it so I can use regex matching, but thinks like the log queueing and being able to easily move to db storage in the future look good.
Since I'm new, I'd considered that I might get a jump on things by sticking with the newest config syntax. But after doing some googling for examples and looking at the examples in the rsyslog wiki, it seems like maybe the newest syntax might be a bit too new for a beginner - I learn best by example.
Are there any serious downsides to NOT going with the most current syntax?


The answer is that the old syntax is still fully supported by the versions and will probably remain for quite some while (except for some very few exceptions, which we couldn't carry over for good reasons - this is documented in the compatibility docs on the web site). Some parts of it are considered so important that they most probably never will go away. Actually, if you want to do simple things, the old syntax has its advantages. The more complex your processing gets, the more you benefit from the new syntax. But you can mix and match new and old style in almost all cases.

So my suggestion would be to get started using the old syntax and as soon as you begin to do more complex things, you can switch over to the new style. That's actually the way it is designed ;) A good indicator of when it would be benefitial to move to new style is when you begin to use a lot of directives beginning with $, especially if they modify an action. Also, if you move to action queues, I would strongly suggest to use new style. It is far more intuitive an less error-prone.

To provide a bit more background information, there is an important non-technical reason why the classical syntax is remain for a long time: basic syslog.conf format is extremely well known, covered in a lot of text books, taught in numerous courses and used in a myriad of Internet tutorials. So if we would abandon it, we would thrash a lot of people's knowledge and help resources. In short: we would make it much harder for folks that it would actually need to be. This has never been rsyslog philosophy. Providing the ability to changed gradually and with growing needs is a core goal.

Sunday, February 10, 2013

multi-character field delimiters

On the rsyslog mailing list, the ability to use multiple characters as field delimiters had been requested recently. Today, I took some time off the my schedule and implemented that functionality. It is probably very useful for a number of cases. An important one is probably in combination with control character escaping, where rsyslog by default expands a single character into a four-byte escape "#ooo" with o being the octal character code (so  e.g. US ASCII HT [horizontal tab] becomes "#011").

The new functionality is available for the RainerScript field() function. I do not intend to add it to template strings.

Some quick usage sample:

The following is the traditional way of single-byte delimiters, here with the comma character (US ASCII decimal code 44):
set $!usr!field2 = field($msg, 44, 2);
template (name="fld" type="string" string="'%$!usr!field2%' -- msg: %msg%\n")
action(type="omfile" file="/path/to/logfile" template="fld")

And this is the same with the string "#011" as delimiter:
set $!usr!field2 = field($msg, "#011", 2);
template (name="fld" type="string" string="'%$!usr!field2%' -- msg: %msg%\n")
action(type="omfile" file="/path/to/logfile" template="fld")

Note that the field number (index) need not necessarily to be fixed. It can be derived from an appropriately formatted message. Here the first field contains the actual field to extract, delimiter is "#011" again:
set $!usr!idx = field($msg, "#011", 1);
set $!usr!field = field($msg, "#011", $!usr!idx);
template (name="fld" type="string" string="'%$!usr!field%' -- msg: %msg%\n")
action(type="omfile" file="/path/to/logfile" template="fld")
In that last sample the $msg of

"3#011val 1#011val 2#011val 32#val 4"

would return

"val 2"

Keep in mind that the first field is the field index, so the actual data fields start at 2 (field 1 is "3", field 2 is "val 1", field 3 "val 2" and so on).

This functionality is already present in git master head and will be released as part of 7.3.7 in the not so distant future. Some more details can be found inside the RainerScript documentation page.

rsyslog 8.31 - an important release

Today, we release rsyslog 8.31. This is probably one of the biggest releases in the past couple of years. While it also offers great new fu...