Thursday, July 14, 2011

new rsyslog config: a thank you to our users

I wanted to thank all those users who have commented during the past three (!) years on config format questions. I have consolidated all input and hope I have come up with a decent solution. Obviously, not everyone will like everything, but I hope I could find a good compromise.

So far, my blog (and the rsyslog site) has the best glimpse at the new format:

It is compatible with the old legacy format, supports simple control-flow structures (no loops, by intension) and builds heavily on name value pairs for things like actions, inputs, global settings, ...

A real-world sample I used for parser development can be found at the rsyslog git web.

You'll also find the grammar files inside that source directory in the git tree. It may be interesting for those used to flex/bison. My next step is to develop the necessary code for the name/value pair objects. That requires some more plumbing inside the core and changes to *all* loadable modules. Sounds like a lot of work, but I still hope to get this done before the summer break.

I have also started thinking about v7. It will contain a tree-based execution engine, which potentially offers even higher speed and far more options for configuration. This change is too big to make it into v6. Note that v6 will support "if .. then" and probably "if .. then .. else" but not nesting of these statements -- because the rule engine does not support that. The main goal for v7 is to support nesting, including the (considerable) relevant engine changes.

I hope the new format is useful and does not upset too many. Sorry for the silence on the final selection. Past experience told me that there were too many totally conflicting views of what  the format should look like. I was deeply concerned that a broader detail discussion would have derailed this effort again. So I used known arguments and my best judgment to create the final format. Please all be assured that your arguments were deeply considered and extremely useful in getting this done.

For example, a recent mailing list discussion brought up very good argument why we actually needed to support old- and new-style config for include files. It turned out that actually the best way to solve that problem was to actually extend legacy format rather than completely replace it. This has the added advantage that textbooks, courses and a myriad of Internet-HowTos do not need to be rewritten. Besides that, I think that constructs like /var/log/maillog
are really hard to beat in simplicity and clearness, so I think it is valuable to have them as part of the config language.

Thanks again to everyone for helping make this happen.


Wednesday, July 13, 2011

rsyslog 6.3.3 config format improvements

In rsyslog 6.3.3, the config processor has finally changed. The old legacy processor (and with it the early RainerScript implementation) is thrown out and has been replaced by the so-called RainerScript processor (why that crazy name?). This is an extremely important step for rsyslog, as it now has the foundation for a much better and intuitive rsyslog.conf format. However, most of that can not be seen in 6.3.3, as it requires more work, especially in the plugin arena. Still, there are a couple of smaller improvements available.

Most importantly, the performance of script based filters has been considerably enhanced. Preliminary testing shows a three times speedup (we'll do more benchmarking at a later stage; there is also still lots of room for optimization ;-)).

The ugliness of continuation lines has been removed. They may still be used, and this may make a lot of sense with some actions, but you are usually no longer forced to use continuation lines. Take this config snippet from a leading distro:

if ( \
     /* kernel up to warning except of firewall  */ \
     ($syslogfacility-text == 'kern')      and      \
     ($syslogseverity <= 4 /* warning */ ) and not  \
     ($msg contains 'IN=' and $msg contains 'OUT=') \
 ) or ( \
     /* up to errors except of facility authpriv */ \
     ($syslogseverity <= 3 /* errors  */ ) and not  \
     ($syslogfacility-text == 'authpriv')           \
 ) \
then /dev/tty10
& |/dev/xconsole
This can now be written as follows:

if ( 
     /* kernel up to warning except of firewall  */
     ($syslogfacility-text == 'kern')      and     
     ($syslogseverity <= 4 /* warning */ ) and not 
     ($msg contains 'IN=' and $msg contains 'OUT=')
 ) or (
     /* up to errors except of facility authpriv */
     ($syslogseverity <= 3 /* errors  */ ) and not 
     ($syslogfacility-text == 'authpriv')          
then /dev/tty10
& |/dev/xconsole
Of course, this is not a real big advantage, but can be very useful during day-to-day operations. Forgetting the continuation marker is easy and has happend quite often, causing many more problems than necessary.

Also, the somewhat unintuitive use of "&" to chain actions together can now (optionally) be replaced by so-called blocks. For example,

authpriv.err /dev/tty10
&            |/dev/xconsole
can now be written as

authpriv.err { /dev/tty10
This looks much more familiar and thus intuitive to many users. Of course, both the old style as well as the new style is supported.

Finally, the need to use single quote characters (') over the usual double quotes (") in script based filters was often a source of confusion. You may now use both, so 'string' and "string" works both. However, strings in double quotes will support parameter replacement in later versions of rsyslog. That is "Message is $msg" will evaluate to exactly this string in 6.3.3, but $msg will be resolved to the actual message content some time in the future. So be careful if you use double quotes.

Of course, none of these changes are the important ones so many users are waiting for, most importantly an intuitively-usable scoping for actions and inputs. These will be coming up shortly. I need to write some more engine code *and* need to enhance plugins to support that. I'll probably start with actions as first. Note that the RainerScript processor already parses some of these constructs, but the rest of the engine simply ignores them. In order to get you an idea of how it will look, see this hypothetical example:

if $msg contains "error" then {
    action(type="omfwd" protocol="tcp" target=""
           queue.type="linkedList" queue.fileName="fwdRule" queue.maxDiskSpace="1g"
    action(type="omfile" target="/var/log/somelog.log")
    action(type="omuser" target="all" action.onceInterval="30")
I hope the example is intuitive enough to grasp it's meaning ;) In current format, you need to write

$ActionQueueFileName fwdRule1
$ActionQueueMaxDiskSpace 1g 
$ActionQueueSaveOnShutdown on 
$ActionQueueType LinkedList
$ActionResumeRetryCount -1
if $msg contains 'error' then @@
& /var/log/somelog.log
$ActionExecOnlyOnceEveryInterval 30
& :omusrmsg:*
At least to me, the upcoming new way looks much nicer ;)

In regard to the distro-example given above, I'll try to simplify it towards this form:

if ( /* kernel up to warning except of firewall  */
     hasPRI("kern.warn") and not 
     ($msg contains 'IN=' and $msg contains 'OUT=')
    ) or hasPRI("authpriv.err")
    then { /dev/tty10

But that's the second step after introducing the new action statements.

Please note that the final format selection was very carefully based on many discussions both on the mailing list and inside the forum as well as needs to preserve backwards compatibility. For example, on Debian, packages drop rsyslog-specific configs into the /etc/rsyslog.d directory and expect them to be understood. In order to break things here, we needed to remain compatible with the legacy format and extend it. Only thanks to the good user feedback we could finally come up with a solution that the majority of users hopefully will find good.

With that said, I'll now see that I create the actual release. For obvious reasons, 6.3.3 will be a bit shaky as far as the config is concerned. Most probably it will also not run the full testbench successfully (due to some very esoteric tests that are broken by actual functionality changes). However, you can be sure that the engine works well as long as it passed the config stage, because there were almost no changes during runtime (well... script filter expression evaluation has been rewritten from scratch).

Tuesday, July 12, 2011

rsyslog: important step to new config format

I have just merged the master-newconf git branch into rsyslog's master branch. With that, the new config parser becomes part of the main line. This is a very important step, as it lays the foundation to an enhanced, easier to use config format. The current version has only few enhancements, but provides the necessary plumbing to do some real nice work within the next couple of weeks. Not only as a side-effect, the performance of script-based filters has been notably increased.

I expect a release with the current state within this week. Mostly cleanup and doc work remains to be done.

Wednesday, July 06, 2011

Why omusrmsg is evil - and how it is fixed...

Traditional syslog files simply use the user name (or "*" for all) to send messages to users. For example, this selector will forward all mail error messages to the poor mail admin named "madmin":

mail.err madmin

This syntax is (somewhat) intuitive, but causes severe issues when it comes to extending the configuration language. Let's assume the mail admin is named Ian Faber and has the user name "if". So the syslog selector would be

mail.err if

This is ok with traditional config files, but creates a problem if the language is extended. For example, rsyslog has an "if expression then" construct. Question now: how to differentiate between the user name "if" and the "if" construct? Rsyslog uses context information in order to do this. At the start of a line "if" must be the "if" construct, because "if" as a user name would require a filter in front of it. This works pretty well, but creates some complexity during config file parsing. It may also be counter-intuitive to many users. If the language is further extended (as I am doing in v6), it creates considerable extra complexity.

To resolve that ambiguity, I have upgraded omusrmsg, which handles this kind of actions, to support the regular rsyslog syntax for action configuration. You now write:

mail.err :omusrmsg:if

The extra ":omusrmsg:" tells rsyslog explicitely that an action starts and so clearly flags what the "if" stand for. This is a very vital update, and so I am extending it into all versions starting with v4. I am right now working on these changes and will release all versions ASAP. I'll create another post when this is done. It is highly recommended to use the new syntax exclusively. The older syntax will go away in a while.

simplifying rsyslog JSON generation

With RESTful APIs, like for example ElasticSearch, you need to generate JSON strings. Rsyslog will soon do this in a very easy to use way. ...