Monday, April 29, 2013

Log Anonymization with rsyslog

Starting with version 7.3.7, rsyslog natively support anonymizing log records. This is done with the help of a new module called mmanon. In short words, the module inspects the message and replaces IP-Addresses with an anonymized alternative representation.

This functionality is implemented via the (message modification module) action interface. Thus, the user has full control over when the anonymization happens. While it generally is desirable to anonymize as soon as possible, there can be situations where some data must be collected or processed un-anonymized. In that case, the anonymization can be deferred until after that processing.

The mmanon module basically has two operation modes: a faster but somewhat less secure and flexible mode ("simple"), in which specific octets are overwritten by user-configurable characters. For example, in this mode the IP address "" could be replaced to "". Note that the size of the specific octets is preserved.

In the other mode ("rewrite"), which is default, parts of the IP address are always zeroed out and the result is written in normalized form. The previous example would become "" in this mode. This also means that the message size may shrink. In rewrite mode, the exact number of bits that shall be anonymized can be specified. The default is 16, but any other value may be selected. If, in the above example, 12 bits would have been selected, the output would have been "". This provides great flexibility to meet corporate and legal requirements. Note that this form still permits to use some coarse-grained analysis tools, like for example Geo-IP lookups (of course, depending on the number of removed bits).

Currently, mmanon does support IPv4, only. However, support for IPv6 is planned, we are just waiting for some feedback before going further. The new module is available immediately and can both be found in the source tarball as well as Adiscon-provided rsyslog RPMs and rsyslog Ubuntu Packages.

Thursday, April 18, 2013

rsyslog TCP stream compression

I have begun to work on a way to "stream-compress" syslog messages over plain TCP syslog protocol, with the intent to support it over standard syslog as well if the idea works out.

Traditionally, rsyslog does message-level compression. That is each single message is compressed and if there is sufficient compression gain, the message is transmitted in compressed form. This works perfectly with UDP and TCP syslog, but the compression ratio is limited. The problem is that a single message does not offer much repetition to be shrinked. This mode still works surprisingly well.

However, we are now doing one step further: for TCP, we have a session, and so we are able to not only compress single messages but rather the full stream of them. That offers considerably larger compression potential. In its extreme end, it can be compared to gzip'ing a log file. Those of you who already did this note that we usually have very high compression ratios 5-to-1 or even 10-to-1 are not uncommon.

To gain these ratios, we need to run the compressor in a mode where it outputs data only when it decides it is ready to do so. This means that upon transaction completion, we may still have some data unsent (possibly even all data!). At the expense of compression ratio, this can be "solved" but forcing the compressor to flush at transaction end. This will degrade compression.

I have now done a first PoC to check the validity of the idea. It is implemented in omfwd and imptcp (NOT imtcp) only. Flushing at transaction end is currently not supported. We are right now practice testing this, and I hope to have some results when I am back from my trip to Tallinn.

Monday, April 15, 2013

Presenting on OpenKSI Meetup

I am happy to have the chance to present on rsyslog and it's recent signing capabilities at an OpenKSI meetup in Tallinn. Right now, rsysog's signing capabilities (introduced in the experiemental devel branch) base on RFC3161. But guess what may be next ;)

If you happen to be there, here is a link to the event:

There are still some free slots :-)

Monday, April 08, 2013

rsyslog output plugin wrangling...

For some hours, I am fighting with parts of rsyslog design around 2006 (or so): initially, we thought that all actions will be terminated by canceling their thread WHEN they not shutdown within the queue shutdown timeout. Then, we saw that it was better to at least try it cooperatively (cancellation is still required if that does not work). Now, with imrelp, I have a situation where I need to pass some information down to librelp when it comes to termination. Supposedly a very simple thing to do (a single call). ... unfortunately, the interface does not provide access to the action in question.

Hopefully, I finally found a work-around, via a "terminate immediately" pointer so far in use internally for the action engine. We'll see...