Thursday, August 30, 2007

rsyslog v3 object model and message flow

I thought quite a while about the upcoming configuration file format today. I
have to admit, I am not very satisfied with the outcome. I have a few potential
formats, but either they are too brief (and quite complex) or there is a lot of
text. Also, I am not satisfied yet with the flexibility of the format. I guess
this is a harder nut to crack than I thought. Unfortunately, I can not yet post
any of the ideas, as I have done them on paper and they are probably unreadable
to anybody but me. Bear with me, I'll create an online version these days.


But to understand any config file formats, you need to know more about the
object model. A first version of that model has materialized on my mind, and
actually rsyslog 1.18.0 and above is already supporting parts of it. To aid
understanding (and get some discussion going), I have created a quick outline of
what I have in mind. It's textual, it's incomplete and it can not be formally
verified. Still, I think (sincerely hope), it is useful. So I decided to post
what I currently have. Besides the actual object model discussion, it will help
a great deal in understanding what needs to be in the config file (thus it's
currently a bit biased towards configurable objects and internal ones are almost
completely missing).


rsyslog v3 Objects


Rsyslog uses objects, even though it is written in C. This can quite well be
done. Only at some points (like inheritance) we need to fiddle a bit with the
language. But nothing to hard to be done. We try to keep object overhead very
low, so it should be more like traditional C than C++.


module


The module object is the base class for anything that even remotely looks
like a loadable object. In the long term, that might eventually be almost
everything. It handles the necessary plumbing like loading modules, keeping
track of module status, querying interface and all those things... It does not
by itself provide any module-specific actions.


input


An input gatheres messages (events) from external sources. Current typical
examples are UDP or TCP based syslog. However, there is no architectural limit,
so in the long term an input module may also gather SNMP traps or file lines.
Please note that an input does not necessarily parse the obtained event by
itself - this may be delegated to a parser module (this whole thing still needs
to be thought out).


output


An output module receives strings from the engine and writes them to some
ultimate destination. Popluar examples are files, databases or remote syslog
servers.


action


The action object is the "engine wrapper" around an output module. It
provides numerous servies to the output. Most importantly, it provides the core
plumbing behind restarting and queueing actions.


function


A function object (and lodable module) provides extensibility for internal
processing. Version 3.0 will support programming-like functions, which replace
and extend the property replacer options. For example, we may want to extract
characters 5 to 10 and convert them to lower case. With current rsyslog, this
works as follows: %msg:5:10:lower%. With Version 3, it will look much nicer (at
least I hope so): lower(substr(%msg, 5, 10)). The exact sytax and semantics of
how this is used in the config file is under development - but I think you can
get the idea from the example. The core improvement featuere-wise is that with
functions, rsyslog can very esaily be extended by just programming a small
plug-in with a new function. Do to the programming-like strucuture, the new
function can be combined with all exisitng ones.


Functions will be supported everywhere a string is supported, which means
everywhere. Examples are filters, output format (templates) and file name
generation.


Message Flow


This is a brief description of how a message (an event) flows inside rsyslog.


The message text is received by the input module. The input module, possibly
with the help of a parser module uses the message text to create a msg-in memory
structure. This msg structure enters rsyslog's main processing queue. As soon as
possible, the msg object is dequeued. Now the rule engine processes the rule set.
A rule set object contains multiple rule object, which are executed in order
until either all rules are processed or a discard action is encountered. If
there are multiple rule set objects bound to a single input module, they are
also executed in the configured order (the normal discard action will discard a
message for one rule set, but an additional rule set will  continue to
process the msg object. A special case of the discard action can be envisioned,
which discards a msg object for all bound rule sets.. Each rule object contains
filter objects and action objects. First, all filter objects are executed. That
means, their expressions are evaluated. If there are multiple filter objects,
their result is combined in a AND operation (other modes are not supported -
that could be easily done by crafting a specific filter expression, so we do not
encourage an additional set of complexity be allowing additional boolean
operators when multiple filter objects are used inside a single rule). If the
outcome of the overall filter object evaluation results in true ("to be
processed"), all action objects are called. The are called in the order of their
appearance in the config file (no exceptions here). The msg object is used to
generate that string required by the action (note that the msg object itself is
NOT passed to the action, this is a security and encapsulation boundary). For
string generation, function plug-ins may posibly be called (this is also the
case during filter procesing). The resulting strings are  passed to the
execute method of the action object. That object decides if queing,
rate-limiting or other functionality generically available to all actions is the
be carried out. All of this is done by the action object itself. Finally, the
strings are passed from the action object to the actual output object (a plug-in).
The output objct takes the strings and performs whatever processing needs to be
done. After all actions, rules and rule sets have been processed (or when a
discard action occurs), the msg object is destroyed. Please note that even than
a copy may be held in memory, because that might be needed for duplicate message
reduction and similiar features. Thus, msg is reference-counted and actually
only destroyed when the count reaches zero.


Please note that there are a number of utility objects involved. For clarify,
these have been omitted. Also, all function calls return a meaningful return
state. The caller will process that state with care. In some cases, however,
this may mean ignore a failed call, because this is the most appropriate thing
to do (e.g. when an action fails - if we'd abort processing the msg object, we'd
do much more harm).


Data flow


This is a brief and not so accurate picture of the data flow. It concentrates
on the role of plug-ins.


input module(parser) -> rule engine (custom functions) -> action (with
aggregated output module)


Known Open Issues with Plug-ins


The object model is not yet fully designed, so there is nothing bad about
open issues. I list them, so that the do not get lost.


Probably the number one question is how message-modifying plug-ins can be
created. I think about things like implementing syslog-sign or TLS security. It
is my current, unverified, view that we must have something like "filter-plug-ins"
which can be placed inside the message data flow to modify the message text and/or
the msg object.


Copyright © 2007 by
Rainer Gerhards

Last Updated: 2007-08-30

Tuesday, August 28, 2007

on the importance of plug-ins for rsyslog

Rsyslog currently offers only very rough support for plug-ins: only the MySQL plugin is available as a plugin. Even the output plug-in structure is not yet finished. So it looks like plug-is are quite unimportant to rsyslog. Right?

Wrong! Totally wrong! If there is a single feature that is absolutely critical to rsyslog, it is plug-in support. Plug-ins offer a way to provide dynamically loadable extensions of the core functionality. Provided the plug-in interface (API) is clean and well-designed, that can come without any performance hit.

Now don't just think about plug-ins for exotic features. What if almost everything is a plug-in? Well, in that case the code loaded in memory (the footprint) can be exacty tailored to what is needed. This is most benefitial for small, embedded system with limited resources available. On the other hand, this doesn't limit a highly capable system of offering a very rich functionality. It just loads different plug-ins.

Remember that almost all functionality can be implemented as a plugin. Inputs (E.g. UDP syslog or local domain sockets) and outputs (e.g. database output or message forwarding) make somewhat natural plugins. But plugins can also be used in filtering. For example, the (expensive) regexp functionality could be provided by a plugin. As could be different message parsers.

This is exactly the way rsyslog is heading: we will try to provide an ultry-slim framework which offers just the basic things needed to orchestrate the plug-ins. Most of the functionality will indeed be available via plug-ins, dynamically loaded as needed.

I am not yet sure when we will reach his goal. But the object model design and the overall design direction heads into this area. With that design philosophy, we can make rsyslog really universally available, even on low-powered devices (loading just a few plug-ins). At the high end, systems with a lot of plug-ins loaded will be able to handle the most demanding tasks.

And, of course, plug-ins will hopefully not only be provided by the rsyslog project itself. It is our sincere hope that others will also begin to contribute functionality in the form of plug-ins. To facilitate that, we strive to create very simple plug-in interfaces. For example, all threading details will be handled by the core engine itself and plug-ins will be able to think they are running on a single thread. The core engine will handle the proper sync operations. Of course, plug-ins in need to use themselves a sophisticated threading model can do so - but it is not a requirement. Hopefully, that'll make it very easy to create plug-ins and thus be very easy to contribute extended functionality for rsyslog. That, we hope, will help create a lively community and thus a breakthrough syslog project.

Friday, August 17, 2007

on the rsyslog config file format...

A strength of rsyslog is that it support stock syslogd's config file format. One of its weaknesses, however, is that it support stock syslogd'd config file format.

Sound strange? Jepp - but it isn't. Understanding the old config format is great when it comes to replacing stock syslogd. It's easy for package maintainers and it is also easy for users (who do not necessarily need to learn anything new). However, that format is clumsy, especially when compared to syslog-ng. So there is already a lot of criticism for supporting that format.

To make matters worse, rsyslog's enhanced functionality requires some enhanced configuration directives. They have evolved as rsyslog has evolved. Currently, they are only somewhat ugly. But the more the project evolves, the more ugly they will become.

In short words: I do not like the current config file format.
But on the other hand, I also do not like spending much time "just" on creating another config file parser. And I do not like the learning curve associated with it.

Seth Vidal, some weeks ago, recommended to have a directive in the config file specifying whether it is old-style or new-style. I'll probably follow this route. That would enable us to keep backwards compatible and allow users to re-use their knowledge. In "old format mode", features would be limited by the file format (and the users' ability to deal with it). The full glory would only be available in "new format mode". That sounds just fair, especially as users need to learn the new functionality and then it does not really matter if the need to learn a new conf format while they go along.

OK, that seems to be (somewhat solved).

But what format should be the new style format? Seth Vidal recommended a natural language like format (see his post on the rsyslog mailing list). However, there are many things unanswered. We don't just have filters, we have many things.

Let's go to the root question: what is a config file good for? In my personal view, it should

  • enable the user to specify what the product should do in an easy and simple to follow way
  • allow the application to create internal control structures that specify how the application works

Both are equally important. But the second one tells us the at least part of the config file structure is dictated by the applications object design (or call it [mnodular] structure).

So I think to do the config file right, one first needs to know the application's object model. For rsyslog, this is not yet published. Even further: it is not yet fully designed. The 1.x versions, as well as part of the 2.x version does not have a real object design (but 2.x has at least a partial one and is somewhat like an interim release on a way to the full model).

As a side-note, this is also the reason stock syslogd and early rsyslog works with such a simple config file: it is a monolithic application and such a one is served perfectly by an equally-monolithic config file. I intend to publish a full object model for rsyslog 3.0 in the fall/winter 2007 time frame. Then, we can look at the specifics of the config file.

Now back to the number one requirement for a config file: user friendliness. We know we need something that is capable to work with a complex object model. But how to phrase this?

I currently see four choices:

  1. a pure natural-language look-alike like suggested by Seth Vidal (actually more a programming like type
  2. a purely XML-based file
  3. something like the apache config
  4. something close to the way syslog-ng does things

I have to admit I am sceptic for a pure #1 approach. Natural language is intuitive for filter conditions, but it looks probably somewhat funny with most other object (which have a large number of properties). With pure XML, it is quite the opposite. It is very well with most objects, but a filter tree in XML is just plain ugly. So the most tempting choices for me are either something apache- or syslog-ng-like.

I have some mental problem with picking up the syslog-ng format, it just sound right to me. But this can be overcome. The question is if we get us into trouble when we extend the syslog-ng format and syslog-ng does a similar thing. That can become quite ugly.

An apache like format also has its drawbacks. I am not sure yet if inheritance will work well in a syslogd (but, hey, isn't that what we currently already have?). But it sounds like a good choice. Is that something that we should model to see if any weaknesses or trouble spots occur?

Well... I can't answer. What do you, our users, think? I would really appreciate feedback on possible config file format. I have set up a wiki page for suggestions. I would really appreciate if users could post what they would find a useful config file format.

When the object model is ready, we as the community at whole can then decide what the best possible config file format would be. If we are lucky, we'll have a clear, easy-to-read config file format that matches the object model by winter 2007.

Thursday, August 16, 2007

on rsyslog versions

What I said about rsyslog versions these days may sound a bit strange. I am talking about 3.x, but yet we are only at 1.x. Do I intend to bump version numbers just for the fun of it. No! But the schema I originally had in my mind has proven to be infective. Initially, minor versions starting from .10 were reserved for devel builds, while anything below was reserved for stable ones. Consequently, we have a stable 1.0.5 right now and a devel of 1.19.0. However, the 1.19.0 is a very far departure from 1.0.5.

So I now have decided that the 1.x devel series will become 2.x stable. In fall/winter, I have a number of very large changes for rsyslog on my mind. These changes will be so considerate that the end result can not be a 2.0 version. This time I want to make this clear from the beginning. As such, the new devel series will actually start at 3.x.

So hopefully this all will make sense now: I'll continue a bit to use the 1.19 (or higher) series for things that will become the next stable release). Around September/October, it will actually be released as 2.0.0 stable (and then receive mostly fixes). As soon as this is done, development will immediately continue with 3.x, which will be the release with new object model, enhanced threading, more plug-ins and all that...

I hope that clarifies now - and makes at least some sense to you.

Tuesday, August 14, 2007

rsyslog changes for 2007-08-14

Do you remember? I promised to make no more major changes to rsyslog until around September. Among the things I did not intend to do was a plug-in interface. And, yes, I did not do that. But varmojfekoj, a frequent contributor, did it ;)

As varmojfekoj says himself, this is not a full-blown plugin implementation. But he has converted the MySQL output module into a loadable plugin. The current output module interface API has some restrictions, so it is not 100% clean - but by far clean enough to be used in practice.

The big advantage of having this plug-in is that package maintainers now can create two packages for rsyslog: one for the base system and another one for the mysql functionality. This removes a lot of problems. For example, when rsyslog was compiled with MySQL support, it would have required the MySQL client to be present. This would not be acceptable for a default syslogd (after all, each system would need to have MySQL client installed...) - clearly unacceptable. Now the base (default) system can be supplied without MySQL (which is no longer tied in). And those who desired database logging, can simply add the MySQL plug-in add-on package. That package will of course pull the MySQL client, but that's fine if you would actually like to use it (sounds only fair, doesn't it?).

OK, finally, here is my (small) work log:

- integrated patch from varmojfekoj to make the mysql module a loadable
one
many thanks for the patch, MUCH appreciated
- added some code to support the forward-compatibilty directive
$ModLoad MySQL - this is now internally translated to the right name
Needs to be revisited, but is clean enough for now.

rsyslog changes for 2007-08-13

As promised, low activity:

- removed debian subdirectory by request of a debian packager (this is a special
subdir for debian and there is also no point in maintaining it when there
is a debian package available - so I gladly did this)
- improved overall doc quality (some pages were quite old) and linked to
more of the online resources.
- released 1.18.2

Monday, August 13, 2007

on reliability and the need to discard messages...

I am a bit involved in creating the next-generation on the wire syslog protocol with the IETF. Unfortunately, that activity had to starve a little while I was busy with getting rsyslog to its current state. However, I have not lost it out of my eye. And in fact the rsyslog implementation effort helped my sharpen my mind for what to suggest in the standardization area.

A recap: the IETF tries to standardize the syslog protocol for quite a while now. There were changing thoughts about what was needed, but this spring we agreed on bringing a draft up to the IESG. From there, we received a number of comments. Some easy ones, some of more substance. In July 2007, more or less one topic was left: that on what to do when a syslog sender blocks. I quote the relevant part of the IESG reply here (accentuation by me):

> First, this starts as an issue with TLS over TCP and the
> syslog base protocol.
> It can also arise teorethically for UDP, but as I understand
> not in practice for todays usage. When you are using TCP, in
> case the syslog sender generates events at an rate that is
> higher than available rate over the path used there will be
> build up of a queue. So I would like to have some words
> somewhere saying what you do when you build up a queue of
> messages that should be transmitted, but the queue simply
> keeps growing. What do I do? To me this situation implies
> that one starts discarding syslog messages and starts
> with the least important ones.
So I would like to have
> a paragraph on this.
>
> I also included UDP in this in the case that you actually
> have reserved or determined that syslog is allowed to use a
> particular amount of bandwidth, but not more. In this case it
> could be possible that one implements a rate limiter
> and run into exactly the same issue as for TCP
.
>
> Please do understand that if syslog was designed from scratch
> today it wouldn't get away without a congestion control that
> guarantees that it isn't missbehaving in any situation. But
> being an "old" protocol we are accepting less than that.
> However, we do require it to contain the limitations and
> assumptions that it can be safely operated with. Using it as
> it currently is used is not an issue because the networks it
> is used in has many magnitudes more bandwidth that syslog
> generates. However, what would happen if some one starts
> using syslog in low-powered, low-bitrate sensor network for
> carrying some information. In that situation syslog becomes a
> potential threat to the stability of the network as it
> doesn't react at all to congestion if run over UDP. Network
> failures are also sitation that are problematic to ensure
> that the inteded resources are available.
Thus we do
> like to protect the utility of what resources do exist.

And this sentence covers it all:

> Please seriously consider adding a paragraph about
> how one can thin a queue of syslog messages in
> the base protocol. This as I think it potentially applies
> to any transport.


This basic requirement caused some confusion in the working group (WG). I have to admit, it confused me at that time, too.

In the mean time, I kept working on rsyslogd. I didn't even realize that I had written code in 2005 (or so) to take care of what was requested by the IESG: if rsyslog runs in single thread mode and it forwards messages via TCP and the send call would block - then we have a problem. Because when we wait, we will potentially hang the whole system. This is unacceptable. So I coded rsyslog in such a way that it discards the message in such a situation.

When I moved rsyslog to a multi-threaded design, that restriction was removed. Now, there was a queue between the receiver and the (potentially blocking) sender. And, as we all know, queues are endless, so we'll never run into problems ;-]. Of course, that simplistic point of view survived only so long as we did not deliberately bash rsyslogd with lots of traffic. What then happens is pretty much the same as in the single-threaded case: rsyslog freezes. Mildew, a frequent contributor, discovered this problem and also sent a good fix: he throws away messages if the queue is full. Sounds familiar? Yes, there seems to be a real-world problem and there is also a well known cure for it.

In some essence, this is what the IESG is asking for - at least as of my understanding. But the IESG, obviously speaking with a lot of experience, goes a step further. What they recommend is not to discard newly arrived packages. The recommend a way to "thin out the queue". So we would not just blindly drop whatever comes when the queue is full. No, we should intelligently decide what to drop, and that based on the severity of the message. It sounds only fair that debugging messages get dropped in favor of emergency messages. And, yes, syslog has a field "severity", which is the right thing to use here.

I thought about implementing a "high watermark" in the rsyslog queuing engine. What that amount of the queue would be full, a filter should be applied to the message. So from that on, only messages with a given severity would be queued, while all others be dropped. Of course, those already in the queue would be sent, no matter how unimportant they are. So far, I have not implemented that algorithm. And I think I am now unlikely to do it that way. It would be much better to follow the advise and, once the queue becomes full, begin to drop messages based on their severity.

I can even envision the algorithm: go forward through the queue starting at the tail and look for messages with the least importance. Once you have found one, drop it. Then enqueue the (new) higher-severity message. When this is done, do not drop any more messages, because it may not be necessary. We can always drop if a newer higher priority message arrives. If, during our queue traversal, we could not find any message with lowest severity, retry with the next lowest severity. Run this algorithm until either a message to drop is found or the drop-severity has reached the severity of the to-be-queued message, which in this case is the youngest and thus the drop target. Of course, that algorithm needs to be tweaked in regard to performance, but basically it should work fine. I even think, a derivative of it will make it into a rsyslog feature request.

... and now back to the IETF. We still need wording for our upcoming standard (called an I-D). Now that I know what I look for, shouldn't it be easy? Well, to be honest I don't think its that easy. First of all, I am not sure if, in a protocol specification, I can demand such queue processing. It looks somewhat too algorithmic, too implementation specific to me. So I need to find a better way to phrase it. And of course, peer review will show if that makes sense or not (maybe we even get a better suggestion).

My first try at it is as follows:

In any implementation, a situation can arise in which an originator or relay would need to block sending messages. A common case is when an internal queue is full. This might happen due to rate-limiting or slow performance of the syslog application. In such cases, it is RECOMMENDED that the originator or relay drops messages of lower severity in favor of higher severity messages. Messages with a numerically lower SEVERITY value have a higher severity than those with a numerically higher one. To-be-dropped messages SHOULD simply be discarded. The syslog application may notify a collector or relay about the fact that it drops messages. If the underlying protocol supports congestion control, discarding of messages SHOULD be avoided.


I have to admit that this text is probably too lengthy and imprecise. But hopefully it is a starting point. I'll post it to the syslog IETF WG and we'll see...

Still, there is one important message to learn:
no matter how hard we try, there is always a chance that we must discard messages. It may be highly unlikely and it may be totally unexpected. But we can not absolutely avoid it. Not even an on-disk queue is large enough, that it can buffer any imaginable message queue. So loss of messages is inherent in any application, also in syslog and auditing. The question is, how we deal smartly with it and how we preserve as much evidence in our logs as possible (and keep the log valid in those cases!). This all sounds pretty basic and easy, we always need to remember ourself's of this fact...

Sunday, August 12, 2007

why does the world need another syslogd? (aka rsyslog vs. syslog-ng)

This was a question I received back from Anton Chuvakin when I asked a number of folks which features they wanted to see in rsyslog. If I could answer that question, he would reply with further details.

Actually, I think that was a very smart reply. It comes down to "why should anybody care about rsyslog at all?". Aren't there already enough syslogds? An interesting question.

At first, I was tempted to answer that question by talking about features, all those things that rsyslog will do to rule the world. But - does this answer the question? Nope, it doesn't. Next, I thought to write about my goals with rsyslog. Why do I do all this work, just to create another free GPLed application. But, again, does this answer why the world needs another syslogd? Oops... no, it doesn't. Its not about me, but about the world, that's you, folks. So why do you need another syslogd...

Let's look. The world is somewhat divided. In this little corner sits Windows, and in the other there is the *nix community (Linux included). There are also some other corners, but I think I can simply ignore them for the time being. Well, I think I can even ignore the Windows world, as rsyslog is not meant to play there. I could answer why the world needed WinSyslog, which I designed over 10 years ago (to be the first syslogd ever on that platform). But besides showing dad is proud of his kid, that will lead us to nowhere.

So let me re-phrase the question a little bit: "why does the *nix world need another syslogd?"

That world uses either

I have bluntly combined all other syslogds (including rsyslog) to "another solution" simply because not a single one of them has enough market share/popularity to be a major player. Face it: the *nix world currently explores two choices when it comes to syslogd. Two big choices seems to be healthy. The bad thing, however, is that stock syslogd has not been notably enhanced in the past years. Syslog-ng was the answer to this problem around '98 (if I remember correcty), and the situation has not much improved since then. In fact, users complain more and more about the featureless stock solution and Linux distributors have begun to look for alternatives.

So in technically sound, feature-rich and widely-deployed solutions we are down to one: syslog-ng. A monoculture always scares me a bit, even if it is with open source software. So one why the world needs another syslogd is to have more choices.

OK, the smart reader will now ask, "but why haven't you joined forces with some of the other projects? Why a new one?". Again, smart questions. Now I need to talk a bit about Rainer, because it has a lot to do with Rainer (and not you, the world ;)). I looked around at the various syslogd projects. To be honest, I tried the easy to find ones. Many of these projects had fine code. But in one way or the other, they didn't look like the right platform for me to start with. From day one, I thought about creating something that in the long term can become a default syslogd (yes, folks, I *am* arrogant at times and my self-confidence is, ummm, not under-developed ;-]). So I didn't like to start from something that receives low attention and has a small installed base. And so did I start from scratch? NOT AT ALL! I may be a bit too self-confident, but I am not a fool ;) I started with sysklogd, the stock syslogd on most Linuxes! That was a natural choice. What could better be targeted to becoming the default syslogd ... than itself.

I tried to extend the sysklogd project. However, I quickly realized that this project's focus was much more on stability than on new features. The maintainers were not very eager to accept substantial changes (and I can not complain about this, given the magnitude of what I intended to do). So I did the natural thing: I forked rsyslog from sysklogd. And, oops, now the world had another syslogd.

But now back from Rainer to you, the world. Why should you need another one.

By now, you probably know that rsyslog aims at becoming one of the major players. And yes, it will even challenge syslog-ng (and I am proud to say that this has already begun, at least in respect to some features exclusively seen in rsyslog).

So one reason the world needs another syslogd is that it needs another major player
in the *nix space. I honestly believe there is none except syslog-ng.

OK, so then we are back to "why is syslog-ng not enough"? Do you think my "I need choices" argument is lame? Ok, ok, then I have some real meat for you: have you noticed that syslog-ng has become dual-licensed? There is the great GPLed open source release and the even greater "Premium Edition", which cost money. Only the premium edition offers features like native database and SSL support or queued syslog sending.

I do not bash Balabit (maker of syslog-ng) for this movement. A project needs funding and it seems only natural that some of funding should be provided by those that gain benefits from it. I have to admit that the long-term funding of rsyslog of course is under review, but I sincerely hope that we can get away with staying GPLed open source all the time (and I can guarantee this will be the case for the foreseeable future including the glory 3.0 release which will really try world domination ... feature-wise, of course).

Do you think Balabit will include the premium features into the GPLed version? I'd say that's highly unlikely - after all, why did they create two editions in the first place. So the GPLed version will become a second-class citizen and the really cool things will go into the premium edition.

"Stop", I hear some say, "syslog-ng is GPLed, so we can take that source and implement the missing features". Of course you can. But do you think Balabit will actually include your patches? I guess we can agree on "nope". So you have just forked from syslog-ng ... and proved my argument that the world needs another syslogd (this time your syslog-ng fork ;-]).

As I side-note, I'd consider such a syslog-ng fork unfair and at least somewhat unethical. Balabit has served the community very well and continues to do so. They are not the kingdom of evil. As such, we should respect their voiced intention and not circumvent what they have done to provide proper funding for their project. Of course, you may have a different opinion, but such a fork would definitely be against my code of ethics. I won't do such a thing and hope others won't do it, too (except, of course, for a *very* good reason, e.g. no further development by Balabit or Balabit turning into something like SCO - which I do not expect).

As you can see, GPL syslog-ng will possibly not be at the (b)leading edge of syslog technology anymore. I may be wrong here, but the current state of the project points into that direction.

Competition with another major player may even help to get more of the cool features into the GPLed fork of syslog-ng. This is a key thing: if there are competing projects, each of them benefits. The strong competition of syslog-vendors in the Windows space (yes folks, there *is* competition and Microsoft has not yet joined in it...) has created much better products. I assume that two *strong* competitors in the *nix workspace can do a similar thing over there. The end result will be much better than what we have today (which is admittedly not bad with syslog-ng and rsyslog in their current releases).

So - why does the world need another syslogd?
It needs one that aims to be a real major player, being installed on a lot of systems. That will help to get the best out of syslog technology (and in the long term the best of logging at all). Either in its project itself of by driving competitors to be better than it. A new major player will prevent monocultures and provide a rich freedom of choice. That's why the world needs it.

Rsyslog tries to live up to that challenge. Of course, it is not sure if it succeeds. But should we try only things we know for sure to be successful? I don't think so - if mankind followed that maxime, we'd probably still be sitting on trees hunting for bananas. As a side note, the critical path for rsyslog will not be features or developer commitment. What is critical is you - critical is whether or not we can manage to build a community around it. And that, too, looks pretty well these days...

"The Clouds" online for July 2007

I was quite busy, but I finally found some time to compute "the clouds" animation for the month of July 2007. The clouds is one of my fun and educational projects. I initially thought of it as an astronomy project, but it soon turned out that it can be equally useful for understanding geography and, of course, physics in general.

The coulds offers stunning, satellite-based views of earth's revolving cloudbands as seen from weather satellites. So far, I used EUMETSAT services, but plan to expand that to other satellite providers, too.

I usually create the animation in the first week of the month. This time, it took much longer. Part of it was that I was quite busy with rsyslog. The other had to do with my laziness. When I started the project back in early may, I thought it would be a one-shot for one month. Consequently, I pulled the sat images via a "secondary" system without any fault tolerance. When it turned out that "the clouds" would run for an extended period of time, I changed none of the setup. And, guess what - the hard disk failed early August. I didn't even monitor that system for health (my second fault, especially *I* should have done that...). So I didn't notice until a few days later.

The really bad thing is that I couldn't get any past images. EUMETSAT thankfully offers current satellite images without charge for non commercial use. However, the archive is a paid option. So I initially feared I'd lost all of July. Thankfully, I could recover that satellite images. So July 2007 is complete. I lost four days at the beginning of August, but that doesn't hurt so much. And, yes, I've at least learned a bit out of the whole situation: The image grabber now runs on two systems concurrently and both of them store images on mirrored disks. That should keep my on the safe side.

Friday, August 10, 2007

a wiki for rsyslog

A modern open-source project, at least IMHO, needs a number of supporting web services: a blog, a bug tracker, a forum ... and a wiki. For rsyslog, most of this services existed for quite a while. What was missing until now was a wiki. Thanks to Andre Lorbach who set it up, we now have one at http://wiki.rsyslog.com. I have finally found time to add at least some information into it.

And now it is the users turn. The wiki is not meant to be a replacement for the official doc. Instead, it should contain user contributed content and ideas. So it is up to you to fill it with live. Of course, the rsyslog devel team will have a keen eye on the wiki and be as helpful as possible. Please give it a try and help creating a great community-driven wiki.

rsyslog changes for 2007-08-09

Yesterday I made a couple of changes:

- fixed a bug in outchannel code that caused templates to be incorrectly
parsed
- fixed a bug in ommysql that caused a wrong ";template" missing message
in some cases
- some text and code cleanup in cflineParseTemplate() to match the changed
design of selector handling
- added new doc files to autotools for make dist
- fixed a bug that would have manifested when module configuration commands
would have been freed. The current code base, however, never does this, so
this is not a real bug fix - it's in preparation of future work
- fixed a potential memory leak in modules.c, again, this could not happen in
current code
- added interface API for unloading module (a dummy)
- added module unload functionality; rsyslogd now unloads modules on exit
(of course, with only statically linked modules, there is little current
value in this - but it is made towards an upcoming dynaload plugin interface)

Wednesday, August 08, 2007

rsyslog changes for 2007-08-07 & 08

This time, I blog the changes for both days. Today was again quite busy, with a number of smaller changes. But all in all you can see that the pace of changes is reducing. I hope that people will test it, but I also hope that we will see no more big bugs the next few weeks. Then, I consider it to be mature for a new stable release.

You will probably see no or very few code changes during that period. You can expect, however, some doc upgrades. All in all, my focus has now moved to the next major release, which I am currently designing. And of course, some other things are also asking for a bit of time now that I have worked very intensely the past two month on rsyslog.

Stay tuned. Any news will appear here in the blog. In the mean time, try 1.18.1 and tell us what needs to be fixed (of course, you may also just praise our work, that's fine, too ;)).

Here is the work log:

2007-08-07

  • applied patch from Michel Samia to fix compilation when the pthreads feature is disabled
  • moved code to create/delete message queue to init(), after reading the conf file. This is a prequisite to allow specifying the queue size in the conf file.
  • add config directive $MainMsgQueueSize, which now allows to configure the queue size dynamically

2007-08-08

  • moved the "after select polling code (for fds)" to its own function
  • optimized select handling, after select polling cycle is now finished when all active selectors are processed
  • all compile-time settings are now shown in rsyslogd -v, not just the active ones
  • changed function name dprintf() to dbgprintf() as it conflicts with the clib
  • added config file directive $ActionResumeInterval
  • solved an issue with debian sid. LARGEFILE preprocessor defines are changed in rsyslog.h, which causes grief for zlib under sid. As a temporary solution, I have moved rsyslog.h right at the beginnng of the include order. It's somewhat dirty, but it works. I think the real solution will be inside the autoconf files.
  • added a database cleanup script based on a suggestion of Michael Mansour. Michael Meckelein adapted it slightly for use with rsyslog.
  • released 1.18.1
  • fixed bug: default for $DropMsgsWithMaliciousDnsPTRRecords was not reset on up and on $ResetConfigVariables

Tuesday, August 07, 2007

next generation rsyslog design, worker pools, parellelism and future hardware

Based on my last post on rsyslog multithreading, an email conversation arose. When I noticed I elaborated about a lot of things possibly of general interest, I decided to turn this into a blog post. Here it is.

The question that all this started was whether or not a worker thread pool is good or evil for a syslogd.

Yes, the worker-thread pool has a lot of pros and cons with syslog. I know this quite well, because I am also the main design lead behind WinSyslog, which was the first syslogd available natively on Windows (a commercial app). It is heavily multi-threaded. I opposed a worker pool for a long time but have accepted it lately. In some high-volume scenarios (unusual cases) it is quite valuable. But of course you lose order of messages. For WinSyslog, we made the compromise to have the worker pool configurable and set it to 1 worker if order of events is very important. I designed WinSyslog in 1995 and released the first version in 1996 - so I know quite well what I am talking about (but to be honest the initial multi threading engine got in somewhat later;)).

HOWEVER, especially in high-volume environments and with network senders, you are somewhat playing Russian roulette if you strongly believe that order in which events come in is exactly the same order in which they were sent. Just think routers, switches, congestions, etc... For small volumes, that's fair enough. But once the volume goes up, you do not get it 100% right. This is one of the reasons I am working quite hard it the IETF to get a better timestamp in into syslog (rsyslog already has it, of course as an option). The right thing to do message sequencing is by looking at a high-precision timestamp, not by looking at time of reception.

For rsyslog, I am not yet sure if it will ever receive a worker pool. From today's view, it does not look necessary. But if I think about future developments in hardware, the only key in getting more performance is by using the enhanced parallelism the hardware will provide. The time of fast single cores is over. We will have relatively slow (as fast as today ;-]) massively parallel hardware. This is a challenge for all of software engineering. Many folks have not yet realized it. I think it has more problem potential than the last "big software crisis". As you can see in the sequencing discussion, parallelism doesn't only mean mutexes and threads - it means you need to re-think on how you do things (e.g. use timestamps for correlation instead of reception/processing order, because the later is no stable concept in a massively parallel program). With the design I am now doing, I will definitely have separate threads for each output action (like file or database writer). I need this, because rsyslog will provide a way to queue messages on disk when a destination is not available. You could also call this "store-and-forward syslog", just like SMTP is queued while in transit. You can not do this concept with a single thread (at least not in a reasonable complex way). I also do not think that multiple processes would be the right solution. First off, they are too slow. Secondly, multiple processes for this use are much more complicated than threads (I know, I've written similar multi-process based things back in the 80s and 90s).

Rsyslog will also most probably have more than one thread for the input. Input will become modular, too. If I use one thread per input, each input can use whatever threading model that it likes. That makes writing input plugins easy (one of my goals). Also, in high volume environments, having the ability to run inputs on multiple CPUs *is* *very* beneficial. One of the first plugins will be RFC 3195 syslog, which is currently run as an external process (communicating via unix local domain socket, which is quite a hack). At the core of 3195 is liblogging, a slim and fast select-server I have written. However, integrating it into a single-threaded application is still a challenge (you need to merge the select() calls and provide an API for that). With multiple threads, you can run that select server on its own thread, which is quite reasonable (after all, why should both UDP and 3195 reception run on the same thread?). The same is true for tcp based processing. Then think about the native ssl support that is coming. Without hardware crypto acceleration, doesn't it make much sense to run the receiver on its own thread? Doesn't it even make sense to run the sender on its own thread?

So the threading model of the next major release of rsyslog (called 3.x for reasons not yet elaborated about) will most probably be:


  1. multiple input threads, at least one for each input module (module may decide about more threads if they like to)
  2. all those input threads serialize messages to a single incoming queue
  3. the queue will most probably be processed by a single worker thread working on filter conditions and passing messages to the relevant outputs. There may be a worker pool, but if so its size can be configured (and set to 1, if needed)
  4. multiple output threads, at least one for each output. Again, it is the output's decision if it runs on more than one thread
  5. there may be a number of housekeeping threads, e.g. for DNS cache maintenance


This design will provide superb performance, is oriented on logical needs, allows for easy intrgration of plugins AND will be reasonable easy to manage - at least I hope so.

But, wait, why have I written all this? OK, one reason is that I wanted to document the upcoming threading model for quite a while and now was a good time for doing so. But I think it also shows that you can not say "multithreading is good/bad" without thinking about the rest of the system design. Multi-threading is inherently complex. Humans think sequentially, at least when we think about something consciously. So parallel programming doesn't match human nature. On the other hand, nature is doing things itself massively parallel. Just think about our human vision: in tech terms, it is massively parallel signal processing. This is where technology is also heading to. As we have learned to program at all, we will also learn to deal with parallel programming. A key thing is that we should never think about parallelism as a feature - it's actually a tool, that can be used right or wrong. So let's hope the rsyslog design will do it right. Of course, comments are very much appreciated!

Monday, August 06, 2007

why is rsyslog multi-threaded (and is it really?)

I found a quite interesting discussion on a debian mailing list on why it may be useful (or even evil) for rsyslog to be multi-threaded.

It's late now, so I can not elaborate in full, but I thought I share a little feedback. First of all, my project description is actually cheating a bit at this time: rsyslog is currently actually dual-threaded, with one thread for message reception and one thread for message processing. And this is even optional, so you can turn it off with a configure option to make it single-threaded. Future versions of rsyslog will have much improved threading model and that has much to do with modular input and output - but that's a different story.

Run rsyslog compiled for single threading and make it write to a MySQL database, then throw lots of messages at it (let's say maybe 500 per second, which is not actually much). You'll soon see that messages are lost, because the receiver can't receive new messages while the MySQL output module (message processor) is writing messages. So reliability and message reception capability is limited by the slowest output module in use.

With a multi-threaded model, this problem does not exists. Receiver and output modules are de-coupled via an in-memory queue. So when the MySQL server is to slow for the current message rate, messages are buffered in the queue, but the receiver can continue receiving messages. As we have UDP, this is quite important. Please note that in a failover case, the output module may be busy for as long as maybe even a few seconds. This time may be needed to probe if the primary has really failed and then connecting to the secondary. The whole idea not only applies to MySQL (though it is most obvious in this use case) but also for TCP based syslog forwarding. Here, the receiver can actually slow down the sender, so the same problem may occurs. De-coupling receiver and output modules is also of great benefits in high volume environments. It enables bursts to be processed reliably where messages were otherwise lost.

So the main reason for the current threading model is reliability and striving to lose as few messages as possible. Of course, there is also a performance benefit, especially if you think that in the future we will see machines with many more cores - but a single core not beeing considerably faster than today. Rsyslog's current threading model is not up for this challenge yet - that will happen later this year. Then, each output will receive its own thread. There are good reasons to do that, for example the need to support re-startable action with local disk queueing (aka "store-and-forward syslog). Without parallelism, this is not doable. But I'll elaborate on that later as the object and threading model design grows.

And, yes, I share concerns that threading makes code more complex. This is the #1 reason that threading is still a compile-time optional feature in the current rsyslog. But as the project evolves, we need to have multiple threads to support a clean object model. And, yes, again, that sounds heavy. All I can say is that I am a performance hog, so you won't probably see rsyslog taking up huge amounts of CPU just for the sake of having an object model. But of course, I may be cheating here, too (can you trust me? - stay tuned and you'll see...).

I hope my short notes help to understand a bit why the current rsyslogd uses multiple threads.

EDIT: I have now found some time to elaborate on the upcoming rsyslog threading model and multi-threaded programming in general.

rsyslog changes for 2007-08-06

I have been on the road today, so not much happened. But at least some things were done ;)

  • varmojfekoj sent me a patch for some selector_t init code. It patched re-init, where rsyslog should happen to segfault (it didn't do it, but why not is a mystery...) - applied it
  • changed action resume interval to a more reasonable 30-second intial value
  • moved action object out of syslogd.c to its own fileset (action.c/h)

Keep in mind that the pace of changes will now go down for a few weeks. The code needs to mature. I'll probably focus more on docs AND *other* projects in the mean time. I guess I'll also work on no-code things, like config file format and final object design (more on those later, this was just a teaser ;)).

Saturday, August 04, 2007

syslog changes on 2007-08-03

here's friday's work log:

- added CODE_STD_FINALIZERparseSelectorAct to module-generation macros
- restructered rsyslogd startup - moved startWorker() to a more
appropriate
place
- updated ommysql.c to fully support suspension/resumption by rule
engine
- I found out that we finally have problems with the (somewhat
recursive)
call to logerror() that many of the modules do. I have not tried it,
but
I think things will become wild when we compile without pthread
support.
Threading prevents full recursion, so we have not seen any bad effects
so far. However, the problems that I experienced in ommysl (that
caused
me to re-structure startWorker()) are actually rooted in this issue.
I
first thought to fix it via a module interace, but I now came to the
conclusion that it is not more effort and much cleaner to do an
internal
error buffering class. This is implemented in errbuf.c/h.
- I just noticed that this is not actually an error buf, but the core of
an input module for all internal messages. As such, I implement it now
as iminternal.c/h. Of course, there is no input module interface yet
designed, but that doesn't matter. Worst-case, I need to re-write the
im, best case I can use the im (at least partly) to define the
interface.
- added a few functions to the linkedlist class
- error messages during startup are now buffered - so we do no longer
need
to think about how emergency logging might work. Actually, these are
logged
to whatever is instatiated in the log file. This enhances the chance
that
we will be able to drop the error message somewhere it is seen.
- released 1.18.0
- reduced number of identical error messages in ommysql.c while
suspended

Thursday, August 02, 2007

rsyslog changes for 2007-08-02

For those curios, here is todays work log as far as it relates to rsyslog:

  • enhanced linkedList class, new method to get count, new method to execute a user-supplied function on all members
  • enhanced syslogd memory structures to support multiple actions per selector however, this can not yet be configured due to missing config code for this case
  • moved cfsysline-calling into cfline() - now a unified interface again
  • enabled functionality to have more than one action per selector
  • changed doAction() syslogd internal functions to allow for larger data & state data - in preparation for actions that shall only be executed when previous action was suspended (the switchover case, e.g. for failed databases or TCP receivers)
  • added tryResume() API to module interface
  • added resumption logic to rsyslogd (but not yet any module)
  • got a working version of suspension/resumption logic including omfwd.c
  • implemented $ActionExecOnlyWhenPreviousIsSuspended config directive
  • MILESTONE REACHED: we now have the ability to switch over to a
    different syslog/tcp server if the primary one fails. It is also done via a generic interface, so it should be sufficiently simple to extend that interface to other actions, namely the database.
  • some cleanup
  • begun to work on ommysql.c

finally: ability to automatically switch to backup servers

I've achieved a long-term goal for rsyslog today: I have finally managed to include the ability to automatically switch to backup actions (servers, databases, whatever ;)) when the primary fails. I have implemented it in a very high level inside the syslogd rule engine, and so it is quite powerful.

Just read this doc excerpt:

ActionExecOnlyIfPreviousIsSuspended
This directive allows to specify if actions should always be executed ("off," the default) or only if the previous action is suspended ("on"). This directive works hand-in-hand with the multiple actions per selector feature. It can be used, for example, to create rules that automatically switch destination servers or databases to a (set of) backup(s), if the primary server fails. Note that this feature depends on proper implementation of the suspend feature in the output module.
The following is an example of what can be done with it (a config file excerpt):

*.* @@primary-syslog.example.com
$ActionExecOnlyIfPreviousIsSuspended on
& @@secondary-1-syslog.example.com
& @@secondary-2-syslog.example.com
& /var/log/localbuffer
$ActionExecOnlyIfPreviousIsSuspended off

This selector processes all messages it receives (*.*). It tries to forward every message to primary-syslog.example.com (via tcp). If it can not reach that server, it tries secondary-1-syslog.example.com, if that fails too, it tries secondary-2-syslog.example.com. If neither of these servers can be connected, the data is stored in /var/log/localbuffer. Please note that the secondaries and the local log buffer are only used if the one before them does not work. So ideally, /var/log/localbuffer will never receive a message. If one of the servers resumes operation, it automatically takes over processing again.

As is written in the doc, this ability depends on proper handling of the new suspend-interface in the output driver. So far, I have only updated the omfwd driver. I plan to do the mysql driver soon. I'll possibly also look after the file driver, but suspension there seems like a foreign concept. The other drivers don't support it (nothing can go wrong enough), so I don't bother about them.

All in all, I am quite happy and I think this is a very useful feature. It will be part of 1.18.0, currently it is "just" in anon cvs. I hope to release 1.18.0 tomorrow - let's see if that works.

I am also glad that this example shows the output interface (and a whole lot of the other new internal plumbing) works pretty well. I think we are in an excellent position for all the other new things to come ;)

Wednesday, August 01, 2007

rsyslog changes for 2007-08-01

Today, I just post the work log. I think it speaks for itself ;)

  • loading default values as done yesterday *is* clean (I've just reviewed the code again)
  • moved omfile-specifc global variables (for cfsysline settings) from syslogd.c to omfile.c
  • MILESTONE reached: initial output modularization is completed. Of
    course, we will work on the interface some more, but all basic goals are reached.
    Note that there are also some accesses of global data and functions, but
    that is not a real design issue but something that can be solved by moving
    code to supporting libraries (which will occur later). Also note that
    output module availability does not indicated the presence of a plug-in
    interface. That will happen some time later when the output module interface has somewhat more matured (my personal bet is in around two month from now).
  • fixed an invalid value for the MARK timer - unfortunately, there was a testing aid left in place. This resulted in quite frequent MARK messages
  • some code cleanup (unused vars and such...)
  • enhanced cfsysline interface to disallow chaining of command handlers
  • added $include config directive
  • applied a patch from mildew to prevent rsyslogd from freezing under
    heavy load. This could happen when the queue was full. Now, we drop messages but rsyslogd remains active.
  • fixed bug in cfsysline that caused iRet to be lost during handler call
  • enhanced $Include to allow inclusion of a complete directory
  • changed name of $Include to $IncludeConfig - sound better as it is
    more explicit (we'll see where else we get includes for...)
  • released 1.17.6
  • modified rklogd to only change the console log level if -c is specified
  • even more cleanup ;)
  • prepared code to move action-specifc selector_t data members to their own structure
  • moved action-describing data fields out of selector_t to its own structure; changed code to use this dynamically-allocated structure; this is in preparation for multiple actions per selector
  • moved definition of selector_t to syslogd.c again, as this now no longer is a global structure
  • shuffled code in cfline() to prepare for a better system of processing the selector_t linked list

Busy at the moment...

Some might have noticed that I am not as active as usual on the rsyslog project . As this seems to turn out to keep at least for the upcomi...