Friday, September 28, 2007

on ommysql packaging

I thought I share an interesting conversation that went over the rsyslog mailing list. Even though it buried in the mailing list archive, I think here is a better place to be displayed. Feedback is much appreciated.

> -----Original Message-----
> From: Michael Biebl
> Sent: Friday, September 28, 2007 10:23 AM
> To: rsyslog-users; Rainer Gerhards
> Subject: Re: [rsyslog] rsyslog 1.19.8 released
> 2007/9/27, Rainer Gerhards :
> > repeat message processing. The MySQL functionality is now taken out
> of
> > the core package, but its tarball is still contained in the main
> tarball
> > (so it is still a single download for everything). This is part of
> the
> > effort to fully support third-party plugins. Rsyslog 1.19.8 is a
> To be honest, I don't particularly like this change. It increases the
> work for package maintainers (like me). You now have to maintain two
> source packages. Having a non-standard tarball inside a tarball
> doesn't help there. It even worsens things, as stuff like "make dist"
> or "make distcheck" doesn't work anymore for a cvs checkout.
> There's also the problem of which version of the plugins will be
> compatible with the core version (upgrade scenarios, keeping them in
> sync, etc.)
> It also adds complexity for the developer (as he has to maintain an
> additonal set of build files).
> As you were talking about 3rd party plugins (with the emphasis on 3rd
> party) I don't understand the benefit of splitting out the mysql
> plugin as it is you who develops the mysql plugin, not a third party.
> Do you actually intend to create a separate tarball for each plugin in
> the future?
> What was wrong with the --enable-mysql configure switch? I don't see
> any benefits, only disadvantages. You know, if it ain't broken, why
> fix it ;-)
> Cheers,
> Michael

Hi Michael,

as I have blogged, I am not yet sure about how to handle the situation. I am also consulting with Autotools experts, any advise is appreciated.

Two packages seem useful, especially when more plugins become available (I myself think about email and a couple of other databases). Many folks also do not like the idea of having to have libmysql present on the system just to install rsyslog - with a core package and the plugin, those can only install the core (and that will probably the majority of cases).

What I have not yet found - and I have very limited expertise in this area - is how to do that in the best possible way.

Oh, and some more background: ones the plugin interace has matured (I expect this in 3 to 6 month), I intend to actually use ommysql with its own version number. Versioning will be handled by the interface (that part is already present, but no code for it yet as there is only a release 1 of it). So, in the medium to long term, ommysql *will* be a separate project - maybe one with a different maintainer, as I am no mysql guy.

Does this make sense? As I said, comments are much appreciated...


Thursday, September 27, 2007

work log for 2007-09-26

work log for rsyslog:

- applied patch provided by varmojfekoj to support building ommysql
in its own way (now also resides in a plugin subdirectory)
- fixed a bug in cvthname() that lead to message loss if part
of the source hostname would have been dropped

Wednesday, September 26, 2007

Did not manage to release 1.19.8 today...

Sorry folks, I didn't manage to release 1.19.8 today. There was simply too much other activity that required attention. So I couldn't finally decide on how to distribute from now on. Anyhow, it is my firm intention to release 1.19.8 tomorrow.

The good news, though, is that some folks are already testing that code. Maybe I get some feedback until tomorrow morning - and maybe that even helps me to get a better release...

needed to pull 1.19.7 release

Unfortunately, there is a serious bug in rsyslog 1.19.7 that can prevent UDP message reception. Totally. No message received at all. This somehow slipped through our own testing as well as test at some third parties.

The root cause of this problem is interesting: I changed an internal interface to make things more reliable. What I changed had some old code in them that did return's right in the middle of the code. I overlooked those returns and so an invalid state was returned.

The interesting fact is that the function now returns an enum type (rsRetVal). Previously, it returned an integer. For some reason, the compiler issued no warning when the old (accidentally remaining) code returned an integer. If I'd receive a "wrong type" warning, I'd probably spotted it before even doing testing at all.

Bottom line: what's wrong with my compiler settings?

Oh, and yes: I'll release the fixed version soon. It then will be 1.19.8 to avoid confusion. I now fight with the distribution system: I received a patch that allows ommysql to be build as a separate module. So now it is separate - even from a distribution point of view. That causes some grief for distributing it and can possibly break some things in distribution packages. I need to think how to tackle that in the best possible way...

worklog for 2007-09-25

- changed ttyname() to ttyname_r() - not a real fix, as this part of the
code was single threaded, but better to be prepared for the future.
- changed localtime() to localtime_r()
- released 1.19.7
- applied contributed patch to improve repeated message processing

Tuesday, September 25, 2007

rsyslog changes until 2007-09-24

Hi all,

I've been a bit lazy reporting what I've done with rsyslog. The primary reason is that it was a lot of review, which is quite boring to report in a blog. Today, I think, I'll do a new release, so at least here is my worklog for up until yesterday.

- possibly found a bug in cvthname() that lead to a wrong size being specified
in a getnameinfo() API call - not sure, though, if it is "the" bug (actually,
it looks like it isn't). - this is EXPERIMENTAL
- fixed a bug that caused signal handlers in cvthname() not to be restored when
a malicious pointer record was detected and processing of the message been
stopped for that reason (this should be really rare and can not be related
to the segfault bug we are hunting).
- split the function cvthname() for clarity. Also changed to using the rsRetVal
status return system
- removed some compiler warnings in regard to signed / unsigned comparison
- code cleanup
- fixed a minor memory leak that occured when the %APPNAME% property was
used (I think nobody used that in practice)
- more review and cleanup
- simplified code in shouldProcessThisMessage() for debug output
- changed strerror() calls to thread-safe strerror_r() variant

Thursday, September 20, 2007

What's going on with rsyslog?

I've not posted much the past days. That doesn't mean nothing has happened. I wanted to post a work log today, but I have to admit I have forgotten it on a machine I right now can not access. OK, first thing tomorrow morning...

In short words, we are still on the bug hunt. I am now again back to reviewing code, this time on a functional basis. I am checking everything based on the message flow, looking at functions as they are called. Today, I completed the review of the reception part (up to the point when it gets into the main message queue). Unfortunately, no serious problem found. I used the review, however, to clean out some nits, add a large number of new comments and even found a memory leak. The later one would currently most probably never occur in practice, but when syslog-protocol gets adopted, it would have hit.

Tomorrow, and probably then next two or three days I'll review the code that is executed once the message leaves to main queue. There is probably more meat for a bug in that part (its by far more complex).

I also think I'll release the cleaned-up version sometime soon - after all, it's better then what is currently released.

I keep you posted. Comments are always welcome.

Wednesday, September 19, 2007

a dedicated shuttle blog

Ah, the joys of blogging. When I started this blog a few years ago, I kept it focussed on one topic - and thus had a few other blogs for other topics. Then, these nice labels appeared in blogger. That looked like an ultimate solution to me: just put everything in a single blog and then use the labels (or tags, as others would call them) to generate topic-specific feeds. Now is the the first time that I really did this. And, guess what, it doesn't seem to work as nicely as I initially thought.

Mixing rsyslog/logging and a field trip to the space shuttle launch causes some confusion. I do not like confusion ;) So I have created another blog for my space shuttle launch viewing trip today.

I hope this will work out. I sincerely think it is in the best interest of all readers. The next days (as time permits, its obviously not a priority task...), I'll create some useful links to get the different pieces together if you are interested in the big picture. Also, I'll remove the shuttle posts here and change them to redirects.

Tuesday, September 18, 2007

paid services for rsyslog

As with a lot of open source projects, rsyslog funding is problematic. Of course, rsyslog currently is funded by my interest in it. And I am glad that Adiscon, my company, permits me to work on. The actual funding for all that action, kind of funny, mostly stems from closed source projects in the Windows world.

However, I'd like to see that over the years rsyslog can fund itself. In my point of view, funding should be provided by those that benefit most from it. So obviously, I do not expect any funding from private folks, people like you and me. Of course, everyone is invented to contribute and new code, doc and bug hunting is definitely a pro. On the other hand, companies (and other organizations, namely the government) take financial advantage of using it. Some of them also contribute in terms of time and code. This is great and much appreciated :)

For the others, I have begun to offer donations. Not surprisingly, this is not a real source of funding ;) But now I have taken an offer from, which I think is interesting. They begin to offer a marketplace for open source services. So those folks that actually need help can be brought in contact with those that have experience. Sounds like a good idea to me and a fair way to fund projects.

I have created two test service offerings for rsyslog. One is to write a nice custom-created rsyslog configuration file. Well, with rsyslog's relative ease of use, I do not expect that many folks use that. But you never know and I am always curios.

The other one is especially targeted towards organizations who must prove they have "official" support for all software they use (I guess this includes at least a number of government agencies). For them, I have created a email support option for rsyslog. It guarantees responses to support questions and this is often needed for auditing and other purposes. Of course, we still answer all support emails and do not plan to stop that.

I have added one real goody, though: and that is that we will provide patches for past releases of rsyslog. As you may know, developers hate to fumble with old releases. And so there is very little motivation to look at an older release when there is a new one out. With the paid support option, however, there is some motivation to do that. So, again, I think this is fair: we are just offering a service that would otherwise never appear. It doesn't hurt any of the other users.

I am very interested to see how this works out. I would also be interested in feedback from the field. How do you like this idea? Do you have any other/additional suggestions?

space shuttle troubles...

Space Shuttle Strut Repair
Yesterday, some info leaked that the shuttle ... well, leaked ;) To be serious, there were reports that there is an unacceptable leak at a shuttle strut - hydraulics fluid seems to have been leaking. But yesterday the decision was made that a repair is actually needed.

Of course, it didn't take long for all sorts of rumors to appear. Some sources even said that the October mission would be canceled and moved to January - which is as far from being true as it only can be. In fact, the NASA source quoted above does not outrule there is a change in the launch date, but it is expected to be not a major hit. As far as I understand it, things may be moved a few days at most. There seem to be buffers all along the process, so I do not yet begin to panic ;)

... but I have to admit that this triggers bad memories. As I said, I flew in into Orlando last year to see the STS-115 launch (which I finally didn't make due to its long delay). With STS-115, all the trouble also started with launch delays, that time caused by a lightning strike. That, too, was quite some time before launch, at least if I correctly remember. It was not that early as the current problem, which leaves me with the firm hope that there is enough buffer time available even to launch on the target date of October, 23rd. Let's see...

Photo: In the Orbiter Processing Facility at NASA's Kennedy Space Center, workers secure the tool storage assembly unit into Discovery's payload bay. Photo credit: NASA/Amanda Diller

Monday, September 17, 2007

viewing a space shuttle launch..

Space Shuttle Launch
As some of you know, I am addicted to astronomy and space travel. Since long, I'd wanted to experience a space shuttle launch. Last summer, I got tickets for the STS-115 mission. I went down to Florida, went through a hurricane and ... had to leave without the shuttle being launched. Well, actually I could view it rocketing into space from far away (Fort Myers) when I had to leave home. That was really bad luck.

Unfortunately, I either could not go for the next missions or I didn't manage to get tickets (they sell out soooooo quickly that it is a real problem even if you type fast ;)).

Now, I was lucky enough to secure tickets and also manged to get enough vacancy to do a re-try. I am going to visit the STS-120 mission now. We'll fly in to Orlando and then move over to the Titusville/Cocoa Beach area. Getting it all together was far from being simple.

After I got the launch tickets, I needed to get flights and then find hotels. This list is sorted in descending order or rareness ;) While I had only a 3-minute shot at obtaining the tickets, the flights were quite complicated too. Hotels were available, but of course not those I hoped to find. Based on experience from fellow travelers, Titusville seems to have only one decent hotel. I can back this, as I found none on my previous trips. The one in question is the Hampton. Everybody seems to know, and I didn't get a room for the 22nd ;) But everybody also seems to expect the shuttle to take off on first launch attempt - because starting from the 23rd there were vacancies. I secured some of them ;)

I hope to have a really great launch experience. I hope I'll find time to post more of my experiences on the way to the launch here in the blog. I'll probably even start a STS-120 section on my site - let's see :)

Friday, September 14, 2007

recent rsyslog changes

Again, I am doing small changes, mostly review, at this time. So I have batched up some things. Probably I'll switch back to a more daily mode, as it allows to keep better in touch. We'll see (comments appreciated).

Here comes the log:

- bumped version number to 1.19.6
- fixed a bug that in --enable-debug mode caused an assertion when the
discard action was used
- added eCmdHdlrGetWord command handler
- added $ModDir config directive
- modified $ModLoad so that an absolute path may be specified as
module name (e.g. /rsyslog/
- applied patch by varmojfekoj two fix two potential segfault situations
- cleaned up some signed/unsinged char issues
- released 1.19.5
- bumped version number to 1.19.6
- fixed a bug that in --enable-debug mode caused an assertion when the
discard action was used
- applied patch by varmojfekoj to change signal handling to the new
sigaction API set (replacing the depreciated signal() calls and its
- did a full review of all remaining TODO items in code - nothing of
importance found (but some minor nits like comments fixed)
- cleaned up compiler warnings
- applied patch by varmojfekoj to FIX a bug that could cause
segfaults if empty properties were processed using modifying
options (e.g. space-cc, drop-cc)
- fixed man bug: rsyslogd supports -l option
- checked -s/-l option and found that they work as expected - closed case
- added some comments in relation to -s/-l option
- released 1.19.6
- removed compiler warnings in non-debug mode
- fixed a bug that caused the CStr class to not honor the parameter to
return NULL on empty string - causes a mem leak and can create favourable
environment for other bugs (as it leads to empty hostnames)
- pulled 1.19.6 (no downloads so far), created new 1.19.6 with the bugfix,
re-released that (now tagged v1-19-6b in cvs)

Tuesday, September 11, 2007

bug hunting...

We have received feedback that the 1.19.x releases of rsyslog contain a bug that leads to a segfault after some time. Of course, this is a very bad thing to happen. After all, the primary goal for the rsyslog project is reliability and it should be able to survive even in the worst conditions (e.g. low system memory). So this bug is clearly unacceptable and has received highest priority.

Unfortunately, it is hiding very well. We are all looking into troubleshooting this beast. Thankfully, there is a lot of community support, especially with testing. What makes this bug so hard to find is, among others, the inability to reproduce it in lab. So the project is totally dependent on user feedback. As current reports have shown, we also currently assume that releases prior to 1.19.0 can also be affected - so it is not as easy as checking the changes.

I receive very valuable help by varmojfekoj. He is already a frequent and great contributor of patches. His skills and experience are extremely valuable and I am very glad to have him work on this project. Please give him a big hand. Thanks, varmojfekoj, the project would not be that far without your help!

While the bug itself is obviously a very bad thing, it has some good side-effects. Most importantly, the code is getting another round of very in-depth review. That review has already brought up some fixes for obscure situations. Situations that I'd expect to happen extremely infrequently (if at all) in reality. But now even those are fixed. And I am sure that the review will bring even more benefit as it continues. Of course, I'd be glad if we find the bug ASAP, but it is good to know that all that work also provides additional benefit (at least that keeps my depression level low ;)).

The bad thing, besides the bug itself, is that the bug hunt obviously defers other work. Most importantly, I refrain from making any changes that may not be related to clean up or bug fixing. I think it is not smart to introduce new code (read: complexity) at this stage of the project. And, of course, all resources should focus on fixing that bug, so there wouldn't even be time to enhance other parts of the code. So for all of you waiting for new features: please bear with us, the implementation schedule will slip a little bit.

If you experience stability problems, please report. Each reports helps us understand the problem somewhat better.

Friday, September 07, 2007

recent rsyslog changes

Now that I am back working on the rsyslog code, I will continue to provide information about what I am doing. In most cases, however, I will not provide daily logs. The reason is that I currently focus on new design and will leave rsyslog code mature. So changes will be relatively infrequent (at least I plan so ;]).

Here is the work log for the past days:

- integrated patches from Michel Samia and varmojfekoj
- released 1.19.4
- changed some calls to CStr class to their "safe" counterpart - they could
cause program aborts if the object in question was an empty string
- added some links to doc
- removed an invalid config sample from sample.conf
- changed part of the CStr interface so that better error tracking
is provided and the calling sequence is more intuitive (there were
invalid calls based on a too-weired interface)
- (hopefully) fixed some remaining bugs rooted in wrong use of
the CStr class. These could lead to program abort.
- added forwarding information to sysklogd (requires special template)
to config doc

Thursday, September 06, 2007

rsyslog config again...

I am writing a lot about rsyslog config file formats these days. Now I think I come closer to a result. The past days, I have thought a lot about potential formats. Now, I have created configuration samples in the rsyslog wiki. Also, some others have posted hints.

I now have two favorites: one is a XML-based format. Its advantage is that it would be easy to use when (if) we implement NETCONF protocol. If I assume NETCONF will actually become *the* configuration standard, this is a big pro argument for that format. However, I consider the format to be exceptionally bulky and pretty less intuitive. In other words, I do not really like it - but that doesn't mean it is a bad choice.

What I like more is a programming-language like format (and especially the last sample in the wiki article). It looks somewhat more "natural" to me, but this may be because I like programming languages ;) In any case, I think it is easier to grasp and modify, but of course it is not standardized. Plus, there are three different samples, each in a different style and level of verbosity.

The question is now what users consider to be best. I sincerely hope to receive feedback, as I probably have only one "shot" at this for the foreseeable future (well... I have to admit that I thought about dynamically loadable config file parsers, but that adds probably too much complexity).

So if you have anything to say, please speak up now!
If you do not know where to make yourself heard, simply post to the rsyslog forum thread on the config file format.

Wednesday, September 05, 2007

where is liblogging heading to?

I just received a question whether or not I plan to change the build system for liblogging to a modern one, just like I have done with rsyslog recently.

As a reminder, liblogging is the library that provides RFC 3195 functionality to rsyslog. Of course, others can integrate it too. Writing liblogging was a considerable effort. When I did this, I had hoped that RFC 3195 would become much more dominant in the logging space. Unfortunately, the current version became a failure. This is also one reason that rsyslog supports receiving messages via 3195 but not sending them - the interest was too low (to phrase it politely) and other features received priority.

Liblogging development is currently "on hold". I was tempted to completely abandon it, but then Cisco moved RFC 3195 support into IOS. That was one thing that made me hope that the standard may become accepted in the long term. Recently, while designing rsyslog v3, I noticed that rfc 3195 (or more precisely BEEP, RFC 3080) indeed offers a very good choice for protocol extensions. I am currently very seriously thinking about re-vitalizing liblogging and make it the foundation of rsyslog-to-rsyslog communication.

But there are also a number of other subtleties. Most importantly, the current RFC 3195 is insufficient for practical purposes. For example, it limits message size to 1K. It also does not support new syslog headers. The IETF is considering a new version of 3195, which will fix these shortcomings. Of course, I can go ahead and use from 3195 what makes sense to me and extend the rest on my own. But that will cause trouble in the long term and I do not really like that.

There are also some funding issues. As you possibly know, we fund our projects also via closed-source Windows software (e.g. MonitorWare Agent or EventReporter). The current liblogging has a BSD-style license. That was chosen in the hope it will help to make it a standard. However, it also enables our closed-source competitors to pull our work without contributing anything back.

So, to be honest, if we go ahead and do a new substantial effort on liblogging, we will probably release it under GPLv3. That enables everybody who does open source to use it, but prohibits our closed-source competitor to integrate the library. Of course, we need to retain the right to use it ourselfs in our commercial products, but I think it can be arranged in a way that is compatible with GPLv3 - at least I have seen a number of projects doing that.

To summarize in short words: liblogging is currently on hold and will be for at least some more weeks. We are deciding if we extend it. If so, that will go along with rsyslog development.

simplifying rsyslog JSON generation

With RESTful APIs, like for example ElasticSearch, you need to generate JSON strings. Rsyslog will soon do this in a very easy to use way. ...