Tuesday, September 11, 2007

bug hunting...

We have received feedback that the 1.19.x releases of rsyslog contain a bug that leads to a segfault after some time. Of course, this is a very bad thing to happen. After all, the primary goal for the rsyslog project is reliability and it should be able to survive even in the worst conditions (e.g. low system memory). So this bug is clearly unacceptable and has received highest priority.

Unfortunately, it is hiding very well. We are all looking into troubleshooting this beast. Thankfully, there is a lot of community support, especially with testing. What makes this bug so hard to find is, among others, the inability to reproduce it in lab. So the project is totally dependent on user feedback. As current reports have shown, we also currently assume that releases prior to 1.19.0 can also be affected - so it is not as easy as checking the changes.

I receive very valuable help by varmojfekoj. He is already a frequent and great contributor of patches. His skills and experience are extremely valuable and I am very glad to have him work on this project. Please give him a big hand. Thanks, varmojfekoj, the project would not be that far without your help!

While the bug itself is obviously a very bad thing, it has some good side-effects. Most importantly, the code is getting another round of very in-depth review. That review has already brought up some fixes for obscure situations. Situations that I'd expect to happen extremely infrequently (if at all) in reality. But now even those are fixed. And I am sure that the review will bring even more benefit as it continues. Of course, I'd be glad if we find the bug ASAP, but it is good to know that all that work also provides additional benefit (at least that keeps my depression level low ;)).

The bad thing, besides the bug itself, is that the bug hunt obviously defers other work. Most importantly, I refrain from making any changes that may not be related to clean up or bug fixing. I think it is not smart to introduce new code (read: complexity) at this stage of the project. And, of course, all resources should focus on fixing that bug, so there wouldn't even be time to enhance other parts of the code. So for all of you waiting for new features: please bear with us, the implementation schedule will slip a little bit.

If you experience stability problems, please report. Each reports helps us understand the problem somewhat better.

No comments:

Introducing new team member

Good news: we have some new folks working on the rsyslog project. In a small mini-series of two blog postings I'd like to introduce the...