Monday, January 16, 2012

imuxsock rate limiting now opt-in feature

I added rate-limiting capabilities directly into rsyslog's imuxsock some time ago. These were turned on by default and blocked processes from writing more than 200 messages within a 5-second interval (accounted on a per-pid level). This default, however, caused quite some confusion. I monitored reactions since the feature was introduced. Last week, I noticed Chris Gaffney, too, telling about his bad experience on twitter and suggesting to change that feature from opt-out to opt-in. That was the missing complaint ;) So I did exactly what Chris suggested, and starting with 5.9.7 that feature requires an explicit opt-in. To do so, I have simply changed the default. Now, you need to tell it the rate limiting interval, which must be non-zero, in order to activate rate limiting. The default is zero, what keeps it deactivated. Currently available via git. It will take some time, though, for this changed default to migrate to the stable branch.

Friday, January 13, 2012

Disappointing press reactions...

Friday, 13th... Systemd v38 went out recently and it includes a journald test release. That's great and congrats to the systemd team to that release. What me saddens is the that the press still conveys only Lennart's position on how bad syslog is. The counter arguments ( are mentioned nowhere. Lesson learned? All it takes is some interesting-sounding claims and everyone believes them. Does anybody still wonder why I think journald will become standard [and would do so even if it were totally evil]? ;-)

But don't get me wrong: while I think the journald proposal and the way it is getting pushed has some flaws, I also see good in it. My initial reaction still stands: Unfortunately, the world, and us, the masses, seems not to be prepared for weighted arguments... On to the weekend, have a great one :-)

rsyslog will remain GPLv3 licensed

Licensing is though topic. I tried to explain some of the upcoming rsyslog license changes with yesterday's blog post. While I tried to cover all aspects, I have probably manged to create some confusion. I try to cleanup this mess today. In doing so, I will leave out some of the fine details but focus on the prime visible facts. 

First of all, the rsyslog project as whole will stay under GPLv3. If you dig down into the details, the current version is licensed under GPLv3, with a large body of code (the rsyslog runtime) being licensed under LGPLv3. In straight words, this means that rsyslog as whole can not be included in a commercial product while the rsyslog runtime can be. So if someone intends to provide rsyslog-like functionality, he or she can use the runtime and rewrite the rest of the system (but not copy it). Also, it is possible to create commercial rsyslog plugins with the current system, but then one relies either on a "creative" interpretation of the GPL or use message passing e.g. via pipes between core and plugin.

With the intended changes these basic facts, in regard to rsyslog as whole, are not changed at all. However, the details change: the body of code that is licensed under a permissive license (allowing use inside commercial applications) will be increasing. Still, some key files will remain under GPLv3, and so will the overall rsyslog project. Also, in the future the permissive license used by rsyslog will probably be the Apache software license (ASL 2.0). This open source license is used by a myriad of well-known software products, with the Apache http server being a prime example. It is not sure if ASL 2.0 will totally replace LGPLv3 inside the rsyslog runtime, this depends on contributor reactions. For many of the same reasons, it is also not yet clear what exactly the GPLv3 core of rsyslog will be.

Why is this change benefitial to the project? Maintaining and developing rsyslog is costly. In the typical open source business model, these costs are covered by the sales of project-related services. Unfortunately, the rsyslog project received relatively little funding via this way and is still heavily sponsored by Adiscon, which can not bear the majority of the cost for all time to come.

One problem with receiving funding is that some potential customers - especially large ones who could considerably contribute to funding - do not like to license under GPLv3 for one reason or the other. One "solution" to that problem would have been to dual-license rsyslog in its current form. We actually considered that (blog posting) but stepped back from the initial approach after discussion with key community members. As described in the mentioned blog posting, it would not have been very hard for Adiscon as the main copyright holder to change the licensing model. However, this would probably have meant that a commercial and a non-commercial fork of rsyslog would have been created with potentially large differences in the code base.

The move of more code under a permissive license prevents this problem. With the new model, only relatively few key files would need to be dual-licensed. This prevents large diversity inside the code base just for licensing reasons. Also, the ASL is far more appealing to many large users, so we hope to gain additional deployments - and thus potential customers - from that move.

Finally, this model facilitates the ability to provide commercial plugins. Commercial plugins were always OK with the project, and as said above, can even be written and distributed under the current licensing scheme. The new licensing scheme makes it easier to support such plugins and encourages technical superior solutions. How exactly the licensing in this regard will be is not yet fully thought out. One solution might be to add a special exemption to the then-smaller GPLv3 core, that explicitely permits plugins (getting the wording right may be somewhat tricky). Another one is that someone who intends to ship commercial plugins must rewrite the rsyslog GPLv3 core, which is no longer that hard even for external entities as the GPLv3 core is smaller (one may argue that Adiscon as the main contributor has an advantage here over others; I can't decline it but I don't find it unfair either - after all, Adiscon has spent considerable effort on rsyslog, so why not reap a small benefit in this situation?). These solutions are the extreme ends of the solution spectrum - it could probably also be anything in between.

Why is this useful to the community? Obviously, the changes help fund the project, and thus help to keep it not only maintained but well-enhanced as well (there are many cool things I have on my mind, but the current time constraints do not permit me to work on them). This is even more important as the journald project will create a new, mostly commercial, environment for rsyslog. In the future, rsyslog needs to compete with syslog-ng primarily in that part of the Linux ecosystem. Balabit, which traditionally dual-licenses syslog-ng under a proprietary license has a big advantage regarding this customer base from that fact alone. So far, rsyslog could make up its disadvantage by the fact that it was installed on each system, an advantage that journald will very probably remove from rsyslog. The other benefit is that the more permissive licensing model will probably attract additional software vendors and maybe additional parties, especially if we can move the full rsyslog runtime to ASL. There seems to be a movement inside the industry towards this type of licenses, at least for projects playing in the enterprise environment. If all works out well, we may even get some more contributors and thus the ability to include additional features into the project.

In short, the licensing change will not affect that much of what actually can be done with rsyslog code, but it provides rsyslog with some additional options that benefit both the project and the community.

I hope I have expressed myself clearly enough this time. Again, it has become a long post, even though I omitted some detail information given in yesterday's post. Licensing obviously is tough ;) I hope to soon be able to use my time for more productive things, again...

Thursday, January 12, 2012

rsyslog licensing update

As regular readers know, there has been discussion on rsyslog's licensing model for quite some time, I guess much like the beginning of the project. Many things influence its licensing, community support and the ability to fund the project are among these. Over the years, we had several license changes, so things tend to evolve. The whole discussion was restarted in 2011 after the introduction of the systemd journal idea in November. A prime reason is that journald pushes rsyslog far more into the commercial-only space than we originally anticipated, as journald will probably push rsyslog out of many low-end systems (for details, see my original argument on rsyslog vs. journald, which I still support).

This lead to a lot of discussion with my peers at Adiscon. This resulted in a blog post on potential dual-licensing of rsyslog (to get the fine details, read that post, I don't repeat them here; the post is still valid in regard to arguments and problems). That post, not unexpectedly, lead to a larger discussion on the rsyslog mailing list as well as quite some private conversations. In these discussions, some folks expressed they like an open core strategy more than dual licensing, while others expressed it right the other way around. Thankfully, some understanding for the need to fund the project was expressed as well ;-). Of course, no solution was reached and nothing was changed.

I have now taken the time to digest the discussion, Adiscon needs, and some overall trends. And I have briefly approached my peers at Adiscon as well as some contributors with a plan to go ahead in the most unintrusive way. So far, the idea was not killed (but also not discussed in-depth), so we may proceed with it. The core idea is that we must receive some funding for the work done. This is especially important in regard to the new environment that journald will create in the medium to long term. One thing that we learned during the past years is that commercial vendors are often hesitant to put GPLv3 code into their systems. Even larger key users slightly associated with software are hesitant (almost everyone falls into this category nowadays, think of embedded systems) for this reason. Ultimately, this somewhat blocks rsyslog's use inside commercial entities. And by doing so, it also limits our funding opportunities because it is hard to sell support and services to someone who does not use the project.

To avoid this obstacle, the idea is to put a larger body of rsyslog under a more permissive license. Looking at current trends, the Apache Software License (ASL 2.0 to be precise) seems to be a good fit. Actually, one thing that Adiscon already decided is to put code written by Adiscon under this license. I have already converted some files after careful examination of contributors, original sources and development history (not a job I really like to spend time on, but...). We also hope to gain support by contributors to place all plugins under the ASL. In order to clean up licensing, it would also be useful (though somewhat optional) to put the rsyslog runtime, currently licensed under LGPLv3, under the ASL as well. We need to see how this works out with contributors, though (this is obviously the largest bunch of code). It is important to note that while LGPL is not "GPL-free" and may still cause some friction with our future primary user base, it is pretty similar to ASL in regard to what it permits. Having all runtime code under the ASL would probably help a bit with those "GPL-skeptic" enterprise customers.

Note that the ASL permits unlimited commercial use to anyone. I would still like to keep some leechers off the code (my original motivation to use GPLv3, among many others; my 2007 blog post on that is probably an interesting read in addition to this posting here - some things seem not to have worked out ;-)). My idea is to keep some files under GPLv3. This is not broadly discussed yet and details need to be worked out. Obviously, that would be counterproductive to what we intend to do with ASL 2.0. One could, however, optionally release these parts under a different license and thus somewhat "solve" the situation via relatively little modifications and different code. I agree, it boils down to some dual licensing, but in a smaller magnitude. Most importantly, everyone would be free to do his own implementation of the GPL parts and use the rest of rsyslog (the majority) with that freshly written code. [Please note that even though I changed some files to ASL, others are still under GPLv3, simply because I can not change licenses where Adiscon is not the sole copyright holder. So this is no indication that my idea is currently being implemented.]

In any case, if we mange to complete the re-licensing -with or without GPLv3 parts- everyone would be free to extend rsyslog with extra functionality. This obviously includes Adiscon as well. It would give us some extra funding opportunities, as we could create some plugins targeted at large commercial entities. One thing that immediately pops up my mind is a new queue subsystem (the queue driver is internally "plugin-ish"), which could integrate many of the improvements I have on my mind. While I would really love to write that beast, it is a considerable effort that Adiscon declines to fund and no other party has yet been willing to fund it either -- but some expressed interest to provide limited financial support for it. If Adiscon could provide such a piece under a commercial license (at least for a while), I could probably persuade my peers to try this out and fund the effort. I honestly think this situation would be a win for everyone. With the current policies, it looks like this new subsystem will probably never materialize, so what is to lose? Highly specific input and output plugins also create similiar opportunity.

The ability to use commercial and non-commercial plugins is something that I always considered a valid use of rsyslog technology (see "writing rsyslog plugins" right at the bottom, under "licensing"). I just never cared to dig into the licensing implications, which make it somewhat hard to provide plugins under a GPL-incompatible license (contrary to what I originally intended). Note that in any case it would even be possible with a totally GPLv3 codebase (even if the rsyslog runtime would not be LGPL, what is is licensed under). The only difference is that with the current system, it would either require a creative interpretation of the GPL (something many open source software vendors have done before) or a somewhat technically inferior approach that avoids direct linking (using pipes, which is really not that bad, so it is a vital option and would probably be the one I'd choose). Again, with the intended re-licensing, we could remove some friction associated with "GPL-phobic" companies, thus increasing our funding abilities.

All in all, I think the proposal to re-license large parts of rsyslog under the Apache license is of benefit both the project as well as the community. It hopefully helps to provide the necessary funding for development and at the same time may even grow the community involvement as it becomes more attractive for other software vendors to participate in the rsyslog development (I found this article by Brian Proffitt a very interesting read in that regard; if you read it, be sure to follow its links for details). I would really appreciate if we could find support for this movement and, yes, I have to admit I consider it rather vital, especially with the new role rsyslog will play in the systemd dominated world. Let me conclude this posting with one more personal opinion: I think that any feature developed thanks to commercial licensing should immediately be available, for free, to non-commercial entities, like private parties, educational institutions and charities. For the same reasons, it should probably reside inside the public code repository and be free to explore by anyone.

PS: back in 2009, I wrote about my philosophy about the "free" in free and open source software. It may be worth a read in addition to this posting, as it probably explains some of the mindset behind my ideas.

UPDATE: This blog posting created some confusion if the rsyslog project will stay under GPLv3. In fact, it will! I wrote on new blog post clarifying the situation, please check it out.

Monday, January 09, 2012

rsyslog v6 stable released

Happy new year! After being back from vacation, I started the new year with finally releasing the first stable rsyslog v6. While it was ready for a while, I was hesitant to release it when there was so much going on and time left for any quirks that may show up (I know I all too often overlook a thing or two with such a release and, if so, it is good to react fast). So today was finally the day. Let's see and wait if I failed somewhere.

The new release is a very important one for me. First of all, because it helps me to get down to a decent set of versions I need to support. Doing all that v4..6 stuff in parallel with multiple branches really became time-consuming (but manageable thanks to git!). V4 has now been retired, and the number of branches is reduced to a minimum.
Very important is the new log classification capability. While it is in the beta (and devel) for quite a while, I think people were hesitant to try it out. At the same time, this prevented us from developing it any further. With it now being part of a stable release, I hope that it will get momentum.

Besides software availability, rule bases for common syslog messages are also important. Adiscon will assist with creating these rule bases upon request. I hope that people will use that facility. All they need to do is provide sufficient information about what they want to normalize and some sample messages for us to work with. We will than see how to create the required rulebase. This whole effort is based on liblognorm, so the technology can also be used by other projects (sagan is a good example that actually currently has more interest in that feature that rsyslog has).