Wednesday, October 17, 2012

new "stable" policy for rsyslog?

For many years, I have strongly opted for the devel-beta-release cycle for rsyslog, where a transition from beta to release only happens after two to three month in order to be sure that a stable version really was stable. This worked well, because early adopters of the beta branches helped to iron out any remaining bugs. Unfortunately, I see this the early adoption rate decline for around two years now (actually starting with v5). The reason is probably obvious: rsyslog has become so feature-rich that users usually do not see any need to work with the beta versions but rather wait for stable. The bottom line is that when everybody waits for stable, the beta won't stabilize at all ;-).

Unfortunately, I have seen this, too, happen in practice: even though we had very lengthy beta periods, problem reports just came in after the beta has been declared to be stable. At that time, many folks seem to start using the system, and as nobody did any testing before, things then begin to break. I guess there is noone to blame, that's just how things work. However, one needs to know that I invest quite some time into the devel-beta-stable cycle. Time, that is most probably better spent at other things, if looked at in the light of what I just described.

As such, I will most probably change our "stable" policy for rsyslog soon: the tag "stable" will no longer mean that the product has actually stabilized, but that we are ready to support it as a stable version (probably very important to those folks with support contracts). While this may sound harsh, it has a lot of good: first of all, this is what actually happens for 2+ years already, just with a 3-month delay. So we spare these months, what means new features will become more readily available, and bugs are probably quicker found AND fixed (because if a bug reports is closer to implementation, it is usually easier to figure out what is wrong). With that said, rsyslog 7.2 will probably the last stable that went through the usual cycle.

If you have a great idea how we can actually get more beta deployments, I'll reconsider that decision.

Note that we will still have devel-beta-stable branches, but beta will much quicker turn into stable than before.

6 comments:

Radu Gheorghe said...

I'd change the "beta" policy instead of the "stable" policy. That is, if there's no bug reported in the beta for 1 or 2 weeks, assume there's no bug and push it to stable. You have a Bugzilla to prove you worked by the policy :)

But that would only solve the beta -> stable relationship, which doesn't seem to be the root of the problem. That would be (in my view, of course), the develop -> beta relationship. How do you know rsyslog is ready to be beta?

I think the problem here is that "beta" is too unstable to inspire "early adopters". What might help here is to have some sort of policy here as well (maybe you already have one that I'm not aware of). For example, when you release a new feature, have some automated tests for every functionality.

I understand that this would mean a lot more work for you, and it just won't work if you'd have to do all that as well. But one can imagine a win-win situation:

If I'd use rsyslog for any serious thing, I'd have some automated tests for the functionality I'm using. Otherwise, I might run into bugs at deployment time. Even if it's a test deployment, it's still inconvenient.

Now if I have (or intend to develop) such automated tests, I could open-source them and send them to you.

You might some with some sort of "reward", like some sort of restricted support contract, limited edition, for contributors who send in a decent test coverage of a decent quality. I can imagine that people who test rsyslog that seriously wouldn't come with support inquiries that much anyway - but it's a good for the peace of mind. One could assign such "testing" tasks via Bugzilla or some other tool.

But I think this sort of approach - to somehow encourage the community to submit tests - would make everybody happier: rsyslog will get more stable, you'd have more confidence in adding new features, major distributions will adopt new versions sooner, etc. which will widen the community, which might submit more tests, etc.

Last but not least, I'd like to emphasize on the fact that I like rsyslog a lot and I admire your work.

Rainer Gerhards said...

Thanks for the thought-out response. There actually exists a devel-beta policy: for each version, I have a feature goal (usually just one, maybe more if they are not too big). Once this is reached, we turn to beta. Turning to beta is quite imporant in the development cycle, as it means NO new features will be added to that version and it just can mature. New features will go into the next devel cycle, only.

There also currently is an automatted testbench, but it is far from covering everything. As you write, it's taken quite some time. Also, testing different config options grows exponentionally, so there need to be some restrictions even if there would be time to write all these tests (I once thought about a generator for some common cases, but it quickly turned out that more than 2^20 tests - at that time - would need to be conducted...).

The reward idea sounds good, but it means I need to have another round of funding talks with my peers, not sure if that is worth it right now...

In any case, I'd be very interested in any tests you craft.

Radu Gheorghe said...

Yes, I had plans to write quite a few tests but unfortunately I don't work in that project with rsyslog anymore. But if the next project will be with rsyslog, I'll do whatever I can to open-source them.

Another idea that popped up would be to apply some of the suggested login in custom development contracts. That is, in the contract usually the customer is responsible for testing. You can either offer a discount if the customer shares the tests (and the tests respect some parameters, of course, such as coverage - or you might supply the test case description). Or, might offer some sort of additional service instead of a discount. For example:
- ensure that the plugin will be supported in the next X versions of rsyslog
- some variant of a support contract for rsyslog core

I hope I didn't come up with too many suggestions, but my point was to centralize the effort that is already done (to a certain extent) by the community and encourage in what already are good practices. Unfortunately, there's no getting away from the fact that you (or someone directly interested) need to put in some effort to make that happen.

giuseppe gianquitto said...

First of all, thanks for rsyslog.
Thanks for making our sysadmin life easier, thanks for giving us a tool that rocks.
I would create something like a "rsyslog official beta tester", and I would aim to companies and/or freelances, we all sysadmins work and use rsyslog.
I imagine a "gentleman" agreement where the IT of any given company/freelance agrees to deploy the rsyslog beta version, test it on test environments, and give you feedback... in return you reward that company/person as an "rsyslog official beta tester".
Well something in that line...

Unknown said...

I'm seeing a lot of places where the old 'develop', 'stabilize', 'release' just doesn't work any longer, so it's not really a surprise that you are having problems with it as well.

While it seems somewhat like just a renaming, doing -rc, release, bugfixes does seem to work a bit better.

Given that you are working from a public git tree, and you usually aren't doing _that_ much disruptive new stuff in each developement cycle, you may not need to do any -rc releases unless you have an especially invasive development cycle (like 7.x was)

I think this flattening of versions will also help the distro problem. They seem be far more comfortable moving from 7.1 to 7.25 than they are moving from 7.9 to 8.0, even though the former is a much larger set of changes

Rainer Gerhards said...

Thanks for all comments. I think there is much value and I especially like the idea of "official beta testers" and re-naming things. I'll see that I can integrate all suggestions within the next release cycles!

I would specifically like to mention the flattening of versions. I see the point, but the datacenter guy in my is still a bit hesitant. Among others, this means that we will probably either need to support more old versions (no-go workload) or acutally force users to upgrade faster.

Busy at the moment...

Some might have noticed that I am not as active as usual on the rsyslog project . As this seems to turn out to keep at least for the upcomi...