Monday, January 26, 2015

rsyslog daily builds and tarballs


rsyslog daily builds on Launchpad
The past days, I have worked on making rsyslog daily builds and tarballs a reality. I hope this will enable users to rapidly deploy the latest features as well as make it easier to help with testing the current development system. Daily builds are what the scheduled v8-devel builds were under the previous release paradigm. Consequently, the archives are named v8-devel.

Right now, builds are only supported for Ubuntu. Users of other platforms are advised to use the daily tarballs to build from source. Depending on feedback on and success of the daily builds, I will make them available for more platforms. 

A daily build is based on the latest git master version. So it really is at the [b]leading edge of technology. So why create them?

A top reason is that I often fix a bug for someone, and that someone then is unable to build from source. In the end result, we have a bugfix, but there is no external confirmation that it really fixed the bug when I merge it into the next release. I hope that now those users can simply pick the daily build and check if that solves their problem.

Also, in general I hope that some users will use the daily tarballs to get not only the latest and greatest but contribute to the project by doing some testing.

Finally, and quite important, with daily builds we will see build problems as early as possible. In the past, we often saw problems only after source release (or very close to it), which was obviously problematic. Now, this should no longer happen. For obvious reasons, the final release build is now more or less a copy of a daily build.

As a technical side-note, daily builds are identified by the git master branch head hash that was used to build them. As a forth version component, they have the first 12 digits of that hash (an example is "8.8.0.35e7f12a2c04"). This enables us to track error reports to the right version. The packages have a different version name, based on the build date. The reason is that the hash does not increment and so newer versions (with lower hash values) are considered as "old" by Launchpad. We avoid this by using an always incrementing package version. Also note that the package changelog just contains a "daily build" entry -- anything else makes limited sense.



I hope you enjoy this new feature! Feedback is appreciated.

Thursday, January 15, 2015

what's next with rsyslog?

Now that we have released version 8.7.0, planning for 8.8.0 is in full force. I thought I'd share some of those things that made it to the top of the todo list:

I already have begun on some experimental research work on a pull model for rsyslog. Scenarios where that would have been extremely helpful surfaced on the mailing list and support forums since long. While never asked for violently, I think it is the time to explore that option. The first pilot implementation will probably very simplistic, but has a big impact: if it works for simple syslog, it will work for other pull protocols as well. That would open up some wholly new use cases. But be careful: it's still unclear if and how fast we can realize such a method.

Secondly, we have receive a grant from GuardTime which enables us to improve the signature-related tooling. While this, too, is a bit of a large project, I will definitely begin to work on it in the 8.8 timeframe.

Finally, the ability to reparse messages is on the list. That's another biggie, and it may be one that requires a handful of release cycles. To make this happen in a clean way, we need to change some of the internal interfaces as well as some of the processing philosophy. It will also need some good discussions on the mailing list.

Note well that these three topics won't necessarily show up in 8.8, but at least they are something we strongly intend to work on - as said, I already started with the pull model.

Besides these three topics, there will be a number of minor improvements and bug fixes. I will also keep some focus on automated testing, but the most urgent need has been solved by the system I set up in Q4 2014. If all goes well, I'll also get some inhouse help on expanding the testbench, what would be a real great plus.

That's it for now, and as always: priorities may shift as needs arise ;)

Monday, January 12, 2015

rsyslog branches and git history

There was a lengthy mailing list discussion in November and December  of 2014 of whether or not to avoid git merge entries. There was also an intermingled discussion on QA and CI. The idea was to trim the git history and make sure tests are run a quickly as possible. As a result of that discussion, I added more automated testbench runs, which also required a new branch master-candidate, which is used as a staging area to run the test, and from which changes are (manually) migrated to master when all testbench runs are OK. In order to avoid merge entries in git log, I made master-candidate the default github branch and also asked contributors to file PRs against that branch.

I've now tried all of this for a couple of weeks. That approach works, but it creates a lot of overhead and quite some confusion for a lot of folks. Some users have voiced they don't really care if there is a merge entry. Fewer have voiced they don't like them. Michael Biebl has pointed out that it is easy to make them disappear from "git log", via the --no-merges command line switch.

After careful consideration and some frustration, I conclude that avoiding merge entries is unnecessary overhead for me. Being the 90%+ contributor for this project, I conclude that avoiding merge entries is unnecessary overhead for the project. As such, I will no longer try to avoid them at all costs. I will, however, try to keep the git history as neat as possible .. but not any more.

As such, I'll reset the default branch on github to "master" and will accept pull requests to master. Internally, everything still needs to go through master-candidate, as this is how the new testbench setup requires. If someone doesn't like this approach to the testbench, I am open to changes, BUT I than ask that someone to actually contribute running code to make that change happen. Good advise only is good, but doesn't help getting things done at this stage. We already know the advise, we just have nobody who has time to implement it!
I would like to thank all users for their comments. I think they have considerable helped move forward. Sorry that I could not accept all suggestions. I guess it's like always in life: not everybody can be fully happy. But I hope we have achieved a sufficient level of overall happiness :-)

Automating Coverity Scan with a complex TravisCI build matrix

This is how you can automate Coverity Scan using Travis CI - especially if you have a complex build matrix: create an additional matrix en...