Tuesday, April 22, 2008

A good blog post on syslog reliability

I found a good blog post describing the problems that go along with reliable logging. It also offers most of the options to resolve them:


The author missed one thing: a buffer must not necessarily exist only in main memory. When the in-core buffer runs out of space, you may also use a disk-based buffer, which offers much more capacity. Of course, even the largest disk-based buffer may be exhausted at some point, where one needs to resort to other strategies. But a disk-based buffer is an excellent solution for temporary (but lengthy) receiver outages.

Rsyslog has implemented everything mentioned in the paper, plus more. You can directly apply the knowledge you got from this paper to rsyslog. And then, you can dig down into the dirty details:



Martin said...

short note in my defense: I just did not found it worth mentioning since disk-based buffer is just a way to further enlarge the memory buffer.

(I guess I read too much in the Varnish Architecture Notes ;-)

Rainer said...

Hi Martin,

there is no need to defend anything - I found your paper to be excellent and thus I mentioned it. I didn't grasp the idea of using large disk chunks as virtual memory from it, though.

Also thanks for the link to the Varnish Architecture Notes. I wasn't aware of this paper and it has some interesting thoughts I did not think about (though you'll find some other things in rsyslog). I am not convinced yet that the memory caching architecture actually matches what I am intending to do in rsyslog with the disk queues. After all, they are not just caches (they have other functions from the app point of view). Anyhow, it's pretty interesting and well worth re-thinking about some things.

Thanks and please keep up the good work.