Tuesday, June 07, 2011

full blown dns cache for rsyslog

Up until and including version 5 rsyslog does actually not implement a real DNS cache. Instead, it uses some sort-of caching methods. They seem to work surprisingly well, as almost no real pain was reported by users in regard to this system. The big exception is UDP traffic, if combined with template options that require host resolution and a larger number of different hosts sending messages.

Starting with 6.3.1, I have now implemented a real, full-blown cache system which will resolve the issues with that use case. The initial implementation is not perfect, but I thought it would be best to gain some feedback from the community first before deciding on the final implementation. Most importantly, it currently does not expire entries (this was considered not necessary in many previous discussions we had on the mailing list). Also the current linear list data structure and locking method used is not optimal. However, it is very simple and easy to maintain. So if there is no need for more advanced (aka "complex") code, it probably is not bad to stay simple.

I hope to get some feedback from the community, and most importantly feedback from folks who actually use the new capability to their benefit. In those cases where it matters, the speedup can be "immense".


SEJeff said...

Rainer, why would anyone ever do this vs using an actual dns cache (that works for other things too) like dnsmasq?

I'm unable to see a situation where you need dns caching for syslog and not for the entire system.

Rainer said...

It's probably only of interest for very high performance sites. With the internal cache, you save the overhead associated with doing context switches for switching into kernel space and into the other app. I will also probably use this facility in the not so distant future to enable overwriting DNS resolution.

But, I agree, for the average user there should not be that much difference (this is why I didn't add it sooner).

simplifying rsyslog JSON generation

With RESTful APIs, like for example ElasticSearch, you need to generate JSON strings. Rsyslog will soon do this in a very easy to use way. ...