Tuesday, November 22, 2016

Would creating a simple Linux log file shipper make sense?

I currently think about creating a very basic shipper for log files, but wonder if it really makes sense. I am especially concerned if good tools already exists. Being lazy, I thought I ask for some wisdom from those in the know before investing more time to search solutions and weigh their quality.

I've more than once read that logstash is far too heavy for a simple shipper, and I've also heard that rsyslog is also sometimes a bit heavy (albeit much lighter) for the purpose. I think with reasonable effort we could create a tool that

  • monitors text files (much like imfile does) and pulls new entries from them
  • does NOT further process or transform these logs
  • sends the resulting file to a very limited number of destionations (for starters, I'd say syslog protocol only)
  • with the focus on being very lightweight, intentionnally not implementing anything complex.
Would this be useful for you? What would be the minimal feature set you need in order to make it useful? Does something like this already exist? Is it really needed or is a stripped-down rsyslog config sufficient?

I'd be grateful for any thoughts in this direction.


Unknown said...

I have not found a good tool yet (I've written or seen written a couple over the years)

The problem tends to be that there is not really such a thing as 'no additional processing needed', any one use requires a tiny fraction of the capabilities that rsyslog offers, but each setup requires a different combination of capabilities.

just look at all the capabilities that imfile has been growing over the last few years.

I suspect that a stripped down compile of rsyslog (no input modules other than imfile, especially no imjournal, etc) would end up being competitive to just about any special-purpose program.

IMHO, The biggest problem with using rsyslog to do this is the same problem we have with using rsyslog to create /dev/log in containers, the fact that the config is fixed at startup time.

Having a command socket that rsyslog listened to that would let you add/remove inputs (files or unix sockets), but didn't allow you to change anything else in the config would let you easily tell rsyslog to start watching a new container or file as needed, and then stop watching so that it doesn't prevent the container or directory from going away when the app/container is removed.

Matthias Runge said...

I'd like to see something like fluentd: install it, it reads from a to be configured list of files and ships that data to a remote destination. Bonus points for: not requiring any current service to be reconfigured and even more bonus for handling multiple destinations (for fail-over scenarios)

Troy said...

papertrail/remote_syslog2 meets most, maybe all, of these requirements: https://github.com/papertrail/remote_syslog2

Specifically, remote_syslog2:
- monitors only files or paths/globs and sends to syslog
- doesn't require root access or kernelspace changes
- is completely self-contained. It's written in Go, so "installation" is copying a binary and an optional config file.
- runs just as well with command-line args as a config file, so it can be used in a totally stateless environment or r/o filesystem
- doesn't modify the messages, other than optionally dropping messages which match regex(es)

It doesn't send to multiple destinations, though of all the feature requests we've received over 3+ years, I don't think that's ever come up. Many folks are hard-pressed to have one destination for logs, let alone 2+ =)

simplifying rsyslog JSON generation

With RESTful APIs, like for example ElasticSearch, you need to generate JSON strings. Rsyslog will soon do this in a very easy to use way. ...