Friday, January 25, 2008

rsyslog work log ... finally

Long time no rsyslog work log post. But this time its a rather long one:

- removed no longer needed "Initialized" logic and variable
- worker shutdown sequence enhanced to try different ways to shut
down and terminate workers if none helps (this protects against
badly written output plugins which hold the queue for too long)
- implemented $MainMsgQueueTimeoutActionCompletion config directive
- implemented $MainMsgQueueTimeoutEnqueue config directive
- implemented $MainMsgQueueTimeoutShutdown config directive
- some cleanup
- removed $MainMsgQueueImmediateShutdown config directive and handling,
this has been superseeded by the Timeout params. Backward compatibility
is no concern, no version with that directive was ever released.
- released 3.10.2
- fixed a bug with standard template definitions - thanks to
varmojfekoj for spotting it
- implemented config file handlers for
* $MainMsgQueueHighWaterMark
* $MainMsgQueueLowWaterMark
* $MainMsgQueueDiscardMark
* $MainMsgQueueDiscardSeverity
but did NOT yet implement the functionality behind these directives!
- implemented $MainMsgQueueDiscardMark and $MainMsgQueueDiscardSeverity
(but serverity needs to be specified numerically for the time being)
- also implemented $MainMsgQueueDiscardMark logic on dequeue, changed
- implemented $MainMsgQueueDiscardMark == 0 --> disable Discard logic
- implemented $MainMsgQueueSize == 0 --> no limit on queue size
- begin disk assisted queue memory queue modes (not fully implemented
- disk assisted queue works quite well, except for startup from disk queue
- changed startup of disk assisted mode to allow for higher concurrency,
most importantly allow the input to continue enqueue msgs while the
disk queue is initialized. This may help somewhat with UDP and other
lossy sources
- improved shutdown processing - in-memory queue is now drained to disk
- first shot at queue restore on startup, but could not finish before
I need to leave ;)
- fixed a compile-time bug in release mode spotted by Michael Biebl
- queue is now able to restore persisted state on startup (but still some
fine tuning to be done)
- re-released 3.10.2 due to bug reported by Michael Biebl
- cleaned up queue disk startup
- implemented dynamic startup and shutdown of worker threads based on
current activity
- some cleanup and fixes
- some more cleanup and flagged places where we need to implement
DA-input-only mode
- changed obj_t destructor interface
- worked on threading
- fixed sync issue on shutdown process if need to persist pure memory
queue to disk
- adapted DA logic to the ability to shut down all workers for inactivity
- created an in-depth description of DA assisted queue mode
- snapshot of new thread coding - DA mode still does not work, but need
to save
- seperated mutex for queue size management from those for queue thread
- some further cleanup on the mutexes
- begun object model redesign for clearer design of queue class, causes
creation of wti and wtp classes
- initial creation of wti class (worker implementation missing)
- continued implementing wti class
- added some mutex-support to srUtils
- initial creation of wti class (still under development)
- continued to work on queue
- implemented debug interface
- added mutex debug instrumentation
- improved debugging support (faster, less memory used, function invocation
count is maintened)
- redesigned queue to utilize helper classes for threading support. This is finally
in a running state for regular (non disk-assisted) queues, with a minor nit
at shutdown. So I can finally commit the work again to CVS...

No comments:

Introducing new team member

Good news: we have some new folks working on the rsyslog project. In a small mini-series of two blog postings I'd like to introduce the...