Thursday, January 31, 2008

rsyslog work log

Yesterday's rsyslog work log:
- renamed Msg object to usual all-lowercase object name (else we ran into
troubles with the framework, also it was somewhat ugly...)
- fixed a memory leak in object destruction (was recently introduced by
object naming, not present in any released version)
- finally implemented infinite action retries via -1 retry count
- fixed a bug that caused $MainMsgQueueCheckpointInterval to work incorrectly
- fixed bug in output module interface, see
- changed the ommysql output plugin so that the (lengthy) connection
initialization now takes place in message processing. This works much
better with the new queued action mode (fast startup)
- fixed a newly introduced bug that caused output module's doAction entry
point to be called on more than one thread under some circumstances
- fixed a bug that could cause invalid string handling via strerror_r
varmojfekoj provided the patch - many thanks!
- adopted varmojfekoj's patch to v3, some more sterror_r's ;)
- bugfix: MsgDup() did not work with new base object data structure
- implemented simple output rate limiting
- addded $ActionQueueDequeueSlowdown config directive
- addded $MainMsgQueueDequeueSlowdown config directive
- implemented limiting disk space allocated to queues
- addded $MainMsgQueueMaxDiskSpace config directive
- addded $ActionQueueMaxDiskSpace config directive

Wednesday, January 30, 2008

Yesterday's rsyslog work log

Yesterday's rsyslog work log:
- improved debug support a bit (assertions)
- restructured code, moved some part out of syslogd.c to action.c, where
they belong (still some more to do in that regard ;))
- moved correct retry logic into action processing queue
- removed debugging support from sync class, debug class now provides
much more
- implemented naming for all objects (mostly as a debug aid, but you never
know what else it will be good for)
- converted queue.c to use dbgoprint() instead of dbgprintf()
- fine tuning on queue naming
- action queues are now also named (otherwise you can't read the debug log ;))
- bugfix: added forgotten docs to package
- bugfixing newly added action code
- action queue params are reset to default for each action
- added ability to re-enqueue objects into the queue when a worker thread is

Tuesday, January 29, 2008

rsyslog work log

Yesterday's rsyslog work log:
- added capability to record last known exec location in debug stack dump
- added some doc on how to handle massive amounts of database inserts
with rsyslog
- some more testing and cleanup with the queue class (pretty stable now)
- cleanup to prepare for release
- PID file is now also written in debug mode
- fixed a segfault issue with the rsyslog shutdown exit debug message when
in release build
- added "runtime instrumentation" feature to ./configure
- some more cleanup
- add Michael Biebl's autoconf patch for netsnmp - thks!
- released 3.10.3 - finally ;)
- implemented the $ActionResumeRetryCount config directive
- added queue between main queue and action executor (currently
works in "direct" mode only, else crashes)
- added $ActionQueueFilename config directive
- added $ActionQueueSize config directive
- added $ActionQueueHighWaterMark config directive
- added $ActionQueueLowWaterMark config directive
- added $ActionQueueDiscardMark config directive
- added $ActionQueueDiscardSeverity config directive
- added $ActionQueueCheckpointInterval config directive
- added $ActionQueueType config directive
- added $ActionQueueWorkerThreads config directive
- added $ActionQueueTimeoutshutdown config directive
- added $ActionQueueTimeoutActionCompletion config directive
- added $ActionQueueTimeoutenQueue config directive
- added $ActionQueueTimeoutworkerThreadShutdown config directive
- added $ActionQueueWorkerThreadMinimumMessages config directive
- added $ActionQueueMaxFileSize config directive
- added $ActionQueueSaveonShutdown config directive

Monday, January 28, 2008

weekend rsyslog work log

Past two day's rsyslog work log:
- improved SIGSEGV handler a bit (now we get a core dump)
- partially fixed bug that caused rsyslogd to stall processing enqueued messages
after turning off DA mode and before any new message were arrived (if a new
message arrived, everything went back to normal, so it was a temporary halt)
- fixed the situation where message processing could be stalled for some period
after DA mode turn off
- reduced number of unnecessary wakeups of DA worker thread when high water mark
is not yet reached
- fixed a cosmetic issue in timeoutValue calculation (just used for displaying)
- fixed a bug when shutting down DA queue
- some more fixing and cleanup on the queue shutdown sequence
- fixed queue termination in case bSaveOnShutdown is 0
- implemented the $MainMsgQueueSaveOnShutdown config directive
- implemented the $MainMsgQueueWorkerThreadMinimumMessages config directive
- implemented the $MainMsgQueueTimeoutWorkerThreadShutdown config directive

Saturday, January 26, 2008

rsyslog work log

Yesterday's rsyslog work log:
- restructured queue shutdown so that the queue timeout is properly applied before
terminatiing the queue
- disk-assisted queue mode finally begins to look good ;)
- added debug-support environment variable RSYSLOG_DEBUGLOG
- fixed worker timeout which was accidently not set (and thus 0)

Friday, January 25, 2008

rsyslog work log ... finally

Long time no rsyslog work log post. But this time its a rather long one:

- removed no longer needed "Initialized" logic and variable
- worker shutdown sequence enhanced to try different ways to shut
down and terminate workers if none helps (this protects against
badly written output plugins which hold the queue for too long)
- implemented $MainMsgQueueTimeoutActionCompletion config directive
- implemented $MainMsgQueueTimeoutEnqueue config directive
- implemented $MainMsgQueueTimeoutShutdown config directive
- some cleanup
- removed $MainMsgQueueImmediateShutdown config directive and handling,
this has been superseeded by the Timeout params. Backward compatibility
is no concern, no version with that directive was ever released.
- released 3.10.2
- fixed a bug with standard template definitions - thanks to
varmojfekoj for spotting it
- implemented config file handlers for
* $MainMsgQueueHighWaterMark
* $MainMsgQueueLowWaterMark
* $MainMsgQueueDiscardMark
* $MainMsgQueueDiscardSeverity
but did NOT yet implement the functionality behind these directives!
- implemented $MainMsgQueueDiscardMark and $MainMsgQueueDiscardSeverity
(but serverity needs to be specified numerically for the time being)
- also implemented $MainMsgQueueDiscardMark logic on dequeue, changed
- implemented $MainMsgQueueDiscardMark == 0 --> disable Discard logic
- implemented $MainMsgQueueSize == 0 --> no limit on queue size
- begin disk assisted queue memory queue modes (not fully implemented
- disk assisted queue works quite well, except for startup from disk queue
- changed startup of disk assisted mode to allow for higher concurrency,
most importantly allow the input to continue enqueue msgs while the
disk queue is initialized. This may help somewhat with UDP and other
lossy sources
- improved shutdown processing - in-memory queue is now drained to disk
- first shot at queue restore on startup, but could not finish before
I need to leave ;)
- fixed a compile-time bug in release mode spotted by Michael Biebl
- queue is now able to restore persisted state on startup (but still some
fine tuning to be done)
- re-released 3.10.2 due to bug reported by Michael Biebl
- cleaned up queue disk startup
- implemented dynamic startup and shutdown of worker threads based on
current activity
- some cleanup and fixes
- some more cleanup and flagged places where we need to implement
DA-input-only mode
- changed obj_t destructor interface
- worked on threading
- fixed sync issue on shutdown process if need to persist pure memory
queue to disk
- adapted DA logic to the ability to shut down all workers for inactivity
- created an in-depth description of DA assisted queue mode
- snapshot of new thread coding - DA mode still does not work, but need
to save
- seperated mutex for queue size management from those for queue thread
- some further cleanup on the mutexes
- begun object model redesign for clearer design of queue class, causes
creation of wti and wtp classes
- initial creation of wti class (worker implementation missing)
- continued implementing wti class
- added some mutex-support to srUtils
- initial creation of wti class (still under development)
- continued to work on queue
- implemented debug interface
- added mutex debug instrumentation
- improved debugging support (faster, less memory used, function invocation
count is maintened)
- redesigned queue to utilize helper classes for threading support. This is finally
in a running state for regular (non disk-assisted) queues, with a minor nit
at shutdown. So I can finally commit the work again to CVS...

Monday, January 21, 2008

rsyslog status

I've not posted anything the past days - not because I don't do anything but because I am terrible busy with new features. The new massively multithreading engine is giving me a somewhat harder time than I expected (and I have to admit a design flaw which I am recovering from). Everything else is currently being pushed away by my desire to finish store-and-forward enabled actions by the end of the month. I am no longer sure I'll meet that deadline, but at least I'll try.

Updates to the blog will hopefully resume soon, as will new releases. I'll post work logs soon.

So you now know at least what's going on... BTW: I am looking for some test environments where store-and-forward and/or high performance (running on as many cores as possible) is in high demand.

Monday, January 14, 2008

rsyslog work log

The past day's rsyslog work log:
- undid part of yesterdays stage work - q worker 0 does not have management
chores, will use another solution (not needed yet)
- begun to permit queue to terminate without being drained
- fixed a starvation condition in queueWorker (pthread_yield() was needed)
could not be seen with any previously released code, came up during
new development
- added $MainMsgQueueImmediateShutdown config directive
- some name cleanup
- added non-circular file stream mode
- added some debug instrumentation to obj_t type, so that invalidly passed
objects can be detected (else we use the jump table and do not know why
everything messes up)
- file stream objects are now persistent on immediate queue shutdown (queue itself
is not yet fully persisted)
- support for object property bags added
- queue can now persist disk queue information on immediate shutdown
- added function to de-serialize a property bag (untested as other code is yet
- support for de-serializing strm objects added
- partial ability to read a disk queue back in (not completed, but would like
to save source for the weekend)
- support for reading back persisted queue information completed
- added $MainMsgQueuePersistUpdateCount config file directive
- renamed $MainMsgQueuePersistUpdateCount config file directive to
- changed queue shutdown procedure a bit - stage work for queue shutdown
timeout setting

Friday, January 11, 2008

rsyslog work log update

rsyslog work log for Jan, 9th and 10th (sorry, forgot to post yesterday):

- implemented new GetSize() handler for config files
- implemented $MainMsgQueueMaxFileSize configuration directive
- implemented queue object method to set the file name prefix
- implemented $MainMsgQueueFilePrefix configuration directive
- created a generic stream class (for file access)
- changed queue class to use stream class
- some cleanup on object model
- changed queue file name generation (to be more generic)
- made queue file names better readable
- added buffered output to stream class
- data record support added to stream output writer
- added write functions for several types to stream class
- changed objSerialize methods to work directly on the stream class
- prepared for 3.10.1 release
- changed some config parameters and some cleanup
- released 3.10.1
- fixed a bug that caused a segfault on startup when no $WorkDir directive
was specified in rsyslog.conf
- fixed a bug that caused a segfault on queues with types other than "disk"
- removed the no longer needed thread TermSyncTool
- re-released 3.10.1
- implemented strm object serializer (untested as the code required for test
is not yet present - hen/egg problem...)
- some cleanup
- implemented management function for worker thread 0 in order to change
queue workers dynamically -- stage work

Wednesday, January 09, 2008

Vodafone Customer Service Misery...

I wanted to share my experience with Vodafone Germany's customer service. And, yes, I have to admit I am a bit upset...

The story begun just after Christmas, roughly two weeks ago. I ordered a phone online and the shop said it would take around two business days. With the holiday period, I wasn't much surprised that nothing happened in 2007, but I had expected a delivery early this year. Well... it took some time, but last Friday a delivery man showed up in the office, of course when I was away. But he couldn't leave the phone for me, because he was required to collect money for it (that's fine), but could not find out how much (that sounds a bit silly, doesn't it?). He promised to find out and come back either the same day or this Monday.

Well, of course nobody showed up. Being patient as I am ;), I waited until today before I even wanted to have a casual look what's going on. The confirmation mail I received after my order contained a web link that should provide status information. Nice. Not so nice is that all I could get out of that page was the fact that the application developer had obviously forgotten to handle some Java exceptions ("javax.servlet.ServletException: Error while looking for EJB" - not exactly what I was looking for...).

OK, software can be buggy, so not a big deal. I called their hotline. Well, I tried to. First thing was that there was no phone number listed at all. Bad. So I resorted to the general Vodafone hotline. The expected happend: I ended up in he wait queue and was served that nice music. But, after roughly three minutes, the unexpected happend: "all of our Agents are still busy, please call us again later" the computer voice said - and quickly hung up the phone. Ummm... not nice. So was I supposed to call back again and start at the beginning of the queue? Looks so (oh man, would I like to have the QUEUE_ENQUE_IN_FRONT setting available to me...). I have to admit that at this point in time I was already a bit annoyed.

I called again. The very same happened. Some time later, I called back again... hangup, too. At this point, I sent a quite angry email to their contact address. I have to admit that it was brief and somewhat impolite and I expressed my expectation that the mail would most probably go to /dev/null immediately. Guess what? I got an auto-responder reply. Of course, a human reply is yet to be seen...

I tried to call the hotline again from time to time, but always I received a hangup after an apparent three-minute timeout (their system seems to be even more impatient than me). So it is obviously impossible to contact Vodafone customer service at all.

Maybe that should staff up their call center - or look for a phone provider who is capable to handle a larger caller queue...

rsyslog work log for 2008-01-08

Yesterday's rsyslog work log:

- fixed doc bug -- thanks to varmojfekoj for pointing it out
- fixed some memory leaks in new code -- thanks to varmojfekoj for the patch
- implemented queue disk reader to switch to multiple files
- first implementation of "disk" queue mode finished. It still needs some
work and the deserializer needs also to be expanded, but the queue at
least performs well now.
- fixed a race condition that could occur when input modules were terminated
- added --enable-mudflap ./configure option
- completed deserialization support in msg object (but not deserializer itself)
- completed object deserializer

Tuesday, January 08, 2008

work log for 2008-01-07

rsyslog work log. I am currently totally focused on the queue (and thus not very responsive on mail and otherwise):

- released 3.10.0
- performance-tuned stringbuf class
- implemented disk queue as far as I could without an object de-serializer
- implemented buffered read calls for the queue file
- implemented class type registry
- MsgSetProperty() implemented
- defined a property class
- implemented deserializer (needs some more work)

Monday, January 07, 2008

rsyslog work log for 2008-01-06

Here is the rsyslog work log for yesterday:

- fixed a bug with integer conversion in srUtils.c
- changed some lib functions to work on long instead of int
to care for 64 bit platforms (just to be on the save side)
- worked a bit on object serialization
- cleaned up msg structure (interestingly, there were for example
two fields with identical meaning and iSyslogVersion was never
used ;))
- completed serializer for msg (but needs review)
- did a little bit performance cleanup
- worked on object header (now also contains the size)

Saturday, January 05, 2008

rsyslog threading

If you followed my work logs or CVS updates, you've probably seen that I have worked quite a bit on rsyslog's threading. So I thought I share a view "design documents" that cover up the big picture.

Michael Biebel asked me for a few graphical representations of how the modules interact and what the message flow is. I am not a real good computer graphics guy, and an old-fashioned one. So I thought before I let you wait any longer, I share some of my hand sketches. They are not fancy, probably hard to read - but maybe still helpful. Find them below. A klick bring up the hires version, which is a bit less hard to read ;)

I'll try to add better graphics and descriptions as soon as I find some time. But I have to admit that I currently have so many things on my mind that I'd like to code first. So it may take a short while.

recent rsyslog work

Here is the rsyslog work log for the past days:

- fixed a few typos noticed by Jonathan Smith - thanks
- moved queue code to its own module (finally)
- restructured queue interface to use rsRetVal and instances, removed
dependency on globals - now more like a real class
- implemented queue type "drivers"
- queue is now a full object and handles threading by itself
- applied Michael Biebl's patch to clean up the makefiles
- added capability to use a linked list for queuing to the queue class
- added $MainMsgQueueType config parameter
- some cleanup
- added $SpoolDirectory config parameter
- added $MainMsgQueueFilePrefix config parameter
- begun working on disk queueing (not completed, do not use this mode!)
- begun some work on Msg Object serializiation
- created a kind of general base class
- removed serialization pointer from queue; used new base class instead
- utilized the new auto-destruction capability so that the queue can now
destruct user objects if needed
- changed queue object Construction/Startup interface
- added capability for concurrent access to the msg class. Can be dynamically
activated. If active, locking is employed.
- added the "direct" queueing mode to queue class (no queing at all)
- added multiple worker thread capability to queue class
- implemented $MainMsgQueueWorkerThreads config directive
- removed some no-longer-needed code (thanks Michael Biebl for the help)

Thursday, January 03, 2008

problems with the email server

We have some problems with the email server where my mailbox is located on. Right now, I can not receive any mails, but they are queued. So do not wonder if I may respond sluggish ;)

recent rsyslog work

A short rsyslog work log form the past days:

- created omtesting, a debug and development aid output module. This is stage
work for the new queueing engine - we need a way to delay rule execution
and that's what the module currently does ;)
- released 2.0.0

simplifying rsyslog JSON generation

With RESTful APIs, like for example ElasticSearch, you need to generate JSON strings. Rsyslog will soon do this in a very easy to use way. ...