Journalwatch - notify for anomalies in your logging

In the old times when logging was done by syslog, there was a very useful tool called logcheck. In todays world with systemd and journald there is a much better solution to logwatch: this is journalwatch. You can find Journalwatch on gitlab.

What is Journalwatch

Journalwatch filters all messages in your logs looking for anomalies. If something important is happening, it will immediately send you an email. All other anomalies are sent once per hour (configurable). All messages are filtered such that everything you want to be ignored is not sent to you. The result is that only anomalies are sent to you, and all the logs that can safely be ignored are just ignored.

How does it work - ordering of filters

First there is a filter that matches important things that should be sent in short notice. Log entries that match this filter are marked as 'warning'. This usually contains a few generic filters such as "anything that has a high priority" and "anything that contains the words error, fail or warning". Second there is a filter that matches on the entries that are marked as 'warning' (so those matching the first filter) in order to ignore them. These are usually very specific filters such as "ignore the warning from sshd that the maximum number of authentication attempts is exceeded". Third there is a filter that matches on all entries in order to ignore them. Anything that matches this filter is never shown. Fourth and last there is a filter that matches only the entries that are not marked as warning, and ignores those. Any entries that have not been matched by the above filters are sent as 'notice'. By default once an hour, but this is configurable. Any entries that have been marked as warning by the first filter, and have not been matched by the second and third filter are sent as 'warning', by default within a minute.

Journalwatch improvements over logcheck or journalcheck (or journalogs)

  1. Logcheck and journalcheck run from the cron daemon and restart every few minutes. That means they have to load and compile all patterns every time they start. Journalwatch runs as a daemon and executes just once, loads the config file just once, compiles the patterns just once, etc. Since the memory footprint is very small this makes it very lightweight.
  2. Logcheck and journalcheck match every log entry (many thousands a day) against every pattern (hundreds of them). On my test machine that means 2000000 matches per day. Journalwatch compiles all patterns that use the same identifier into a single compiled regex, so usually every log entry only needs to be matched against 5 or so compiled patterns. On the same test machine that is 50000 matches per day. This is over 50 times more efficient in both cpu time as well as memory usage.
  3. Logcheck and journalcheck need to start multiple tools to output logs, pipe them trough egrep, and email. Journalwatch is a single python program that starts up once, reads logs directly from journald and does all this in a single program. Which is much more lightweight.
  4. Logcheck and journalcheck run from cron in a predefined rhythm which does not distinguish between warnings or notices. If something slightly-interesting-but-not-so-important happens regularly it spams your email box with thousands of messages. Journalwatch sends warnings almost immediately, but groups notices and sends them as a group just once in a while.
  5. Logcheck and journalcheck can only filter on the message of the entry. Because Journalwatch integrates with journald it can filter on any property of the log entry, like the user that caused the log entry, the log entry priority, the name of the process that created the log entry, the gid, cgroup, systemd unit, etc. etc.

Now go get journalwatch from https://gitlab.com/olivierbf/journalwatch!