Browse Source

documentation

tags/1.0^0
Y 2 years ago
parent
commit
91b1f15a9d
16 changed files with 1793 additions and 35 deletions
  1. +27
    -33
      README.md
  2. +3
    -2
      TODO.md
  3. +70
    -0
      doc/action_dailyReport.md
  4. +33
    -0
      doc/action_email.md
  5. +101
    -0
      doc/action_nftBan.md
  6. +99
    -0
      doc/builtinfilters.md
  7. +121
    -0
      doc/conffile.md
  8. +42
    -0
      doc/configure.md
  9. +37
    -0
      doc/counters.md
  10. +67
    -0
      doc/customize.md
  11. +48
    -0
      doc/intro_func.md
  12. +40
    -0
      doc/intro_tech.md
  13. +9
    -0
      doc/noop.md
  14. +1042
    -0
      extra/examples/full_pyruse.json
  15. +50
    -0
      extra/examples/get-systemd-stats.sh
  16. +4
    -0
      pyruse/log.py

+ 27
- 33
README.md View File

@@ -1,41 +1,35 @@
# Python peruser of systemd-journal

This program is intended to be used as a lightweight replacement for both epylog and fail2ban.
Its purpose is to peruse the system log entries, warn of important situations, report daily on the latest events, and act on specific patterns (IP address bans…).

The wanted features are these:

* Peruse all log entries from systemd’s journal, and only those (ie: no log files).
* Passively wait on new entries; no active polling.
* Filter-out uninteresting log lines according to the settings.
* Act on matches in the journal, with some pre-defined actions.
* Create a daily report with 2 parts:
- events of interest (according to the settings),
- and other non-filtered-out log entries.
* Send an immediate email when something important happens (according to the settings).

Interesting [filtering entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html) are:
* `_TRANSPORT`: how the log entry got to the journal (`stdout`, `syslog`, `journal`)
* `PRIORITY`: see https://en.wikipedia.org/wiki/Syslog#Severity_level
* `SYSLOG_FACILITY`: see https://en.wikipedia.org/wiki/Syslog#Facility
* `_CAP_EFFECTIVE`: effective capabilities as an hexadecimal mask
* `_BOOT_ID`: boot identifier (may be used to detect reboots)
* `_MACHINE_ID`: internal systemd ID for the machine where the log entry occurred
* `_HOSTNAME`: short hostname of the machine where the log entry occurred
* `_UID`: user ID of the systemd service that produced the log entry
* `_GID`: group ID of the systemd service that produced the log entry
* `SYSLOG_IDENTIFIER`: service name as reported to the “syslog” API
* `_COMM`: name of the command that produced the log entry
* `_EXE`: path to the executable file launched by systemd
* `_SYSTEMD_CGROUP`: cgroup of the service, eg. `/system.slice/systemd-uwsgi.slice/uwsgi@nextcloud.service`
* `_SYSTEMD_UNIT`: name of the systemd unit that produced the log entry
* `_SYSTEMD_SLICE`: name of the systemd slice
* `_CMDLINE`: process name as reported by the main process of the systemd service
* `_PID`: process ID of the systemd unit’s main process
* `MESSAGE`: the actual message of the log entry
* `__REALTIME_TIMESTAMP`: Python `datetime` of the log entry, formatted as: `YYYY-MM-DD HH:MM:SS:µµµµµµ`
* [Functional overview](doc/intro_func.md)
* [Technical overview](doc/intro_tech.md)

The software requirements are:

* a modern systemd-based Linux operating system (eg. [Archlinux](https://archlinux.org/)- or [Fedora](https://getfedora.org/)-based distributions);
* python, at least version 3.1 (or [more, depending on the modules](doc/intro_tech.md) being used);
* [python-systemd](https://www.freedesktop.org/software/systemd/python-systemd/journal.html);
* [nftables](http://wiki.nftables.org/) _if_ IP address bans are to be managed;
* a sendmail-like program _if_ emails are wanted.

The `/etc/pyruse` directory is where system-specific files are looked-for:
* the `pyruse.json` file that contains the configuration,
* the `pyruse/actions` and `pyruse/filters` subfolders, which may contain additional actions and filters.

* the `pyruse.json` file that contains the [configuration](doc/conffile.md),
* the `pyruse/actions` and `pyruse/filters` subfolders, which may contain [additional actions and filters](doc/customize.md).

Instead of using `/etc/pyruse`, an alternate directory may be specified with the `PYRUSE_EXTRA` environment variable.

For more in-depth documentation, please refer to these pages:

* [General structure of the `pyruse.json` file](doc/conffile.md)
* [How do I write the `pyruse.json` file?](doc/configure.md)
* [Writing custom filters and actions](doc/customize.md)
* More information about:
- [the built-in filters](doc/builtinfilters.md)
- [the counter-based actions](doc/counters.md)
- [the `action_noop` module](doc/noop.md)
- [the `action_email` module](doc/action_email.md)
- [the `action_dailyReport` module](doc/action_dailyReport.md)
- [the `action_nftBan` module](doc/action_nftBan.md)

+ 3
- 2
TODO.md View File

@@ -1,5 +1,6 @@
# TODO
# Backlog

* Improve documentation, especially on the contents of `pyruse.json`.
* Switch the `dailyReport` from a static layout to a light-weight template system.
* Maybe persist counters; they are currently lost on restart.
* Maybe switch from storing the daily journal in a file, to storing it in a database.
* Eventually make the code more elegant, as I learn more about Python…

+ 70
- 0
doc/action_dailyReport.md View File

@@ -0,0 +1,70 @@
# The daily report

This module fulfills one of the initial goal of the project: recording significant or unexpected log entries, and sending a report by email at the end of each day.

Currently, this module generates emails in both HTML and text ([asciidoc](http://asciidoctor.org/docs/asciidoc-syntax-quick-reference/)), with a static layout.
The layout goes roughtly like this:

```asciidoc
= Main title

== Title of the “WARN” section

… a table with the warnings of the past day …

== Title of the “INFO” section

… a table with the notable facts of the past day …

== Title of the “OTHER” section

… a list of the unexpected log entries for the past day …
```

The intended usage of Pyruse is to write the configuration so that:

1. When a log entry requires immediate attention, an email is sent; the fact may also be put in the report, for example in the `INFO` section (logically, the issue has already been taken care of when the daily report arrives).
2. If the issue is less urgent but still suspect or important, and should be reviewed, it is put in the `WARN` section of the report.
3. Other significant facts go to the `INFO` section; depending on later observations, these facts may evolve to the `WARN` section or get dismissed, by altering the configuration.
4. All known uninteresting log entries are discarded. The last execution chain would be an `action_dailyReport` for the `OTHER` section, that would record all log entries that went through all previous execution chains without being handled nor being discarded: these are the unexpected log entries, or log entries that are known but are too rare to be granted dedicated configuration rules.

When an `action_dailyReport` is used, there are two mandatory parameters:

* the `level` must be one of `WARN`, `INFO`, or `OTHER`;
* the `message` is a Python [string format](https://docs.python.org/3/library/string.html#formatstrings):
- this means that any key in the current entry may be referrenced by its name between curly braces;
- and that literal curly braces must be doubled, lest they are read as the start of a template placeholder.

In the `WARN` and `INFO` sections, there is one table row by unique message, and the messages are sorted in alphabetical order.
On each row, the table cells contain first the number of times the message was added to the section, then the message itself, and finally all the dates and times of occurrence.
_Note_: As a consequence, it is useless to put the date and time of occurrence in the message.

In the `OTHER` section, the messages are kept in chronological order, and prepended by their date and time of occurrence: “`date+time: message`”. It is thus useless to put the date and time of occurrence in the message.

Here are examples for each of the sections:

```json
{
"action": "action_dailyReport",
"args": { "level": "WARN", "message": "Nextcloud query failed because the buffer-size was too low" }
}

{
"action": "action_dailyReport",
"args": { "level": "INFO", "message": "XMPP server {xmppServer} did not provide a secure connection" }
}

{
"action": "action_dailyReport",
"args": { "level": "OTHER", "message": "[{PRIORITY}/{_HOSTNAME}/{_SYSTEMD_UNIT}] {MESSAGE}" }
}
```

I chose the `WARN` level for the first situation because, although there is no immediate security risk associated with this fact, I know that some users will experience a loss of functionality.

I chose the `INFO` level for the second situation because all is well with my server; however, depending on who the remote `xmppServer` is, I might want to add it to a whitelist of allowed unsecured peers.

As for the last example, it is the catch-all action, that will report unexpected log lines.

_Tip_: System administrators should know that the contents of the next daily report can always be read in Pyruse’s [storage directory](conffile.md), in the file named `action_dailyReport.py.journal`.
In this file, `L` is the section (aka. level: `1` for `WARN`, `2` for `INFO`, and `0` for `OTHER`), `T` is the Unix timestamp, and `M` is the message.

+ 33
- 0
doc/action_email.md View File

@@ -0,0 +1,33 @@
# Send an email

Action `action_email`’s purpose is to send a text-only email.
To this end, the [core email settings](conffile.md) must be set.
The only mandatory parameter for this action is `message`, which is a template for the message to be sent.
Optionally, the subject can be changed from the default (which is “Pyruse Notification”) by setting the `subject` parameter, a simple string.

The `message` parameter is a Python [string format](https://docs.python.org/3/library/string.html#formatstrings).
This means that any key in the current entry may be referrenced by its name between curly braces.
This also means that literal curly braces must be doubled, lest they are read as the start of a template placeholder.

Here are some examples:

```json
{
"action": "action_email",
"args": { "message": "Error on {_HOSTNAME} on {__REALTIME_TIMESTAMP}:\n{MESSAGE}" }
}

{
"action": "action_email",
"args": {
"subject": "Failure notification",
"message": "[{numberOfFailures:^9d}] Failed login from {thatIP}."
}
}
```

This last example renders as a centered space-padded-to-9-characters number between square brackets followed by a message, for example:

```
[ 12 ] Failed login from 12.34.56.78.
```

+ 101
- 0
doc/action_nftBan.md View File

@@ -0,0 +1,101 @@
# Ban IP addresses after they misbehaved

Linux provides a number of firewall solutions: [iptables](http://www.netfilter.org/), its successor [nftables](http://wiki.nftables.org/), and many iptables frontends like [Shorewall](http://www.shorewall.net/) or RedHat’s [firewalld](http://www.firewalld.org/).
For Pyruse, **nftables** was chosen, because it is modern and light-weight, and provides interesting features.

Action `action_nftBan` takes three mandatory arguments:

* `nftSetIPv4` and `nftSetIPv6` are the nftables sets where IP addresses will be added. The IP address is considered to be IPv6 if it contains at least one colon (`:`) character, else it is supposed to be IPv4.
* `IP` gives the name of the entry field to read, in order to get the IP address.

Note that nftables sets are kind-of paths.
For example, if your nftables ruleset is such:

```nftables
table ip Inet4 {
set mail_ban {
type ipv4_addr
flags timeout
}

chain FilterIn {
type filter hook input priority 0
policy drop
ip saddr @mail_ban drop
}
}
table ip6 Inet6 {
set mail_ban {
type ipv6_addr
flags timeout
}

chain FilterIn {
type filter hook input priority 0
policy drop
ip6 saddr @mail_ban drop
}
}
```

Then the values for `nftSetIPv4` and `nftSetIPv6` will be respectively “`Inet4 mail_ban`” and “`Inet6 mail_ban`”.

Optionally, a number may be specified with `banSeconds` to limit the time this ban will last.
The nice thing with nftables, is that it handles the timeouts itself: no need to keep track of the active bans and remove them using a Python program; the only reason why bans are recorded in a file, is to be able to restore them on reboot.

Here are examples:

```json
{
"action": "action_nftBan",
"args": { "IP": "thatIP", "banSeconds": 86400, "nftSetIPv4": "Inet4 mail_ban", "nftSetIPv6": "Inet6 mail_ban" }
}

{
"action": "action_nftBan",
"args": { "IP": "thatIP", "nftSetIPv4": "Inet4 sshd_ban", "nftSetIPv6": "Inet6 sshd_ban" }
}
```

## List the currently banned addresses

To see what IP addresses are currently banned, here is the `nft` command:

```bash
$ sudo nft 'list set Inet4 mail_ban'
table ip Inet4 {
set mail_ban {
type ipv4_addr
flags timeout
elements = { 37.49.226.159 timeout 5d expires 3d11h11m58s,
71.6.167.142 timeout 2d20h23m57s expires 6h12m49s,
91.200.12.96 timeout 4d18h22m29s expires 2d4h11m24s,
91.200.12.156 timeout 4d22h59m19s expires 2d8h48m15s,
91.200.12.203 timeout 4d22h53m38s expires 2d8h42m33s,
91.200.12.213 timeout 4d22h54m25s expires 2d8h43m21s,
91.200.12.217 timeout 4d18h1m14s expires 2d3h50m9s,
91.200.12.230 timeout 4d22h54m27s expires 2d8h43m23s,
139.201.42.59 timeout 5d expires 3d17h29m51s,
183.129.89.243 timeout 5d expires 4d9h4m37s }
}
}
```

_Note_: The un-rounded timeouts are post-reboot restored bans.

## Un-ban an IP address

It is bound to happen some day: you will want to un-ban a banned IP address.

Since `action_nftBan` does not keep the current bans in memory, it is enough to remove the ban using the `nft` command:

```bash
$ sudo nft 'delete element Inet4 mail_ban {10.0.0.10}'
```

However, the ban may be restored when restarting Pyruse.
To avoid that, also delete the corresponding record from the `action_nftBan.py.json` file in Pyruse’s [storage directory](conffile.md).

To go further, you could tweak your configuration, so that your trusted IP addresses never reach `action_nftBan`.

+ 99
- 0
doc/builtinfilters.md View File

@@ -0,0 +1,99 @@
# Built-in filters

Pyruse comes with a few very simple filters.

## `=`, `≤`, `≥`, `in`

The filters `filter_equals`, `filter_lowerOrEquals`, and `filter_greaterOrEquals` simply check equality on inequality between a given field, given by the parameter `field`, and a constant value, given py the parameter `value`.
Both parameters are mandatory.
Here are two examples:

```json
{
"filter": "filter_greaterOrEquals",
"args": { "field": "IPfailures", "value": 6 }
}

{
"filter": "filter_equals",
"args": { "field": "_SYSTEMD_UNIT", "value": "nginx.service" }
}
```

Filter `filter_in` works the same way as `filter_equals` does, except that instead of a single `value`, a `values` list is given, and equality between the field’s contents and any of the list’s items is considered a success.
Here is an example:

```json
{
"filter": "filter_in",
"args": { "field": "PRIORITY", "values": [ 2, 3 ] }
}
```

For any of these filters, the constant values must be of the same type as the typical contents of the chosen field.

## Perl-compatible regular expressions (pcre)

Filter `filter_pcre` should only be used on character strings.
Like the above filters, it works on a field given by the `field` parameter, and the [regular expression](https://docs.python.org/3/library/re.html) being looked for is given by the `re` parameter.
Both parameters are mandatory.

The regular expression in the `re` parameter may contain capturing groups:

* Named capturing groups use the `(?P<groupName>…)` notation; the captured value is always stored under the key `groupName` in the current entry.
* Anonymous capturing groups stem from the use of simple parenthesis: `(…)`; these are not saved by default, but a `save` parameter (a list) may be specified, so that the captured values get stored in the current entry, using the names given by `save`.

Here are two identical examples:

```json
{
"filter": "filter_pcre",
"args": {
"field": "MESSAGE",
"re": "^\\{core\\} Login failed: '(.*)' \\(Remote IP: '(.*)'\\)",
"save": [ "thatUser", "thatIP" ]
}
}

{
"filter": "filter_pcre",
"args": {
"field": "MESSAGE",
"re": "^\\{core\\} Login failed: '(?P<thatUser>.*)' \\(Remote IP: '(?P<thatIP>.*)'\\)"
}
}
```

Filter `filter_pcreAny` is to `filter_pcre` what `filter_in` is to `filter_equals`.
It works the same way as `filter_pcre`, except that instead of a single regular expression, its `re` parameter contains a list of regular expressions, and a match in the field’s contents is accepted with any of these regular expressions.

In contrast with `filter_pcre`, `filter_pcreAny` does not accept the `save` parameter: the order of fields cannot be guaranted to be the same accross several regular expressions.

Here is an example:

```json
{
"filter": "filter_pcreAny",
"args": {
"field": "MESSAGE",
"re": [
"^Failed password for (?P<thatUser>.*) from (?P<thatIP>(?!192\\.168\\.1\\.201 )[^ ]*) port",
"^Invalid user (?P<thatUser>.*) from (?P<thatIP>(?!192\\.168\\.1\\.201 )[^ ]*) port"
]
}
}
```

## User existence

Filter `filter_userExists` knows of only one —mandatory— parameter: `field`.
This filter is passing, if the system reports the user whose name is the value of the chosen field [as existing](https://docs.python.org/3/library/pwd.html#pwd.getpwnam), and non-passing otherwise.

Here is an example:

```json
{
"filter": "filter_userExists",
"args": { "field": "thatUser" }
}
```

+ 121
- 0
doc/conffile.md View File

@@ -0,0 +1,121 @@
# General structure of `pyruse.json` file

## Syntax

The Pyruse configuration file is in [JSON format](http://json.org/).
In short:

* There are dictionaries, where keys of type `string` are mapped to values of any type, using this syntax:

```json
{
"1st key": "value1",
"2nd key": "value2",
"last key": "no trailing comma"
}
```

* Lists are similar, with comma-separated items between square-brackets:

```json
[ 1, 10, 100 ]
```

* Values in dictionaries and lists can be:
- Unicode character strings, enclosed in straight double-quotes;
- all-digit numbers, with an optional dot if a decimal separator is needed;
- or the `true`, `false`, and `null` keywords, that have the same meaning as `True`, `False`, and `None` in Python.
* Inside strings, literal straight double-quotes (`"`) and back-slashes (`\`) must be prepended with a `\`. Besides, a `\` may be placed before some characters to alter their meaning, from a simple letter to a control character:
- `b`, `f`, `n`, `r`, `t` translate to backspace, form-feed, newline, carriage-return, and horizontal tabulation;
- `u` followed by 4 hexadecimal digits allows to enter hard-to-type Unicode characters using their codes.

The whole configuration file is a dictionary, where the core of Pyruse as well as some modules expect configuration entries.

## Core entries

The main entry in the configuration dictionary is the **`"actions"`** key.
Its value is a dictionary of execution chains, where each chain is identified by its label (a key in the `"actions"` dictionary).
Each chain is a list of filters and actions, described below.

Another core entry in the configuration dictionary is the **`"email"`** key.
Its value is a dictionary containing:

* `"from"` (string): the email address that will appear as sender of emails sent by Pyruse;
* `"to"` (list): the email addresses of the people to whom Pyruse emails will be sent;
* `"subject"` (string): the default subject of Pyruse emails;
* `"sendmail"` (list): the binary that will send emails (it is the first item in the list; usually sendmail, which may be a link to exim, msmtp, etc.), then optionally other strings that will be used as parameters of this binary; this program will be launched once for each email, and receive them on the standard input.

Also important are the **`"storage"`** key, a string that gives the path where Pyruse core and its modules are allowed to write state data (usually under `/var/lib`);
and the **`"8bit-message-encoding"`** key, a string that gives an 8-bit encoding to be used in cases when the systemd message is not valid UTF-8 (for example `iso-8859-1`).

Finally, the **`"debug"`** key (boolean), when set to `true`, lets Pyruse remember the action-chains’ names while reading the configuration, which results in better information when Pyruse encounters a problem.

### Actions and filters in execution chains

In an execution chain, both filters and chains are written as dictionaries. For filters, the `"filter"` key gives the module basename of the filter to use; likewise for actions, the `"action"` key gives the module basename of the action to use.
Both filters and actions also use the `"args"` key to give parameters to the module (it is a dictionary).

Besides, both filters and actions implicitely use a `"then"` key; the former also implicitely use an `"else"` key:

* `"then"` is automatically linked to the next module in the chain; if there is none:
- filters then pass control to the next available execution chain,
- while actions just stop there (no further handling of the log entry).
* `"else"` is only for filters, and acts the same as `"then"` does, except it is used when the filter does not pass, whereas `"then"` is used when the filter does pass.

It should be noted, that if an error happens inside an action, this action simply stops and no more processing happens for the current log entry; if an error happens inside a filter, this filter is considered non-passing, and the rules that apply are those of the `"else"` link.

If the default linking does not achieve the wanted result, it can be overriden, by explicitely giving the wanted execution chain’s name as a value of `"then"` or `"else"`.
_Important_: The configuration file is read from top to bottom. When an execution chain that has already been used as the target of an explicit `"then"` or `"else"` is encountered, this chain is skipped where default linking is concerned. This allows to write “jump chains” (as in Netfilter). Note that such chains should be declared below the location where they are first called, not above, or the result may not be what is expected.

See the documentation associated with specific filters and actions to get details on their expectations regarding the `"args"` dictionary.

## Module entries

By convention, a module that needs its own configuration entries puts them in a dictionary, introduced by a key reflecting the module’s name.

See the documentation associated with specific filters and actions to get details on their specific configuration needs.

## Example configuration file

Here is a minimal example:

```json
{
"actions": {
"Immediately warn of fatal errors": [
{
"filter": "filter_lowerOrEquals",
"args": { "field": "PRIORITY", "value": 1 }
},
{
"action": "action_email",
"args": { "subject": "Pyruse", "message": "Error on {_HOSTNAME} on {__REALTIME_TIMESTAMP}:\n{MESSAGE}" }
}
],
"Discard info entries": [
{
"filter": "filter_greaterOrEquals",
"args": { "field": "PRIORITY", "value": 4 }
},
{
"action": "action_noop"
}
],
"Report everything else": [
{
"action": "action_dailyReport",
"args": { "level": "OTHER", "message": "[{PRIORITY}] {_SYSTEMD_UNIT}/{_HOSTNAME}: {MESSAGE}" }
}
]
},
"email": {
"from": "pyruse@example.org",
"to": [ "hostmaster@example.org" ],
"subject": "Pyruse Report",
"sendmail": [ "/usr/bin/sendmail", "-t" ]
},
"8bit-message-encoding": "iso-8859-1",
"storage": "/var/lib/pyruse",
"debug": false
}
```

+ 42
- 0
doc/configure.md View File

@@ -0,0 +1,42 @@
# Configuration tips

In contrast with legacy log parsers, Pyruse works with structured [systemd-journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html). This allows for better performance, since targeted comparisons become possible.

The general intent, when writing the configuration file, should be to handle the log entries that appear the most often first, in as few steps as possible. For example, I ran some stats on my server before writing my own configuration file; I got:

| systemd units | number of journal entries |
| ------------------------------- | -------------------------:|
| `prosody.service` | 518019 |
| `gitea.service` | 329389 |
| `uwsgi@nextcloud.service` | 217342 |
| `session-*.scope` | 89813 |
| `nginx.service` | 80762 |
| `dovecot.service` | 61898 |
| `exim.service` | 60743 |
| `init.scope` | 43021 |
| `nextcloud-maintenance.service` | 20775 |
| `haproxy.service` | 18445 |
| `user@*.service` | 7306 |
| `minidlna.service` | 6032 |
| `loolwsd.service` | 5797 |
| `sshd.service` | 4959 |
| `spamassassin-update.service` | 2383 |
| `systemd-nspawn@*.service` | 1497 |
| `nslcd.service` | 867 |
| `nfs-mountd.service` | 723 |
| `systemd-logind.service` | 696 |
| `nfs-server.service` | 293 |
| `systemd-networkd.service` | 121 |
| misc. units with < 100 entries | |

For reference, here is the command that gives these statistics:

```bash
$ bash ./extra/examples/get-systemd-stats.sh >~/systemd-units.stats.tsv
```

One should also remember, that numeric comparison are faster that string comparison, which in turn are faster than regular expression pattern-matching. Further more, some log entries are not worth checking for, because they are too rare: it costs more to grab them with filters (that most log entries will have to pass through), than letting them get caught by the catch-all last execution chain, which typically uses the `action_dailyReport` module.

An efficient way to organize the configuration file is by handling units from the most verbose to the least verbose, and for each unit, filter-out useless entries based on the `PRIORITY` (which is an integer number) whenever it is possible. In short, filtering on the actual message, while not entirely avoidable, is the last-resort operation.

An [example based on the above statistics](../extra/examples/full_pyruse.json) is available in the `extra/examples/` source directory.

+ 37
- 0
doc/counters.md View File

@@ -0,0 +1,37 @@
# Counter-based actions

Pyruse currently allows to raise a counter, with `action_counterRaise`, or to reset a counter (to zero), with `action_counterReset`. Adding an `action_counterLower` would be trivial, but has not been necessary so far.

Counters are kept in memory, by category; there are as many categories as wanted per the configuration file.
In each category, independent counters are kept for the different keys encountered while reviewing the log entries.
For example, there may be categories “mailFailures”, “sshFailures”, and “sshRecidives”, and then in each category there would be one counter per IP address that failed to use the service properly.

Thus, all counter-based actions need two mandatory parameters: the `counter` parameter gives the category name, and the `for` parameter indicates the field of the current entry in which the counter key must be read.

Besides, all counter-based actions accept the optional `save` parameter, which gives the name under which the resulting value of the counter should be stored in the current entry, for further processing (_note_: the value of a counter after being processed by `action_counterReset` is always `0`).
Finally:

* `action_counterRaise` may be given the `keepSeconds` parameter to specify how long this counter-raise should be recorded (indefinitely by default);
* `action_counterReset` may be given the `graceSeconds` parameter to specify how long this counter-reset should be enforced (the default is to immediately allow counter-raises).

Here are some examples:

```json
{
"action": "action_counterRaise",
"args": { "counter": "http", "for": "thatIP", "keepSeconds": 300, "save": "IPfailures" }
}

{
"action": "action_counterRaise",
"args": { "counter": "ssh", "for": "keyUser" }
}

{
"action": "action_counterReset",
"args": { "counter": "mail", "for": "emailSender", "graceSeconds": 900 }
}
```

Counters are auto-cleaned: they disapear when their value becomes zero (either with a reset, or due to `keepSeconds`), and they have no `graceSeconds` left.
If you use unlimited counters (no `keepSeconds`), be sure to reset them when you act on them after they have crossed a chosen threshold, so these counters can be “garbage-collected”.

+ 67
- 0
doc/customize.md View File

@@ -0,0 +1,67 @@
# Writing custom modules

Custom filters are Python files written in `/etc/pyruse/pyruse/filters/`.
Custom actions are Python files written in `/etc/pyruse/pyruse/actions/`.

Filters must define a class named `Filter` that extends Pyruse’s own `Filter` class from the `pyruse.base` namespace.
By convention, a filter module name starts with `filter_`. A filter module looks like this:

```python
from pyruse import base

class Filter(base.Filter):
def __init__(self, args):
super().__init__()
# get mandatory arguments with args["param_name"]
# get optional arguments with args.get("param_name", default_value)
# store in self.whatever the data that is needed at each run of filter below

def filter(self, entry):
# return true for the "then" link, or false for the "else" link
return some_check(entry["a_field"], entry["another_field"])
```

Actions must define a class named `Action` that extends Pyruse’s own `Action` class from the `pyruse.base` namespace.
By convention, an action module name starts with `action_`. An action module looks like this:

```python
from pyruse import base

class Action(base.Action):
def __init__(self, args):
super().__init__()
# get mandatory arguments with args["param_name"]
# get optional arguments with args.get("param_name", default_value)
# store in self.whatever the data that is needed at each run of act below

def act(self, entry):
# do whatever this action is supposed to do
```

Some actions may need to restore a state at boot, or each time the main Pyruse program is restarted. The aim usually is to configure an external tool (firewall, etc.), based on files, or a database…
In such cases:

* The action’s constructor must be altered so that it does not fail if `args` is `None`:

```python
def __init__(self, args):
super().__init__()
if args is None:
return
```

* A new `boot` method must be defined; it will get called at boot and this is where the wanted state shall be restored:

```python
def boot(self):
# do whatever must be done
```

* Assuming the action is named `action_myModule`, the systemd unit `pyruse-boot@action_myModule.service` should be enabled. If this unit has dependencies, these must be declared before enabling the specific `pyruse-boot` service, by creating a drop-in with the dependencies, for example:

```
# /etc/systemd/system/pyruse-boot@action_myModule.service.d/action_myModule.conf
[Unit]
Requires=iptables.service
After=iptables.service
```

+ 48
- 0
doc/intro_func.md View File

@@ -0,0 +1,48 @@
# Introduction to what Pyruse is

Everyone knows [Fail2ban](http://www.fail2ban.org/).
This program is excellent at what it does, which is matching patterns in a number of log files, and keeping track of counters in order to launch actions (ban an IP, send an email…) when thresholds are crossed.
[Fail2ban is extremely customizable](http://yalis.fr/cms/index.php/post/2014/11/02/Migrate-from-DenyHosts-to-Fail2ban)… to a point; and to my knowledge it cannot benefit from all the features of modern-day logging with systemd.

Then, there is the less-known and aging [Epylog](http://freshmeat.sourceforge.net/projects/epylog/); several programs exist with the same features.
Just like Fail2ban, this program continuously scans the logs to do pattern matching.
In contrast to Fail2ban, though, this tool’s purpose is to deliver an email every day, with a summary of the past day’s events.
Epylog is unfortunately very limited, with few hooks, if the default behaviour is not exactly what you want.

The point is, both kinds of tools have a lot of overlap in their inner workings, even though their outcome differ.
This is a waste of resources.
Besides, these tools suffer from the legacy handling of logs, in files, where text messages are roughtly the only information available.

Guidelines for Pyruse:

* **modern**: systemd, python3, no deprecated API…
* **light-weight**;
* **efficient**.

At the origin of the project are these wanted features:

* Peruse all log entries from systemd’s journal, and only those (ie: no log files).
* Passively wait on new entries; no active polling.
* Filter-out uninteresting log lines according to the settings.
* Act on matches in the journal, with some pre-defined actions available.
* Create a daily report with 3 parts:
- events of importance (according to the settings) that should be checked,
- events of interest (according to the settings),
- and other non-filtered-out log entries.
* Send an immediate email when something very important happens (according to the settings).

The result looks a bit like the way a Netfilter firewall is built, with [execution chains made of filters and actions](configure.md).
Both filters and actions work on a systemd-journal entry, where all fields are available, and more fields can be computed and stored, to be worked-on later, and so on.

The most interesting [filtering or informational entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html) are:

* `PRIORITY`: see [Syslog at Wikipedia](https://en.wikipedia.org/wiki/Syslog#Severity_level) for the definitions
* `SYSLOG_FACILITY`: see [Syslog at Wikipedia](https://en.wikipedia.org/wiki/Syslog#Facility) for the definitions
* `_HOSTNAME`: short hostname of the machine where the log entry occurred
* `_UID`: user ID of the systemd service that produced the log entry
* `_GID`: group ID of the systemd service that produced the log entry
* `_SYSTEMD_UNIT`: name of the systemd unit that produced the log entry
* `MESSAGE`: the actual message of the log entry
* `__REALTIME_TIMESTAMP`: Python `datetime` of the log entry (gets formatted as: `YYYY-MM-DD hh:mm:ss:µµµµµµ`)

Pyruse already comes with some common modules. [More can be easily written](customize.md).

+ 40
- 0
doc/intro_tech.md View File

@@ -0,0 +1,40 @@
# Technical overview of Pyruse

Pyruse is built on [python-systemd](https://www.freedesktop.org/software/systemd/python-systemd/journal.html).
This API already brings a number of benefits:

* Each log entry is obtained as a Python dictionary where all systemd-journal fields are available.
* Each log field has the good data type, eg. the systemd timestamp becomes a Python `datetime`.
* An API call is provided to passively wait for new log entries.

The core of Pyruse has no other dependency.
However, each module may have its own dependencies, for example the availability of a program that can accept emails on its standard input (usually `/usr/bin/sendmail`), for modules that send emails.

It should be noted, that modern Python API are used. Thus:

* Python version ≥ 3.1 is required for managing modules (`importlib`);
* Python version ≥ 3.1 is required for loading the configuration (json’s `object_pairs_hook`);
* Python version ≥ 3.2 is required for the daily report and emails (string’s `format_map`);
* Python version ≥ 3.5 is required for IP address bans and emails (subprocess’ `run`);
* Python version ≥ 3.6 is required for sending emails (`headerregistry`, `EmailMessage`).

In order to be fast, this program avoids dynamic decisions while running.
To this end, a static workflow of filters and actions is built upon start, based on the configuration file.
After that, log entries are pushed into the workflow, much like a train is on a railway, the switchpoints being the Pyruse filters.

Pyruse is split into several Python files:

* `main.py`: As expected, this is the conductor, responsible for interfacing with the configuration, the workflow, and systemd. It also has a “boot” mode, where actions are allowed to set the stage before Pyruse starts.
* `config.py`: The `Config` class is responsible for reading the configuration file, and making it available as a Python dictionary.
* `log.py`: This one allows Pyruse’s own logs to be forwarded to systemd’s journal, which opens the road to recidive detection for example.
* `workflow.py`: This is where the configuration file get translated into actual execution chains, linked together into a single static workflow.
* `module.py`: Whenever the workflow needs to add a filter or an action to an execution chain, this module finds it in the filesystem.
* `base.py`: All actions and filters inherit from the `Action` and `Filter` classes defined in there; they act as an abstraction layer between the workflow and the modules.
* `counter.py`: This utility class is parent to modules that manage a counter (Python supports multiple-inheritance).
* `email.py`: This utility class is parent to modules that send emails (Python supports multiple-inheritance).

All else is actions and filters…
Some are delivered with Pyruse itself; [more can be added](customize.md).

_Tip_: The documentation is part of the source repository.
Contributions to the code or the documentation are welcome `;-)`

+ 9
- 0
doc/noop.md View File

@@ -0,0 +1,9 @@
# The `action_noop` module

This action does nothing.
Its main purpose is to discard uninteresting log entries.
This is the only module that takes no parameter:

```json
{ "action": "action_noop" }
```

+ 1042
- 0
extra/examples/full_pyruse.json
File diff suppressed because it is too large
View File


+ 50
- 0
extra/examples/get-systemd-stats.sh View File

@@ -0,0 +1,50 @@
#!/bin/bash
# $1 (optional): systemd-nspawn machine name

{
printf 'Units\tTotal\tP7\tP6\tP5\tP4\tP3\tP2\tP1\tP0\n'
sudo journalctl ${1:+-M "$1"} -o json-pretty --output-fields=_SYSTEMD_UNIT,PRIORITY \
| tr -d $'"\t, ' \
| awk -F: -vOFS=: '
/^\{/ {
u = ""
p = -1
}
$1 == "PRIORITY" {
p = $2
}
$1 == "_SYSTEMD_UNIT" {
u = gensub(\
"@.*(\\.[^.]*)$",\
"@*\\1",\
1,\
gensub("-[^-]*[0-9][^-]*(\\.[^.]*)$", "-*\\1", 1, $2)\
)
}
/^\}/ {
if (p >= 0) print u, p
}
' \
| sort \
| awk -F: -vOFS=$'\t' '
function out() {
if (u != "")
print u,\
(p[8]+p[7]+p[6]+p[5]+p[4]+p[3]+p[2]+p[1]),\
p[8], p[7], p[6], p[5], p[4], p[3], p[2], p[1]
split("0:0:0:0:0:0:0:0", p)
u = ""
}
$1 != u {
out()
u = $1
}
{
p[1 + $2] += 1
}
END {
out()
}
' \
| sort -t$'\t' -k2,2rn
}

+ 4
- 0
pyruse/log.py View File

@@ -19,6 +19,10 @@ def debug(string):
global DEBUG
log(DEBUG, string)

def notice(string):
global NOTICE
log(NOTICE, string)

def error(string):
global ERR
log(ERR, string)

Loading…
Cancel
Save