Double quote to prevent globbing

https://www.shellcheck.net/wiki/SC2086 Signed-off-by: Strykar <strykar@masked.invalid>
ipset support; fixes #1
2022-10-21 05:25:40 +02:00 · 2018-03-17 17:46:22 +01:00 · 2018-03-16 13:42:01 +01:00 · 2018-03-12 18:18:28 +01:00
24 changed files with 1556 additions and 535 deletions
--- a/Changelog.md
+++ b/Changelog.md
@ -0,0 +1,12 @@
+# Changelog
+
+This file is not intended as a dupplicate of Git logs.
+Its purpose is to warn of important changes between version, that users should be aware of.
+
+## v2.0
+
+After this version is installed, the following command should be run on the `action_nftBan.py.json` file:
+
+```bash
+$ sudo sed -i s/nftSet/nfSet/g action_nftBan.py.json
+```
--- a/README.md
+++ b/README.md
@ -1,14 +1,36 @@
 # Python peruser of systemd-journal

+## Summary
+
 This program is intended to be used as a lightweight replacement for both epylog and fail2ban.
 Its purpose is to peruse the system log entries, warn of important situations, report daily on the latest events, and act on specific patterns (IP address bans…).

 * [Functional overview](doc/intro_func.md)
 * [Technical overview](doc/intro_tech.md)

+The benefits of Pyruse over products of the same kind are:
+
+* **Optimization brought by systemd**  
+[systemd-journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html) play an important role in Pyruse: instead of matching log entries against message patterns only, the whole range of systemd’s journal fields is available. This allows for the much faster integer comparisons (`PRIORITY`, `_UID`…), or even faster comparisons with short strings like the `SYSLOG_IDENTIFIER`, `_SYSTEMD_UNIT`, or `_HOSTNAME`, with the opportunity to test more often for equality, and less for regular expressions.
+
+* **Optimization brought by context**  
+Programs that peruse the system logs usually apply a set of rules on each log entry, rule after rule, regardless of what can be deduced by the already-applied rules.  
+In contrast, each fact learnt by applying a rule in Pyruse can be taken into account so that rules that do not apply are not even considered.  
+For example, after matching the `SYSLOG_IDENTIFIER` of a journal entry to the value `sshd`, only SSH-related rules are applied, not Nginx-related rules, nor Prosody-related rules.
+
+* **Modularity**  
+Each filter (ie. a matching step) or action (eg. a ban, an email, etc.) is a Python module with a very simple API. As soon as a new need arises, a module can be written for it.  
+For example, to my knowledge, there is no equivalent in any tool of the same scale, for the [DNAT-correcting actions](doc/dnat.md) now included with Pyruse.
+
+## Get Pyruse
+
 Pyruse is [packaged for Archlinux](https://aur.archlinux.org/packages/pyruse/).
 For other distributions, please [read the manual installation instructions](doc/install.md).

+Whenever your upgrade Pyruse, make sure to check the [Changelog](Changelog.md).
+
+## Configuration
+
 The `/etc/pyruse` directory is where system-specific files are looked-for:

 * the `pyruse.json` file that contains the [configuration](doc/conffile.md),
@ -16,6 +38,8 @@ The `/etc/pyruse` directory is where system-specific files are looked-for:

 Instead of using `/etc/pyruse`, an alternate directory may be specified with the `PYRUSE_EXTRA` environment variable.

+## Documentation
+
 For more in-depth documentation, please refer to these pages:

 * [General structure of the `pyruse.json` file](doc/conffile.md)
@ -24,7 +48,8 @@ For more in-depth documentation, please refer to these pages:
 * More information about:
    - [the built-in filters](doc/builtinfilters.md)
    - [the counter-based actions](doc/counters.md)
+    - [the DNAT-related actions](doc/dnat.md)
+    - [the actions that log and ban](doc/logandban.md)
    - [the `action_noop` module](doc/noop.md)
    - [the `action_email` module](doc/action_email.md)
    - [the `action_dailyReport` module](doc/action_dailyReport.md)
-    - [the `action_nftBan` and `action_log` modules](doc/logandban.md)
--- a/doc/configure.md
+++ b/doc/configure.md
@ -1,33 +1,46 @@
 # Configuration tips

-In contrast with legacy log parsers, Pyruse works with structured [systemd-journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html). This allows for better performance, since targeted comparisons become possible.
+In contrast with legacy log parsers, Pyruse works with structured [systemd-journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html).
+This allows for better performance, since targeted comparisons become possible.

-The general intent, when writing the configuration file, should be to handle the log entries that appear the most often first, in as few steps as possible. For example, I ran some stats on my server before writing my own configuration file; I got:
+The general intent, when writing the configuration file, should be to handle the log entries that appear the most often first, in as few steps as possible.
+For example, I ran some stats on my server on the log entries of the past week; I got:

-| systemd units                   | number of journal entries |
-| ------------------------------- | -------------------------:|
-| `prosody.service`               |                    518019 |
-| `gitea.service`                 |                    329389 |
-| `uwsgi@nextcloud.service`       |                    217342 |
-| `session-*.scope`               |                     89813 |
-| `nginx.service`                 |                     80762 |
-| `dovecot.service`               |                     61898 |
-| `exim.service`                  |                     60743 |
-| `init.scope`                    |                     43021 |
-| `nextcloud-maintenance.service` |                     20775 |
-| `haproxy.service`               |                     18445 |
-| `user@*.service`                |                      7306 |
-| `minidlna.service`              |                      6032 |
-| `loolwsd.service`               |                      5797 |
-| `sshd.service`                  |                      4959 |
-| `spamassassin-update.service`   |                      2383 |
-| `systemd-nspawn@*.service`      |                      1497 |
-| `nslcd.service`                 |                       867 |
-| `nfs-mountd.service`            |                       723 |
-| `systemd-logind.service`        |                       696 |
-| `nfs-server.service`            |                       293 |
-| `systemd-networkd.service`      |                       121 |
-| misc. units with < 100 entries  |                           |
+| SYSLOG_IDENTIFIER       | number of journal entries |
+| ----------------------- | -------------------------:|
+| `uwsgi` (for Nextcloud) |                     55930 |
+| `gitea`                 |                     38923 |
+| `prosody`               |                     25596 |
+| `haproxy`               |                     21877 |
+| `postgres`              |                     12990 |
+| `nginx`                 |                     12808 |
+| `dovecot`               |                      7062 |
+| `exim`                  |                      2540 |
+| `systemd`               |                      1997 |
+| `su`                    |                      1458 |
+| `ownCloud` (Nextcloud)  |                      1067 |
+| `sshd`                  |                      1051 |
+| `mandb`                 |                       953 |
+| `spamd`                 |                       855 |
+| `pyruse`                |                       615 |
+| `kernel`                |                       420 |
+| `msmtp`                 |                       295 |
+| `sa-compile`            |                       255 |
+| `ansible-*`             |                       103 |
+| `systemd-logind`        |                       102 |
+| `python`                |                        78 |
+| `rpc.mountd`            |                        52 |
+| `ldapwhoami`            |                        42 |
+| `prosody_auth`          |                        42 |
+| `minidlnad`             |                        39 |
+| `kill`                  |                        28 |
+| `sudo`                  |                        26 |
+| `loolwsd`               |                        17 |
+| `exportfs`              |                        15 |
+| `dehydrated`            |                         6 |
+| `sa-update`             |                         5 |
+| `nslcd`                 |                         4 |
+| `rpc.idmapd`            |                         1 |

 For reference, here is the command that gives these statistics:

@ -37,6 +50,16 @@ $ bash ./extra/examples/get-systemd-stats.sh >~/systemd-units.stats.tsv

 One should also remember, that numeric comparison are faster that string comparison, which in turn are faster than regular expression pattern-matching. Further more, some log entries are not worth checking for, because they are too rare: it costs more to grab them with filters (that most log entries will have to pass through), than letting them get caught by the catch-all last execution chain, which typically uses the `action_dailyReport` module.

-An efficient way to organize the configuration file is by handling units from the most verbose to the least verbose, and for each unit, filter-out useless entries based on the `PRIORITY` (which is an integer number) whenever it is possible. In short, filtering on the actual message, while not entirely avoidable, is the last-resort operation.
+An efficient way to organize the configuration file is by handling Syslog-identifiers from the most verbose to the least verbose, and for each one, filter-out useless entries based on the `PRIORITY` (which is an integer number) whenever it is possible.
+In short, filtering on the actual message, while not entirely avoidable, is the last-resort operation.
+
+NOTE: I used to group my log entries (and Pyruse execution chains) by `_SYSTEMD_UNIT`, which seemed logical at the time.
+However, for some reason, there is some “leaking” of logs from some units to others; for example, I had Nginx logs appearing in the Exim `_SYSTEMD_UNIT`… The reason probably lies somewhere in inter-process communication, or with the launching of external commands.
+Anyway, I found that grouping by `SYSLOG_IDENTIFIER` actually gives better results:
+
+* `SYSLOG_IDENTIFIER` names are shorter than `_SYSTEMD_UNIT` names, hence probably quicker to compare `:-p`
+* Several `_SYSTEMD_UNIT` names from generic units (like `unit-name@instance-name`) end up into the same `SYSLOG_IDENTIFIER`, which allows to occasionaly replace `filter_pcre` with `filter_equals`.
+* A single program often does several tasks, and `SYSLOG_IDENTIFIER` reflects this diversity, which makes writing rules much easier.  
+For example, Pyruse sends emails using msmtp; I do not care about `msmtp`’s logs, but I do about `pyruse`’s. Filtering-out logs from the `msmtp` `SYSLOG_IDENTIFIER` is much easier to do than getting rid of email-related logs from the `pyruse.service` systemd unit.

 An [example based on the above statistics](../extra/examples/full_pyruse.json) is available in the `extra/examples/` source directory.
--- a/doc/dnat.md
+++ b/doc/dnat.md
@ -0,0 +1,175 @@
+# DNAT-correcting actions
+
+## Introduction
+
+Pyruse provides two actions, namely `action_dnatCapture` and `action_dnatReplace`, that work together towards a single goal: giving to Pyruse’s filters and actions the illusion that client connections come directly to the examined services, instead of going through a firewall Network Address Translation or a reverse-proxy.
+
+If for example you run a Prosody XMPP server (or anything that does not handle the [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt)) behind an HAProxy reverse-proxy, or if all your network connections go through a NAT’ing router in your DMZ, then the following happens: the services report your proxy as being the client, ie. usually `127.0.0.1` (same machine) or your LAN’s router IP.
+
+Here is a simplified illustration of the network configuration:
+
+```ditaa
+/-------------------------------\
+|         +---------------------+
+| Client  | ClientIP:ClientPort +---\==\
+|         +---------------------+   |  :
+\-------------------------------/   |  :
+                                 (1)|  :(2)
+/-------------------------------\   |  :
+|       +--++-------------------+   |  :
+|       |  PublicIP:PublicPort  +<--/  :
+|       +-----------------------+      :    /---------------------------------\
+| Proxy                         |      :    +-----------------------+         |
+|       +-----------------------+      \===>+                       | Service |
+|       | ProxyLanIP:RandomPort +---------->+ ServiceIP:ServicePort |         |
+|       +-----------------------+   (1)     +-----------------------+         |
+\-------------------------------/           \---------------------------------/
+```
+
+The circuit number `(1)` is the real one, which is why the service sees `ProxyLanIP:RandomPort` as the client.
+The circuit number `(2)` is what Pyruse will fake, using the fore-mentionned actions.
+
+First some “vocabulary”. In `action_dnatCapture` and `action_dnatReplace`:
+
+* `ClientIP` and `ClientPort` are called `saddr` and `sport` (`s` for **s**ource)
+* `ProxyLanIP` and `RandomPort` are called `addr` and `port`
+* `ServiceIP` and `ServicePort` are called `daddr` and `dport` (`d` for **d**estination)
+
+Pyruse’s actions work by storing the link between these 6 values in memory, and later replacing `addr` with `saddr`, and optionaly `port` with `sport`.
+
+## Action `action_dnatCapture`
+
+For Pyruse to be able to capture the wanted values, the proxy software must first be configured to provide them.
+
+### HAProxy configuration
+
+Here is an example configuration that reproduces the default `tcplog` format, simply adding the missing information between square braquets:
+
+```haproxy
+global
+    log        /dev/log local0 info
+    … misc. other options …
+
+defaults
+    mode       tcp
+    log        global
+    option     log-separate-errors
+    log-format "%ci:%cp [%t] %ft %b[%bi:%bp]/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"
+    … misc. other options …
+```
+
+The above configuration would produce log lines like this one:
+
+```log
+12.34.56.78:54321 [dd/MM/yyyy:HH:mm:ss.…] tls~ xmpp[10.0.0.1:43210]/xmpp …/…/… … -- …/…/…/…/… …/…
+```
+
+### nftables configuration
+
+Here is an example rule for nftables on the proxy:
+
+```nftables
+tcp dport 22 log prefix "DNAT/ssh: " dnat to 10.0.0.2
+```
+
+Having an easily recognizable log prefix helps.
+The above would result in a line like this one:
+
+```log
+DNAT/ssh: IN=… OUT=… MAC=… SRC=12.34.56.78 DST=10.0.0.1 LEN=… … PROTO=… SPT=43210 DPT=22 WINDOW=… …
+```
+
+Besides, Netfilter logs (part of the kernel logs) must be enabled for those log lines to actually appear in the logs.
+For example, it [may be required](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2851940ffee313e0ff12540a8e11a8c54dea9c65) to run this:
+
+```bash
+sysctl net.netfilter.nf_log_all_netns=1
+```
+
+### Pyruse configuration
+
+Action `action_dnatCapture` must tell Pyruse what fields in the current log entry match the 6 parameters described in the introduction.
+If some values are constant and known (or marker values are wished for), but these values are in no available field in the log entry, then those values may be given in the parameters `addrValue`, `portValue`, `daddrValue`, and `dportValue`.
+When an information is given by both a field reference and a plain value, the field reference is used first, and the plain value is used as a default value if the referenced field is not found.
+
+The `saddr` parameter is mandatory (else there is no point), and either `addr` or `addrValue` (or both) must be given as well.
+
+In addition to all these parameters, a `keepSeconds` parameter may be given to indicate how many seconds the detected correspondance should be kept in memory (default value is 63).
+
+NOTE: For performance reasons, the `keepSeconds` value is rounded up to the _next_ power of 2 — eg. values 4 to 7 are rounded up to 8 —, and the retention countdown only begins at the next occurrence of that power of two in the current time, expressed as a Unix timestamp.  
+As a consequence, the actual length of time a correspondance is kept in memory, varies between 1× and 4× the length of time given by the parameter, depending on the chosen value, and depending on the current time-of-the-day when the correspondance is found.
+
+Here is an example configuration that would work fine with log lines as produced by nftables (see above):
+
+```json
+{ "filter": "filter_pcre",
+  "args": {
+    "field": "MESSAGE",
+    "re": "^DNAT/ssh:.* SRC=([^ ]+) DST=([^ ]+) .* SPT=([^ ]+) DPT=([^ ]+) ",
+    "save": [ "dnatSaddr", "dnatAddr", "dnatPort", "dnatDport" ]
+  }
+},
+{ "action": "action_dnatCapture",
+  "args": {
+    "saddr": "dnatSaddr",
+    "addr":  "dnatAddr",  "addrValue":  "127.0.0.1",
+    "port":  "dnatPort",
+    "dport": "dnatDport", "dportValue": "22"
+  }
+}
+```
+
+## Action `action_dnatReplace`
+
+Action `action_dnatReplace` should be inserted whenever there is a chance that the values stored in a log entry’s fields for a client IP address (and possibly the port as well) are those of a proxy instead.
+
+Properties `addr`, `port`, `daddr`, and `dport` are used to match against a correspondance currently held in memory; at least one of these properties must be given.
+Each property corresponds to the name of a log entry field in which to read the corresponding value.
+
+Properties `saddrInto` and `sportInto` indicate the log entry fields in which to store the corrected source IP address or port; at least one of those properties must be given.
+
+For example, consider the following (simplified) log entries:
+
+```json
+{ '_HOSTNAME': 'dmz',
+  'SYSLOG_IDENTIFIER': 'kernel',
+  'MESSAGE': 'DNAT/ssh: … SRC=12.34.56.78 DST=10.0.0.1 … SPT=43210 …'
+}
+{ '_HOSTNAME': 'sshserv',
+  'SYSLOG_IDENTIFIER': 'sshd',
+  '_SYSTEMD_UNIT': 'sshd.service',
+  'MESSAGE': 'Failed password for ME from 10.0.0.1 port 43210 ssh2'
+}
+{ '_HOSTNAME': 'dmz',
+  'SYSLOG_IDENTIFIER': 'sshd',
+  '_SYSTEMD_UNIT': 'sshd.service',
+  'MESSAGE': 'Failed password for KeyUser from 87.65.43.21 port 24680 ssh2'
+}
+```
+
+Assuming the first log entry is correctly handled by `action_dnatCapture`, a good configuration to handle SSH failed logins could be:
+
+```json
+{ "filter": "filter_equals",
+  "args": { "field": "SYSLOG_IDENTIFIER", "value": "sshd" }
+},
+{ "filter": "filter_pcre",
+  "args": {
+    "field": "MESSAGE",
+    "re": "^Failed password for (.*) from ([^ ]+) port ([^ ]+) ssh2$",
+    "save": [ "sshUser", "clientIP", "clientPort" ]
+},
+{ "action": "action_dnatReplace",
+  "args": { "addr": "clientIP", "port": "clientPort", "saddrInto": "clientIP" }
+},
+{ "action": "action_email",
+  "args": { "message": "SSH attack from {clientIP} on {sshUser}@{_HOSTNAME}." }
+}
+```
+
+As a result, two emails would be sent, with these messages:
+
+```
+SSH attack from 12.34.56.78 on ME@sshserv.
+SSH attack from 87.65.43.21 on KeyUser@dmz.
+```
--- a/doc/install.md
+++ b/doc/install.md
@ -5,7 +5,7 @@ The software requirements are:
 * a modern systemd-based Linux operating system (eg. [Archlinux](https://archlinux.org/)- or [Fedora](https://getfedora.org/)-based distributions);
 * python, at least version 3.4 (or [more, depending on the modules](intro_tech.md) being used);
 * [python-systemd](https://www.freedesktop.org/software/systemd/python-systemd/journal.html);
-* [nftables](http://wiki.nftables.org/) _if_ IP address bans are to be managed;
+* [nftables](http://wiki.nftables.org/) or [ipset](http://ipset.netfilter.org/) _if_ IP address bans are to be managed;
 * a sendmail-like program _if_ emails are wanted.

 Besides, getting the software requires [Git](http://git-scm.com/), and packaging it requires [python-setuptools](http://pypi.python.org/pypi/setuptools).
@ -45,4 +45,6 @@ To install Pyruse on the system, run these commands as root, in the root directo
 The `package` line is the one that actually alters the system. Until Pyruse is packaged for your operating system, you may want to change this line to `checkinstall package`. [Checkinstall](https://en.wikipedia.org/wiki/CheckInstall) should be able to turn your Pyruse installation into a native Linux package.

 Then, to run Pyruse, start (and enable) `pyruse.service`.
+
 If you use nftables bans, you should also start (and enable) `pyruse-boot@action_nftBan.service`.
+Likewise, if you use ipset bans, you should start (and enable) `pyruse-boot@action_ipsetBan.service`.
--- a/doc/intro_func.md
+++ b/doc/intro_func.md
@ -2,7 +2,7 @@

 Everyone knows [Fail2ban](http://www.fail2ban.org/).
 This program is excellent at what it does, which is matching patterns in a number of log files, and keeping track of counters in order to launch actions (ban an IP, send an email…) when thresholds are crossed.
-[Fail2ban is extremely customizable](http://yalis.fr/cms/index.php/post/2014/11/02/Migrate-from-DenyHosts-to-Fail2ban)… to a point; and to my knowledge it cannot benefit from all the features of modern-day logging with systemd.
+[Fail2ban is extremely customizable](http://yalis.fr/cms/index.php/post/2014/11/02/Migrate-from-DenyHosts-to-Fail2ban)… to a point; to my knowledge it cannot benefit from all the features of modern-day logging with systemd.

 Then, there is the less-known and aging [Epylog](http://freshmeat.sourceforge.net/projects/epylog/); several programs exist with the same features.
 Just like Fail2ban, this program continuously scans the logs to do pattern matching.
@ -38,6 +38,7 @@ The most interesting [filtering or informational entries](https://www.freedeskto

 * `PRIORITY`: see [Syslog at Wikipedia](https://en.wikipedia.org/wiki/Syslog#Severity_level) for the definitions
 * `SYSLOG_FACILITY`: see [Syslog at Wikipedia](https://en.wikipedia.org/wiki/Syslog#Facility) for the definitions
+* `SYSLOG_IDENTIFIER`: short name for the program that produced the log entry (better accuracy than `_SYSTEMD_UNIT`)
 * `_HOSTNAME`: short hostname of the machine where the log entry occurred
 * `_UID`: user ID of the systemd service that produced the log entry
 * `_GID`: group ID of the systemd service that produced the log entry
--- a/doc/intro_tech.md
+++ b/doc/intro_tech.md
@ -31,8 +31,10 @@ Pyruse is split into several Python files:
 * `workflow.py`: This is where the configuration file get translated into actual execution chains, linked together into a single static workflow.
 * `module.py`: Whenever the workflow needs to add a filter or an action to an execution chain, this module finds it in the filesystem.
 * `base.py`: All actions and filters inherit from the `Action` and `Filter` classes defined in there; they act as an abstraction layer between the workflow and the modules.
-* `counter.py`: This utility class is parent to modules that manage a counter (Python supports multiple-inheritance).
-* `email.py`: This utility class is parent to modules that send emails (Python supports multiple-inheritance).
+* `ban.py`: This utility class is parent to modules that ban IP addresses using [Netfilter](https://netfilter.org/) (Python supports multiple-inheritance).
+* `counter.py`: This utility class is parent to modules that manage a counter.
+* `dnat.py`: This file contains utility parent classes for actions that try and restore the actual client IP addresses.
+* `email.py`: This utility class is parent to modules that send emails.

 All else is actions and filters…
 Some are delivered with Pyruse itself; [more can be added](customize.md).
--- a/doc/logandban.md
+++ b/doc/logandban.md
@ -30,9 +30,22 @@ Here are some examples:
 ## Ban IP addresses after they misbehaved

 Linux provides a number of firewall solutions: [iptables](http://www.netfilter.org/), its successor [nftables](http://wiki.nftables.org/), and many iptables frontends like [Shorewall](http://www.shorewall.net/) or RedHat’s [firewalld](http://www.firewalld.org/).
-For Pyruse, **nftables** was chosen, because it is modern and light-weight, and provides interesting features.

-Action `action_nftBan` takes three mandatory arguments:
+For Pyruse, **nftables** was initially chosen, because it is modern and light-weight, and provides interesting features. Besides, only nftables is actually tested in real use.  
+Now, an **ipset**-based [alternative is also available](#iptables-with-ipset).
+
+### nftables
+
+Action `action_nftBan` requires that nftables is installed.
+In addition, the binary to run (and parameters if needed) should be set in the configuration file; here is the default value:
+
+```json
+"nftBan": {
+  "nft": [ "/usr/bin/nft" ]
+}
+```
+
+This action takes three mandatory arguments:

 * `nftSetIPv4` and `nftSetIPv6` are the nftables sets where IP addresses will be added. The IP address is considered to be IPv6 if it contains at least one colon (`:`) character, else it is supposed to be IPv4.
 * `IP` gives the name of the entry field to read, in order to get the IP address.
@ -69,7 +82,7 @@ table ip6 Inet6 {
 }
 ```

-Then the values for `nftSetIPv4` and `nftSetIPv6` will be respectively “`Inet4 mail_ban`” and “`Inet6 mail_ban`”.
+Then the values for `nftSetIPv4` and `nftSetIPv6` will be respectively “`ip Inet4 mail_ban`” and “`ip6 Inet6 mail_ban`”.

 Optionally, a number may be specified with `banSeconds` to limit the time this ban will last.
 The nice thing with nftables, is that it handles the timeouts itself: no need to keep track of the active bans and remove them using a Python program; the only reason why bans are recorded in a file, is to be able to restore them on reboot.
@ -79,21 +92,21 @@ Here are examples:
 ```json
 {
  "action": "action_nftBan",
-  "args": { "IP": "thatIP", "banSeconds": 86400, "nftSetIPv4": "Inet4 mail_ban", "nftSetIPv6": "Inet6 mail_ban" }
+  "args": { "IP": "thatIP", "banSeconds": 86400, "nftSetIPv4": "ip Inet4 mail_ban", "nftSetIPv6": "ip6 Inet6 mail_ban" }
 }

 {
  "action": "action_nftBan",
-  "args": { "IP": "thatIP", "nftSetIPv4": "Inet4 sshd_ban", "nftSetIPv6": "Inet6 sshd_ban" }
+  "args": { "IP": "thatIP", "nftSetIPv4": "ip Inet4 sshd_ban", "nftSetIPv6": "ip6 Inet6 sshd_ban" }
 }
 ```

-### List the currently banned addresses
+#### List the currently banned addresses

 To see what IP addresses are currently banned, here is the `nft` command:

 ```bash
-$ sudo nft 'list set Inet4 mail_ban'
+$ sudo nft 'list set ip Inet4 mail_ban'
 table ip Inet4 {
        set mail_ban {
                type ipv4_addr
@ -114,17 +127,124 @@ table ip Inet4 {

 _Note_: The un-rounded timeouts are post-reboot restored bans.

-### Un-ban an IP address
+#### Un-ban an IP address

 It is bound to happen some day: you will want to un-ban a banned IP address.

 Since `action_nftBan` does not keep the current bans in memory, it is enough to remove the ban using the `nft` command:

 ```bash
-$ sudo nft 'delete element Inet4 mail_ban {10.0.0.10}'
+$ sudo nft 'delete element ip Inet4 mail_ban {10.0.0.10}'
 ```

 However, the ban may be restored when restarting Pyruse.
 To avoid that, also delete the corresponding record from the `action_nftBan.py.json` file in Pyruse’s [storage directory](conffile.md).

 To go further, you could tweak your configuration, so that your trusted IP addresses never reach `action_nftBan`.
+
+#### Manual ban of an IP address
+
+To add a ban yourself, run a command like this:
+
+```bash
+$ sudo nft 'add element ip Inet4 ssh_ban {192.168.1.1 timeout 5d}
+```
+
+The `timeout …` part can be omitted to add a permanent ban. The timeout can be any combination of days (`d`), hours (`h`), minutes (`m`), and seconds (`s`), eg. “`3d31m16s`”.
+
+In order to make the ban persistent across reboots, a corresponding record should also be appended to the `action_nftBan.py.json` file in Pyruse’s [storage directory](conffile.md) (the IP address, Nftables Set, days, hours, minutes, seconds, and actual path to the file should be adapted to your needs):
+
+* either a time-limited ban:
+
+```bash
+$ sudo sed -i "\$s/.\$/$(date +', {"IP": "192.168.1.1", "nfSet": "ip Inet4 ssh_ban", "timestamp": %s.000000}' -d 'now +3day +31minute +16second')]/" /var/lib/pyruse/action_nftBan.py.json
+```
+
+* or an unlimited ban:
+
+```bash
+$ sudo sed -i '$s/.$/, {"IP": "192.168.1.1", "nfSet": "ip Inet4 ssh_ban", "timestamp": 0}]/' /var/lib/pyruse/action_nftBan.py.json
+```
+
+### iptables with ipset
+
+Action `action_ipsetBan` requires that ipset and iptables are installed.
+In addition, the ipset binary to run (and parameters if needed) should be set in the configuration file; here is the default value:
+
+```json
+	"ipsetBan": {
+		"ipset": [ "/usr/bin/ipset", "-exist", "-quiet" ]
+	}
+```
+
+This action works exactly [like `action_nftBan`](#nftables), except parameters `nftSetIPv4` and `nftSetIPv6` are named `ipSetIPv4` and `ipSetIPv6` instead.
+
+The name of the set in the `ipSetIPv4` parameter must have been created before running Pyruse, with:
+
+```bash
+$ sudo ipset create SET_NAME hash:ip family inet hashsize 1024 maxelem 65535
+```
+
+Likewise, the set given by `ipSetIPv6` must have been created before running Pyruse, with:
+
+```bash
+$ sudo ipset create SET_NAME hash:ip family inet6 hashsize 1024 maxelem 65535
+```
+
+Here are examples of usage for this action:
+
+```json
+{
+  "action": "action_ipsetBan",
+  "args": { "IP": "thatIP", "banSeconds": 86400, "ipSetIPv4": "mail_ban4", "ipSetIPv6": "mail_ban6" }
+}
+
+{
+  "action": "action_ipsetBan",
+  "args": { "IP": "thatIP", "ipSetIPv4": "sshd_ban4", "ipSetIPv6": "sshd_ban6" }
+}
+```
+
+#### List the currently banned addresses
+
+To see what IP addresses are currently banned, here is the `ipset` command:
+
+```bash
+$ sudo ipset list mail_ban4'
+```
+
+#### Un-ban an IP address
+
+To remove an IP address from a set, here is the `ipset` command:
+
+```bash
+$ sudo ipset del mail_ban4 10.0.0.10'
+```
+
+To make the change persistent across reboots, also delete the corresponding record from the `action_ipsetBan.py.json` file in Pyruse’s [storage directory](conffile.md).
+
+To go further, you could tweak your configuration, so that your trusted IP addresses never reach `action_ipsetBan`.
+
+#### Manual ban of an IP address
+
+To add a ban yourself, run a command like this:
+
+```bash
+$ sudo ipset add ssh_ban4 192.168.1.1 timeout 261076
+```
+
+The `timeout …` part can be omitted to add a permanent ban; otherwise it is a number of seconds.
+
+In order to make the ban persistent across reboots, a corresponding record should also be appended to the `action_ipsetBan.py.json` file in Pyruse’s [storage directory](conffile.md) (the IP address, Nftables Set, days, hours, minutes, seconds, and actual path to the file should be adapted to your needs):
+
+* either a time-limited ban:
+
+```bash
+$ sudo sed -i "\$s/.\$/$(date +', {"IP": "192.168.1.1", "nfSet": "ssh_ban4", "timestamp": %s.000000}' -d 'now +3day +31minute +16second')]/" /var/lib/pyruse/action_ipsetBan.py.json
+```
+
+* or an unlimited ban:
+
+```bash
+$ sudo sed -i '$s/.$/, {"IP": "192.168.1.1", "nfSet": "ssh_ban4", "timestamp": 0}]/' /var/lib/pyruse/action_ipsetBan.py.json
+```
--- a/extra/examples/full_pyruse.json
+++ b/extra/examples/full_pyruse.json
--- a/extra/examples/get-systemd-stats.sh
+++ b/extra/examples/get-systemd-stats.sh
@ -1,11 +1,15 @@
 #!/bin/bash
-# $1 (optional): systemd-nspawn machine name
+# $1 (optional): grouping criteria: SYSLOG_IDENTIFIER (default) or _SYSTEMD_UNIT
+# $+ (optional): journalctl options (-M machine, -S date…)
+
+CRIT=${1:-SYSLOG_IDENTIFIER}
+shift

 {
-	printf 'Units\tTotal\tP7\tP6\tP5\tP4\tP3\tP2\tP1\tP0\n'
-	sudo journalctl ${1:+-M "$1"} -o json-pretty --output-fields=_SYSTEMD_UNIT,PRIORITY \
+	printf '%s\tTotal\tP7\tP6\tP5\tP4\tP3\tP2\tP1\tP0\n' "$CRIT"
+	sudo journalctl "$@" -o json-pretty --output-fields="${CRIT}",PRIORITY \
 	| tr -d $'"\t, ' \
-	| awk -F: -vOFS=: '
+	| awk -F: -vOFS=: -vCRIT="$CRIT" '
 		/^\{/ {
 			u = ""
 			p = -1
@ -13,7 +17,7 @@
 		$1 == "PRIORITY" {
 			p = $2
 		}
-		$1 == "_SYSTEMD_UNIT" {
+		$1 == CRIT {
 			u = gensub(\
 				"@.*(\\.[^.]*)$",\
 				"@*\\1",\
--- a/extra/systemd/action_ipsetBan.conf
+++ b/extra/systemd/action_ipsetBan.conf
@ -0,0 +1,3 @@
+[Unit]
+Requires=iptables.service
+After=iptables.service
--- a/pyruse/actions/action_dnatCapture.py
+++ b/pyruse/actions/action_dnatCapture.py
@ -0,0 +1,18 @@
+# pyruse is intended as a replacement to both fail2ban and epylog
+# Copyright © 2017–2018 Y. Gablin
+# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
+from pyruse import base, dnat
+
+class Action(base.Action, dnat.Mapper):
+    def __init__(self, args):
+        base.Action.__init__(self)
+        sa = (args    ["saddr"],       None                        )
+        sp = (args.get("sport", None), None                        )
+        a  = (args.get("addr",  None), args.get("addrValue",  None))
+        p  = (args.get("port",  None), args.get("portValue",  None))
+        da = (args.get("daddr", None), args.get("daddrValue", None))
+        dp = (args.get("dport", None), args.get("dportValue", None))
+        dnat.Mapper.__init__(self, sa, sp, a, p, da, dp, args.get("keepSeconds", 63))
+
+    def act(self, entry):
+        self.map(entry)
--- a/pyruse/actions/action_dnatReplace.py
+++ b/pyruse/actions/action_dnatReplace.py
@ -0,0 +1,18 @@
+# pyruse is intended as a replacement to both fail2ban and epylog
+# Copyright © 2017–2018 Y. Gablin
+# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
+from pyruse import base, dnat
+
+class Action(base.Action, dnat.Matcher):
+    def __init__(self, args):
+        base.Action.__init__(self)
+        sa = args    ["saddrInto"]
+        sp = args.get("sportInto", None)
+        a  = args.get("addr",  None)
+        p  = args.get("port",  None)
+        da = args.get("daddr", None)
+        dp = args.get("dport", None)
+        dnat.Matcher.__init__(self, a, p, da, dp, sa, sp)
+
+    def act(self, entry):
+        self.replace(entry)
--- a/pyruse/actions/action_ipsetBan.py
+++ b/pyruse/actions/action_ipsetBan.py
@ -0,0 +1,37 @@
+# pyruse is intended as a replacement to both fail2ban and epylog
+# Copyright © 2017–2018 Y. Gablin
+# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
+import os
+import subprocess
+from pyruse import ban, base, config
+
+class Action(base.Action, ban.NetfilterBan):
+    _storage = config.Config().asMap().get("storage", "/var/lib/pyruse") \
+        + "/" + os.path.basename(__file__) + ".json"
+    _ipset = config.Config().asMap().get("ipsetBan", {}).get("ipset", ["/usr/bin/ipset", "-exist", "-quiet"])
+
+    def __init__(self, args):
+        base.Action.__init__(self)
+        ban.NetfilterBan.__init__(self, Action._storage)
+        if args is None:
+            return # on-boot configuration
+        ipv4Set = args["ipSetIPv4"]
+        ipv6Set = args["ipSetIPv6"]
+        field = args["IP"]
+        banSeconds = args.get("banSeconds", None)
+        self.initSelf(ipv4Set, ipv6Set, field, banSeconds)
+
+    def act(self, entry):
+        ban.NetfilterBan.act(self, entry)
+
+    def setBan(self, nfSet, ip, seconds):
+        cmd = list(Action._ipset)
+        cmd.extend(["add", nfSet, ip])
+        if seconds > 0:
+            cmd.extend(["timeout", str(seconds)])
+        subprocess.run(cmd)
+
+    def cancelBan(self, nfSet, ip):
+        cmd = list(Action._ipset)
+        cmd.extend(["del", nfSet, ip])
+        subprocess.run(cmd)
--- a/pyruse/actions/action_nftBan.py
+++ b/pyruse/actions/action_nftBan.py
@ -1,97 +1,39 @@
 # pyruse is intended as a replacement to both fail2ban and epylog
 # Copyright © 2017–2018 Y. Gablin
 # Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
-import datetime
-import json
 import os
 import subprocess
-from pyruse import base, config
+from pyruse import ban, base, config

-class Action(base.Action):
+class Action(base.Action, ban.NetfilterBan):
    _storage = config.Config().asMap().get("storage", "/var/lib/pyruse") \
        + "/" + os.path.basename(__file__) + ".json"
    _nft = config.Config().asMap().get("nftBan", {}).get("nft", ["/usr/bin/nft"])

    def __init__(self, args):
-        super().__init__()
+        base.Action.__init__(self)
+        ban.NetfilterBan.__init__(self, Action._storage)
        if args is None:
            return # on-boot configuration
-        self.ipv4Set = args["nftSetIPv4"]
-        self.ipv6Set = args["nftSetIPv6"]
-        self.field = args["IP"]
-        self.banSeconds = args.get("banSeconds", None)
+        ipv4Set = args["nftSetIPv4"]
+        ipv6Set = args["nftSetIPv6"]
+        field = args["IP"]
+        banSeconds = args.get("banSeconds", None)
+        self.initSelf(ipv4Set, ipv6Set, field, banSeconds)

    def act(self, entry):
-        ip = entry[self.field]
-        nftSet = self.ipv6Set if ":" in ip else self.ipv4Set
-        newBan = {"IP": ip, "nftSet": nftSet}
+        ban.NetfilterBan.act(self, entry)

-        now = datetime.datetime.utcnow()
-        bans = []
-        previousTS = None
-        try:
-            with open(Action._storage) as dataFile:
-                for ban in json.load(dataFile):
-                    if ban["timestamp"] > 0 and ban["timestamp"] <= now.timestamp():
-                        continue
-                    elif {k: ban[k] for k in newBan.keys()} == newBan:
-                        # should not happen, since the IP is banned…
-                        previousTS = ban["timestamp"]
-                    else:
-                        bans.append(ban)
-        except IOError:
-            pass # new file
-
-        if previousTS is not None:
-            try:
-                cmd = list(Action._nft)
-                cmd.append("delete element %s {%s}" % (nftSet, ip))
-                subprocess.run(cmd)
-            except Exception:
-                pass # too late: not a problem
-
-        if self.banSeconds:
-            until = now + datetime.timedelta(seconds = self.banSeconds)
-            newBan["timestamp"] = until.timestamp()
-            timeout = self.banSeconds
-        else:
-            newBan["timestamp"] = 0
-            timeout = 0
-
-        self._doBan(timeout, ip, nftSet)
-        bans.append(newBan)
-        with open(Action._storage, "w") as dataFile:
-            json.dump(bans, dataFile)
-
-    def boot(self):
-        now = datetime.datetime.utcnow()
-        bans = []
-        try:
-            with open(Action._storage) as dataFile:
-                for ban in json.load(dataFile):
-                    if ban["timestamp"] == 0:
-                        self._doBan(0, ban["IP"], ban["nftSet"])
-                        bans.append(ban)
-                    elif ban["timestamp"] <= now.timestamp():
-                        continue
-                    else:
-                        until = datetime.datetime.utcfromtimestamp(ban["timestamp"])
-                        timeout = (until - now).total_seconds()
-                        self._doBan(int(timeout), ban["IP"], ban["nftSet"])
-                        bans.append(ban)
-        except IOError:
-            pass # no file
-
-        with open(Action._storage, "w") as dataFile:
-            json.dump(bans, dataFile)
-
-    def _doBan(self, seconds, ip, nftSet):
-        if seconds < 0:
-            return # can happen when the threshold is crossed while computing the duration
+    def setBan(self, nfSet, ip, seconds):
        if seconds == 0:
            timeout = ""
        else:
            timeout = " timeout %ss" % seconds
        cmd = list(Action._nft)
-        cmd.append("add element %s {%s%s}" % (nftSet, ip, timeout))
+        cmd.append("add element %s {%s%s}" % (nfSet, ip, timeout))
+        subprocess.run(cmd)
+
+    def cancelBan(self, nfSet, ip):
+        cmd = list(Action._nft)
+        cmd.append("delete element %s {%s}" % (nfSet, ip))
        subprocess.run(cmd)
--- a/pyruse/ban.py
+++ b/pyruse/ban.py
@ -0,0 +1,85 @@
+# pyruse is intended as a replacement to both fail2ban and epylog
+# Copyright © 2017–2018 Y. Gablin
+# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
+import abc
+import datetime
+import json
+
+class NetfilterBan(abc.ABC):
+    def __init__(self, storage):
+        self.storage = storage
+
+    def initSelf(self, ipv4Set, ipv6Set, ipField, banSeconds):
+        self.ipv4Set = ipv4Set
+        self.ipv6Set = ipv6Set
+        self.field = ipField
+        self.banSeconds = banSeconds
+
+    @abc.abstractmethod
+    def setBan(self, nfSet, ip, seconds):
+        pass
+
+    @abc.abstractmethod
+    def cancelBan(self, nfSet, ip):
+        pass
+
+    def act(self, entry):
+        ip = entry[self.field]
+        nfSet = self.ipv6Set if ":" in ip else self.ipv4Set
+        newBan = {"IP": ip, "nfSet": nfSet}
+
+        now = datetime.datetime.utcnow()
+        bans = []
+        previousTS = None
+        try:
+            with open(self.storage) as dataFile:
+                for ban in json.load(dataFile):
+                    if ban["timestamp"] > 0 and ban["timestamp"] <= now.timestamp():
+                        continue
+                    elif {k: ban[k] for k in newBan.keys()} == newBan:
+                        # should not happen, since the IP is banned…
+                        previousTS = ban["timestamp"]
+                    else:
+                        bans.append(ban)
+        except IOError:
+            pass # new file
+
+        if previousTS is not None:
+            try:
+                self.cancelBan(nfSet, ip)
+            except Exception:
+                pass # too late: not a problem
+
+        if self.banSeconds:
+            until = now + datetime.timedelta(seconds = self.banSeconds)
+            newBan["timestamp"] = until.timestamp()
+            timeout = self.banSeconds
+        else:
+            newBan["timestamp"] = 0
+            timeout = 0
+
+        self.setBan(nfSet, ip, timeout)
+        bans.append(newBan)
+        with open(self.storage, "w") as dataFile:
+            json.dump(bans, dataFile)
+
+    def boot(self):
+        now = int(datetime.datetime.utcnow().timestamp())
+        bans = []
+        try:
+            with open(self.storage) as dataFile:
+                for ban in json.load(dataFile):
+                    if ban["timestamp"] == 0:
+                        self.setBan(ban["nfSet"], ban["IP"], 0)
+                        bans.append(ban)
+                    elif int(ban["timestamp"]) <= now:
+                        continue
+                    else:
+                        timeout = int(ban["timestamp"]) - now
+                        self.setBan(ban["nfSet"], ban["IP"], timeout)
+                        bans.append(ban)
+        except IOError:
+            pass # no file
+
+        with open(self.storage, "w") as dataFile:
+            json.dump(bans, dataFile)
--- a/pyruse/dnat.py
+++ b/pyruse/dnat.py
@ -0,0 +1,97 @@
+# pyruse is intended as a replacement to both fail2ban and epylog
+# Copyright © 2017–2018 Y. Gablin
+# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
+from datetime import datetime
+
+_mappings = []
+
+def _cleanMappings():
+    global _mappings
+    now = int(datetime.today().timestamp())
+    _mappings = [m for m in _mappings if (now >> m["bits"]) <= m["time"]]
+
+def putMapping(mapping):
+    global _mappings
+    _cleanMappings()
+    _mappings.append(mapping)
+
+def getMappings():
+    global _mappings
+    _cleanMappings()
+    return _mappings
+
+def periodBits(keepSeconds):
+    seconds, bits = keepSeconds, 0
+    while seconds:
+        bits += 1
+        seconds = seconds >> 1
+    return bits # number of significant bits in keepSeconds
+
+def valueFor(spec, entry):
+    return spec[1] if spec[0] is None else entry.get(spec[0], spec[1])
+
+class Mapper():
+    def __init__(self, saddr, sport, addr, port, daddr, dport, keepSeconds):
+        for spec in [saddr, addr]:
+            if spec[0] is None and spec[1] is None:
+                raise ValueError("Neither field nor value was specified for address")
+        self.saddr = saddr
+        self.sport = sport
+        self.addr  = addr
+        self.port  = port
+        self.daddr = daddr
+        self.dport = dport
+        self.keepBits = periodBits(keepSeconds)
+
+    def map(self, entry):
+        saddr = valueFor(self.saddr, entry)
+        addr  = valueFor(self.addr,  entry)
+        if saddr is None or addr is None:
+            return
+        sport = valueFor(self.sport, entry)
+        port  = valueFor(self.port,  entry)
+        daddr = valueFor(self.daddr, entry)
+        dport = valueFor(self.dport, entry)
+        putMapping(dict(
+            bits  = self.keepBits,
+            time  = 1 + (int(entry["__REALTIME_TIMESTAMP"].timestamp()) >> self.keepBits),
+            saddr = saddr, sport = sport,
+            addr  = addr,  port  = port,
+            daddr = daddr, dport = dport
+        ))
+
+class Matcher():
+    def __init__(self, addr, port, daddr, dport, saddr, sport):
+        if addr is None and port is None and daddr is None and dport is None:
+            raise ValueError("No field was provided on which to do the matching")
+        if saddr is None and sport is None:
+            raise ValueError("No field was provided in which to store the translated values")
+        matchers = []
+        updaters = []
+        if addr is not None:
+            matchers.append((addr, "addr"))
+        if port is not None:
+            matchers.append((port, "port"))
+        if daddr is not None:
+            matchers.append((daddr, "daddr"))
+        if dport is not None:
+            matchers.append((dport, "dport"))
+        if saddr is not None:
+            updaters.append((saddr, "saddr"))
+        if sport is not None:
+            updaters.append((sport, "sport"))
+        self.matchers = matchers
+        self.updaters = updaters
+
+    def replace(self, entry):
+        for field, _void in self.matchers:
+            if field not in entry:
+                return
+        for mapping in getMappings():
+            for field, mapEntry in self.matchers:
+                if entry[field] != mapping[mapEntry]:
+                    break
+            else:
+                for field, mapEntry in self.updaters:
+                    entry[field] = mapping[mapEntry]
+                return
--- a/tests/action_dnatCapture.py
+++ b/tests/action_dnatCapture.py
@ -0,0 +1,93 @@
+# pyruse is intended as a replacement to both fail2ban and epylog
+# Copyright © 2017–2018 Y. Gablin
+# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
+from datetime import datetime
+from pyruse import dnat
+from pyruse.actions.action_dnatCapture import Action
+
+def whenNoSaddrThenError():
+    try:
+        Action(dict(addr=1))
+    except Exception:
+        return
+    assert False, "An exception should be raised when saddr is absent"
+
+def whenNoAddrNorAddrvalueThenError():
+    try:
+        Action(dict(saddr=1))
+    except Exception:
+        return
+    assert False, "An exception should be raised when addr and addrValue are absent"
+
+def whenNoAddrButAddrvalueThenNoError():
+    Action(dict(saddr=1, addrValue=1))
+
+def whenNoAddrvalueButAddrThenNoError():
+    Action(dict(saddr=1, addr=1))
+
+def whenNoKeepsecondsThen6bits():
+    a = Action(dict(saddr=1, addr=1))
+    assert a.keepBits == 6, "Default keepSeconds (63) should be on 6 bits, not " + str(a.keepBits)
+
+def whenKeepsecondsIs150Then8bits():
+    a = Action(dict(saddr=1, addr=1, keepSeconds=150))
+    assert a.keepBits == 8, "150 for keepSeconds should be on 8 bits, not " + str(a.keepBits)
+
+def whenInsufficientEntryThenNoMapping():
+    dnat._mappings = []
+    Action({"saddr": "sa", "addrValue": "x"}).act({"__REALTIME_TIMESTAMP": datetime(2018,1,1)})
+    assert dnat._mappings == [], "Got:\n" + str(dnat._mappings) + "\ninstead of []"
+
+def whenFieldAndOrValueThenCheckMapping(spec, entryWithAddr, entryWithDAddr, expect):
+    dnat._mappings = []
+
+    # specify the Action
+    spec.update({"saddr": "sa"})
+
+    # prepare the entry
+    entry = {
+        "__REALTIME_TIMESTAMP": datetime(2018,1,1),
+        "sa": "vsa", "sp": "vsp"}
+    if entryWithAddr:
+        entry.update({"a":  "va",  "p":  "vp"})
+    if entryWithDAddr:
+        entry.update({"da": "vda", "dp": "vdp"})
+
+    # run
+    Action(spec).act(entry)
+
+    # check the result
+    expect.update({"bits": 6, "time": 23668144, "saddr": "vsa"})
+    assert dnat._mappings == [expect], "Got:\n" + str(dnat._mappings) + "\ninstead of:\n" + str([expect])
+
+def unitTests():
+    whenNoSaddrThenError()
+    whenNoAddrNorAddrvalueThenError()
+    whenNoAddrButAddrvalueThenNoError()
+    whenNoAddrvalueButAddrThenNoError()
+    whenNoKeepsecondsThen6bits()
+    whenKeepsecondsIs150Then8bits()
+    whenInsufficientEntryThenNoMapping()
+
+    whenFieldAndOrValueThenCheckMapping({"addr": "a"}, True, True,
+        {"sport": None, "addr": "va", "port": None, "daddr": None, "dport": None})
+    whenFieldAndOrValueThenCheckMapping({"addrValue": "x"}, True, True,
+        {"sport": None, "addr": "x", "port": None, "daddr": None, "dport": None})
+    whenFieldAndOrValueThenCheckMapping({"addr": "a", "addrValue": "x"}, True, True,
+        {"sport": None, "addr": "va", "port": None, "daddr": None, "dport": None})
+    whenFieldAndOrValueThenCheckMapping({"addr": "a", "addrValue": "x"}, False, True,
+        {"sport": None, "addr": "x", "port": None, "daddr": None, "dport": None})
+
+    whenFieldAndOrValueThenCheckMapping({"addr": "a", "daddr": "da"}, True, True,
+        {"sport": None, "addr": "va", "port": None, "daddr": "vda", "dport": None})
+    whenFieldAndOrValueThenCheckMapping({"addr": "a", "daddrValue": "x"}, True, True,
+        {"sport": None, "addr": "va", "port": None, "daddr": "x", "dport": None})
+    whenFieldAndOrValueThenCheckMapping({"addr": "a", "daddr": "da", "daddrValue": "x"}, True, True,
+        {"sport": None, "addr": "va", "port": None, "daddr": "vda", "dport": None})
+    whenFieldAndOrValueThenCheckMapping({"addr": "a", "daddr": "da", "daddrValue": "x"}, True, False,
+        {"sport": None, "addr": "va", "port": None, "daddr": "x", "dport": None})
+
+    whenFieldAndOrValueThenCheckMapping({"addr": "a", "port": "p"}, True, True,
+        {"sport": None, "addr": "va", "port": "vp", "daddr": None, "dport": None})
+    whenFieldAndOrValueThenCheckMapping({"addr": "a", "dport": "dp"}, True, True,
+        {"sport": None, "addr": "va", "port": None, "daddr": None, "dport": "vdp"})
--- a/tests/action_dnatReplace.py
+++ b/tests/action_dnatReplace.py
@ -0,0 +1,74 @@
+# pyruse is intended as a replacement to both fail2ban and epylog
+# Copyright © 2017–2018 Y. Gablin
+# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
+from datetime import datetime
+from pyruse import dnat
+from pyruse.actions.action_dnatReplace import Action
+
+def whenNoSaddrintoThenError():
+    try:
+        Action(dict(addr=1))
+    except Exception:
+        return
+    assert False, "An exception should be raised when saddrInto is absent"
+
+def whenNoMatchFieldThenError():
+    try:
+        Action(dict(saddrInto=1))
+    except Exception:
+        return
+    assert False, "An exception should be raised when no match-field is present"
+
+def whenSaddrintoAndAtLeastOneMatchFieldThenNoError():
+    a = Action(dict(saddrInto=1, dport=1))
+    assert a.matchers == [(1, "dport")], "Got:\n" + str(a.matchers) + "\ninstead of:\n" + str([(1, "dport")])
+    assert a.updaters == [(1, "saddr")], "Got:\n" + str(a.updaters) + "\ninstead of:\n" + str([(1, "saddr")])
+
+def whenNoMatchingEntryThenNoChange():
+    dnat._mappings = [{
+        "bits": 7, "time": 1183407200,
+        "saddr": "bad",  "sport": None,
+        "addr": "prox",  "port": 12345,
+        "daddr": "serv", "dport": None}]
+    a = Action(dict(saddrInto="sa", port="sp"))
+
+    entryIn = dict(sa = "prox", da = "serv")
+    entryOut = entryIn.copy()
+    a.act(entryOut)
+    assert entryIn == entryOut, "Got:\n" + str(entryOut) + "\ninstead of:\n" + str(entryIn)
+
+def whenNoMatchingValueThenNoChange():
+    dnat._mappings = [{
+        "bits": 7, "time": 1183407200,
+        "saddr": "bad",  "sport": None,
+        "addr": "prox",  "port": 12345,
+        "daddr": "serv", "dport": None}]
+    a = Action(dict(saddrInto="sa", port="sp"))
+
+    entryIn = dict(sa = "prox", sp = 1234, da = "serv")
+    entryOut = entryIn.copy()
+    a.act(entryOut)
+    assert entryIn == entryOut, "Got:\n" + str(entryOut) + "\ninstead of:\n" + str(entryIn)
+
+def whenMatchingEntryThenChange():
+    dnat._mappings = [{
+        "bits": 7, "time": 1183407200,
+        "saddr": "bad",  "sport": None,
+        "addr": "prox",  "port": 12345,
+        "daddr": "serv", "dport": None}]
+    a = Action(dict(saddrInto="sa", port="sp"))
+
+    entryIn = dict(sa = "prox", sp = 12345, da = "serv")
+    expect = entryIn.copy()
+    expect.update({"sa": "bad"})
+    entryOut = entryIn.copy()
+    a.act(entryOut)
+    assert expect == entryOut, "Got:\n" + str(entryOut) + "\ninstead of:\n" + str(expect)
+
+def unitTests():
+    whenNoSaddrintoThenError()
+    whenNoMatchFieldThenError()
+    whenSaddrintoAndAtLeastOneMatchFieldThenNoError()
+    whenNoMatchingEntryThenNoChange()
+    whenNoMatchingValueThenNoChange()
+    whenMatchingEntryThenChange()
--- a/tests/action_ipsetBan.py
+++ b/tests/action_ipsetBan.py
@ -0,0 +1,147 @@
+# pyruse is intended as a replacement to both fail2ban and epylog
+# Copyright © 2017–2018 Y. Gablin
+# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
+import json
+import os
+import time
+from pyruse.actions.action_ipsetBan import Action
+
+ipBanCmd = "ipsetBan.cmd"
+ipBanState = "action_ipsetBan.py.json"
+
+def _clean():
+    if os.path.exists(ipBanCmd):
+        os.remove(ipBanCmd)
+    if os.path.exists(ipBanState):
+        os.remove(ipBanState)
+
+def whenBanIPv4ThenAddToIPv4Set():
+    _clean()
+    Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban"}).act({"ip": "10.0.0.1"})
+    assert os.path.exists(ipBanCmd)
+    assert os.path.exists(ipBanState)
+    nbLines = 0
+    with open(ipBanCmd, "rt") as c:
+        for line in c:
+            assert line == "add I4ban 10.0.0.1\n", line
+            nbLines += 1
+    assert nbLines == 1, nbLines
+    nbBans = 0
+    with open(ipBanState) as s:
+        for ban in json.load(s):
+            assert ban["IP"] == "10.0.0.1" and ban["nfSet"] == "I4ban", str(ban)
+            nbBans += 1
+    assert nbBans == 1, nbBans
+    _clean()
+
+def whenBanIPv6ThenAddToIPv6Set():
+    _clean()
+    Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban"}).act({"ip": "::1"})
+    assert os.path.exists(ipBanCmd)
+    assert os.path.exists(ipBanState)
+    nbLines = 0
+    with open(ipBanCmd, "rt") as c:
+        for line in c:
+            assert line == "add I6ban ::1\n", line
+            nbLines += 1
+    assert nbLines == 1, nbLines
+    nbBans = 0
+    with open(ipBanState) as s:
+        for ban in json.load(s):
+            assert ban["IP"] == "::1" and ban["nfSet"] == "I6ban", str(ban)
+            nbBans += 1
+    assert nbBans == 1, nbBans
+    _clean()
+
+def whenBanTwoIPThenTwoLinesInState():
+    _clean()
+    action = Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban"})
+    action.act({"ip": "10.0.0.1"})
+    action.act({"ip": "::1"})
+    action.act({"ip": "10.0.0.1"})
+    assert os.path.exists(ipBanState)
+    nbBans = 0
+    with open(ipBanState) as s:
+        for ban in json.load(s):
+            if ban["IP"] == "10.0.0.1":
+                assert ban["nfSet"] == "I4ban", str(ban)
+            elif ban["IP"] == "::1":
+                assert ban["nfSet"] == "I6ban", str(ban)
+            else:
+                assert false, str(ban)
+            nbBans += 1
+    assert nbBans == 2, nbBans
+    _clean()
+
+def whenBanAnewThenNoDuplicate():
+    _clean()
+    action = Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban"})
+    action.act({"ip": "10.0.0.1"})
+    action.act({"ip": "10.0.0.1"})
+    assert os.path.exists(ipBanCmd)
+    assert os.path.exists(ipBanState)
+    lineCount = 0
+    with open(ipBanCmd, "rt") as c:
+        for line in c:
+            lineCount += 1
+            if lineCount == 1:
+                assert line == "add I4ban 10.0.0.1\n", line
+            elif lineCount == 2:
+                assert line == "del I4ban 10.0.0.1\n", line
+            elif lineCount == 3:
+                assert line == "add I4ban 10.0.0.1\n", line
+    assert lineCount == 3, lineCount
+    nbBans = 0
+    with open(ipBanState) as s:
+        for ban in json.load(s):
+            if ban["IP"] == "10.0.0.1":
+                assert ban["nfSet"] == "I4ban", str(ban)
+            nbBans += 1
+    assert nbBans == 1, nbBans
+    _clean()
+
+def whenFinishedBanThenAsIfNotThere():
+    _clean()
+    action = Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban", "banSeconds": 1})
+    action.act({"ip": "10.0.0.1"})
+    time.sleep(1)
+    action.act({"ip": "10.0.0.1"})
+    assert os.path.exists(ipBanCmd)
+    lineCount = 0
+    with open(ipBanCmd, "rt") as c:
+        for line in c:
+            lineCount += 1
+            if lineCount == 1:
+                assert line == "add I4ban 10.0.0.1 timeout 1\n", line
+            elif lineCount == 2:
+                assert line == "add I4ban 10.0.0.1 timeout 1\n", line
+    assert lineCount == 2, lineCount
+    _clean()
+
+def whenUnfinishedBanThenTimeoutReset():
+    _clean()
+    action = Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban", "banSeconds": 2})
+    action.act({"ip": "10.0.0.1"})
+    time.sleep(1)
+    action.act({"ip": "10.0.0.1"})
+    assert os.path.exists(ipBanCmd)
+    lineCount = 0
+    with open(ipBanCmd, "rt") as c:
+        for line in c:
+            lineCount += 1
+            if lineCount == 1:
+                assert line == "add I4ban 10.0.0.1 timeout 2\n", line
+            elif lineCount == 2:
+                assert line == "del I4ban 10.0.0.1\n", line
+            elif lineCount == 3:
+                assert line == "add I4ban 10.0.0.1 timeout 2\n", line
+    assert lineCount == 3, lineCount
+    _clean()
+
+def unitTests():
+    whenBanIPv4ThenAddToIPv4Set()
+    whenBanIPv6ThenAddToIPv6Set()
+    whenBanTwoIPThenTwoLinesInState()
+    whenBanAnewThenNoDuplicate()
+    whenFinishedBanThenAsIfNotThere()
+    whenUnfinishedBanThenTimeoutReset()
--- a/tests/action_nftBan.py
+++ b/tests/action_nftBan.py
@ -17,45 +17,45 @@ def _clean():

 def whenBanIPv4ThenAddToIPv4Set():
    _clean()
-    Action({"IP": "ip", "nftSetIPv4": "I4 ban", "nftSetIPv6": "I6 ban"}).act({"ip": "10.0.0.1"})
+    Action({"IP": "ip", "nftSetIPv4": "ip I4 ban", "nftSetIPv6": "ip6 I6 ban"}).act({"ip": "10.0.0.1"})
    assert os.path.exists(nftBanCmd)
    assert os.path.exists(nftBanState)
    nbLines = 0
    with open(nftBanCmd, "rt") as c:
        for line in c:
-            assert line == "add element I4 ban {10.0.0.1}\n", line
+            assert line == "add element ip I4 ban {10.0.0.1}\n", line
            nbLines += 1
    assert nbLines == 1, nbLines
    nbBans = 0
    with open(nftBanState) as s:
        for ban in json.load(s):
-            assert ban["IP"] == "10.0.0.1" and ban["nftSet"] == "I4 ban", str(ban)
+            assert ban["IP"] == "10.0.0.1" and ban["nfSet"] == "ip I4 ban", str(ban)
            nbBans += 1
    assert nbBans == 1, nbBans
    _clean()

 def whenBanIPv6ThenAddToIPv6Set():
    _clean()
-    Action({"IP": "ip", "nftSetIPv4": "I4 ban", "nftSetIPv6": "I6 ban"}).act({"ip": "::1"})
+    Action({"IP": "ip", "nftSetIPv4": "ip I4 ban", "nftSetIPv6": "ip6 I6 ban"}).act({"ip": "::1"})
    assert os.path.exists(nftBanCmd)
    assert os.path.exists(nftBanState)
    nbLines = 0
    with open(nftBanCmd, "rt") as c:
        for line in c:
-            assert line == "add element I6 ban {::1}\n", line
+            assert line == "add element ip6 I6 ban {::1}\n", line
            nbLines += 1
    assert nbLines == 1, nbLines
    nbBans = 0
    with open(nftBanState) as s:
        for ban in json.load(s):
-            assert ban["IP"] == "::1" and ban["nftSet"] == "I6 ban", str(ban)
+            assert ban["IP"] == "::1" and ban["nfSet"] == "ip6 I6 ban", str(ban)
            nbBans += 1
    assert nbBans == 1, nbBans
    _clean()

 def whenBanTwoIPThenTwoLinesInState():
    _clean()
-    action = Action({"IP": "ip", "nftSetIPv4": "I4 ban", "nftSetIPv6": "I6 ban"})
+    action = Action({"IP": "ip", "nftSetIPv4": "ip I4 ban", "nftSetIPv6": "ip6 I6 ban"})
    action.act({"ip": "10.0.0.1"})
    action.act({"ip": "::1"})
    action.act({"ip": "10.0.0.1"})
@ -64,9 +64,9 @@ def whenBanTwoIPThenTwoLinesInState():
    with open(nftBanState) as s:
        for ban in json.load(s):
            if ban["IP"] == "10.0.0.1":
-                assert ban["nftSet"] == "I4 ban", str(ban)
+                assert ban["nfSet"] == "ip I4 ban", str(ban)
            elif ban["IP"] == "::1":
-                assert ban["nftSet"] == "I6 ban", str(ban)
+                assert ban["nfSet"] == "ip6 I6 ban", str(ban)
            else:
                assert false, str(ban)
            nbBans += 1
@ -75,7 +75,7 @@ def whenBanTwoIPThenTwoLinesInState():

 def whenBanAnewThenNoDuplicate():
    _clean()
-    action = Action({"IP": "ip", "nftSetIPv4": "I4 ban", "nftSetIPv6": "I6 ban"})
+    action = Action({"IP": "ip", "nftSetIPv4": "ip I4 ban", "nftSetIPv6": "ip6 I6 ban"})
    action.act({"ip": "10.0.0.1"})
    action.act({"ip": "10.0.0.1"})
    assert os.path.exists(nftBanCmd)
@ -85,24 +85,24 @@ def whenBanAnewThenNoDuplicate():
        for line in c:
            lineCount += 1
            if lineCount == 1:
-                assert line == "add element I4 ban {10.0.0.1}\n", line
+                assert line == "add element ip I4 ban {10.0.0.1}\n", line
            elif lineCount == 2:
-                assert line == "delete element I4 ban {10.0.0.1}\n", line
+                assert line == "delete element ip I4 ban {10.0.0.1}\n", line
            elif lineCount == 3:
-                assert line == "add element I4 ban {10.0.0.1}\n", line
+                assert line == "add element ip I4 ban {10.0.0.1}\n", line
    assert lineCount == 3, lineCount
    nbBans = 0
    with open(nftBanState) as s:
        for ban in json.load(s):
            if ban["IP"] == "10.0.0.1":
-                assert ban["nftSet"] == "I4 ban", str(ban)
+                assert ban["nfSet"] == "ip I4 ban", str(ban)
            nbBans += 1
    assert nbBans == 1, nbBans
    _clean()

 def whenFinishedBanThenAsIfNotThere():
    _clean()
-    action = Action({"IP": "ip", "nftSetIPv4": "I4 ban", "nftSetIPv6": "I6 ban", "banSeconds": 1})
+    action = Action({"IP": "ip", "nftSetIPv4": "ip I4 ban", "nftSetIPv6": "ip6 I6 ban", "banSeconds": 1})
    action.act({"ip": "10.0.0.1"})
    time.sleep(1)
    action.act({"ip": "10.0.0.1"})
@ -112,15 +112,15 @@ def whenFinishedBanThenAsIfNotThere():
        for line in c:
            lineCount += 1
            if lineCount == 1:
-                assert line == "add element I4 ban {10.0.0.1 timeout 1s}\n", line
+                assert line == "add element ip I4 ban {10.0.0.1 timeout 1s}\n", line
            elif lineCount == 2:
-                assert line == "add element I4 ban {10.0.0.1 timeout 1s}\n", line
+                assert line == "add element ip I4 ban {10.0.0.1 timeout 1s}\n", line
    assert lineCount == 2, lineCount
    _clean()

 def whenUnfinishedBanThenTimeoutReset():
    _clean()
-    action = Action({"IP": "ip", "nftSetIPv4": "I4 ban", "nftSetIPv6": "I6 ban", "banSeconds": 2})
+    action = Action({"IP": "ip", "nftSetIPv4": "ip I4 ban", "nftSetIPv6": "ip6 I6 ban", "banSeconds": 2})
    action.act({"ip": "10.0.0.1"})
    time.sleep(1)
    action.act({"ip": "10.0.0.1"})
@ -130,11 +130,11 @@ def whenUnfinishedBanThenTimeoutReset():
        for line in c:
            lineCount += 1
            if lineCount == 1:
-                assert line == "add element I4 ban {10.0.0.1 timeout 2s}\n", line
+                assert line == "add element ip I4 ban {10.0.0.1 timeout 2s}\n", line
            elif lineCount == 2:
-                assert line == "delete element I4 ban {10.0.0.1}\n", line
+                assert line == "delete element ip I4 ban {10.0.0.1}\n", line
            elif lineCount == 3:
-                assert line == "add element I4 ban {10.0.0.1 timeout 2s}\n", line
+                assert line == "add element ip I4 ban {10.0.0.1 timeout 2s}\n", line
    assert lineCount == 3, lineCount
    _clean()

--- a/tests/main.py
+++ b/tests/main.py
@ -20,7 +20,7 @@ def main():

    # Unit tests
    import filter_equals, filter_greaterOrEquals, filter_in, filter_inNetworks, filter_lowerOrEquals, filter_pcre, filter_pcreAny, filter_userExists
-    import action_counterRaise, action_counterReset, action_dailyReport, action_email, action_log, action_nftBan
+    import action_counterRaise, action_counterReset, action_dailyReport, action_dnatCapture, action_dnatReplace, action_email, action_ipsetBan, action_log, action_nftBan

    filter_equals.unitTests()
    filter_greaterOrEquals.unitTests()
@ -33,7 +33,10 @@ def main():
    action_counterRaise.unitTests()
    action_counterReset.unitTests()
    action_dailyReport.unitTests()
+    action_dnatCapture.unitTests()
+    action_dnatReplace.unitTests()
    action_email.unitTests()
+    action_ipsetBan.unitTests()
    action_log.unitTests()
    action_nftBan.unitTests()

--- a/tests/nftBan.cmd.test_ref
+++ b/tests/nftBan.cmd.test_ref
@ -1 +1 @@
-add element I4 bans {1.2.3.4 timeout 100s}
+add element ip I4 bans {1.2.3.4 timeout 100s}
--- a/tests/pyruse.json
+++ b/tests/pyruse.json
@ -58,7 +58,7 @@
 			},
 			{
 				"action": "action_nftBan",
-				"args": { "IP": "ip", "banSeconds": 100, "nftSetIPv4": "I4 bans", "nftSetIPv6": "I6 bans" },
+				"args": { "IP": "ip", "banSeconds": 100, "nftSetIPv4": "ip I4 bans", "nftSetIPv6": "ip6 I6 bans" },
 				"then": "… finalize after last action"
 			}
 		],
@ -87,5 +87,8 @@
 	"nftBan": {
 		"nft": [ "/bin/sh", "-c", "echo \"$0\" >>\"nftBan.cmd\"" ]
 	},
+	"ipsetBan": {
+		"ipset": [ "/bin/sh", "-c", "echo \"$0 $*\" >>\"ipsetBan.cmd\"" ]
+	},
 	"storage": "."
 }
Author	SHA1	Message	Date
Strykar	8c43300b45	Double quote to prevent globbing https://www.shellcheck.net/wiki/SC2086 Signed-off-by: Strykar <strykar@masked.invalid>	2022-10-21 05:25:40 +02:00
Y	8aaa04389f	ipset support; fixes #1	2018-03-17 17:46:22 +01:00
Yves G	f3b674ca26	dnat actions: simulate a transparent proxy, where logs are concerned	2018-03-16 13:42:01 +01:00
Y	e8d7d8df5c	fix in action_nftBan documentation	2018-03-12 18:18:28 +01:00