@@ -0,0 +1,12 @@ | |||
# Changelog | |||
This file is not intended as a dupplicate of Git logs. | |||
Its purpose is to warn of important changes between version, that users should be aware of. | |||
## v2.0 | |||
After this version is installed, the following command should be run on the `action_nftBan.py.json` file: | |||
```bash | |||
$ sudo sed -i s/nftSet/nfSet/g action_nftBan.py.json | |||
``` |
@@ -27,6 +27,8 @@ For example, to my knowledge, there is no equivalent in any tool of the same sca | |||
Pyruse is [packaged for Archlinux](https://aur.archlinux.org/packages/pyruse/). | |||
For other distributions, please [read the manual installation instructions](doc/install.md). | |||
Whenever your upgrade Pyruse, make sure to check the [Changelog](Changelog.md). | |||
## Configuration | |||
The `/etc/pyruse` directory is where system-specific files are looked-for: | |||
@@ -47,7 +49,7 @@ For more in-depth documentation, please refer to these pages: | |||
- [the built-in filters](doc/builtinfilters.md) | |||
- [the counter-based actions](doc/counters.md) | |||
- [the DNAT-related actions](doc/dnat.md) | |||
- [the actions that log and ban](doc/logandban.md) | |||
- [the `action_noop` module](doc/noop.md) | |||
- [the `action_email` module](doc/action_email.md) | |||
- [the `action_dailyReport` module](doc/action_dailyReport.md) | |||
- [the `action_nftBan` and `action_log` modules](doc/logandban.md) |
@@ -1,33 +1,46 @@ | |||
# Configuration tips | |||
In contrast with legacy log parsers, Pyruse works with structured [systemd-journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html). This allows for better performance, since targeted comparisons become possible. | |||
The general intent, when writing the configuration file, should be to handle the log entries that appear the most often first, in as few steps as possible. For example, I ran some stats on my server before writing my own configuration file; I got: | |||
| systemd units | number of journal entries | | |||
| ------------------------------- | -------------------------:| | |||
| `prosody.service` | 518019 | | |||
| `gitea.service` | 329389 | | |||
| `uwsgi@nextcloud.service` | 217342 | | |||
| `session-*.scope` | 89813 | | |||
| `nginx.service` | 80762 | | |||
| `dovecot.service` | 61898 | | |||
| `exim.service` | 60743 | | |||
| `init.scope` | 43021 | | |||
| `nextcloud-maintenance.service` | 20775 | | |||
| `haproxy.service` | 18445 | | |||
| `user@*.service` | 7306 | | |||
| `minidlna.service` | 6032 | | |||
| `loolwsd.service` | 5797 | | |||
| `sshd.service` | 4959 | | |||
| `spamassassin-update.service` | 2383 | | |||
| `systemd-nspawn@*.service` | 1497 | | |||
| `nslcd.service` | 867 | | |||
| `nfs-mountd.service` | 723 | | |||
| `systemd-logind.service` | 696 | | |||
| `nfs-server.service` | 293 | | |||
| `systemd-networkd.service` | 121 | | |||
| misc. units with < 100 entries | | | |||
In contrast with legacy log parsers, Pyruse works with structured [systemd-journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html). | |||
This allows for better performance, since targeted comparisons become possible. | |||
The general intent, when writing the configuration file, should be to handle the log entries that appear the most often first, in as few steps as possible. | |||
For example, I ran some stats on my server on the log entries of the past week; I got: | |||
| SYSLOG_IDENTIFIER | number of journal entries | | |||
| ----------------------- | -------------------------:| | |||
| `uwsgi` (for Nextcloud) | 55930 | | |||
| `gitea` | 38923 | | |||
| `prosody` | 25596 | | |||
| `haproxy` | 21877 | | |||
| `postgres` | 12990 | | |||
| `nginx` | 12808 | | |||
| `dovecot` | 7062 | | |||
| `exim` | 2540 | | |||
| `systemd` | 1997 | | |||
| `su` | 1458 | | |||
| `ownCloud` (Nextcloud) | 1067 | | |||
| `sshd` | 1051 | | |||
| `mandb` | 953 | | |||
| `spamd` | 855 | | |||
| `pyruse` | 615 | | |||
| `kernel` | 420 | | |||
| `msmtp` | 295 | | |||
| `sa-compile` | 255 | | |||
| `ansible-*` | 103 | | |||
| `systemd-logind` | 102 | | |||
| `python` | 78 | | |||
| `rpc.mountd` | 52 | | |||
| `ldapwhoami` | 42 | | |||
| `prosody_auth` | 42 | | |||
| `minidlnad` | 39 | | |||
| `kill` | 28 | | |||
| `sudo` | 26 | | |||
| `loolwsd` | 17 | | |||
| `exportfs` | 15 | | |||
| `dehydrated` | 6 | | |||
| `sa-update` | 5 | | |||
| `nslcd` | 4 | | |||
| `rpc.idmapd` | 1 | | |||
For reference, here is the command that gives these statistics: | |||
@@ -37,6 +50,16 @@ $ bash ./extra/examples/get-systemd-stats.sh >~/systemd-units.stats.tsv | |||
One should also remember, that numeric comparison are faster that string comparison, which in turn are faster than regular expression pattern-matching. Further more, some log entries are not worth checking for, because they are too rare: it costs more to grab them with filters (that most log entries will have to pass through), than letting them get caught by the catch-all last execution chain, which typically uses the `action_dailyReport` module. | |||
An efficient way to organize the configuration file is by handling units from the most verbose to the least verbose, and for each unit, filter-out useless entries based on the `PRIORITY` (which is an integer number) whenever it is possible. In short, filtering on the actual message, while not entirely avoidable, is the last-resort operation. | |||
An efficient way to organize the configuration file is by handling Syslog-identifiers from the most verbose to the least verbose, and for each one, filter-out useless entries based on the `PRIORITY` (which is an integer number) whenever it is possible. | |||
In short, filtering on the actual message, while not entirely avoidable, is the last-resort operation. | |||
NOTE: I used to group my log entries (and Pyruse execution chains) by `_SYSTEMD_UNIT`, which seemed logical at the time. | |||
However, for some reason, there is some “leaking” of logs from some units to others; for example, I had Nginx logs appearing in the Exim `_SYSTEMD_UNIT`… The reason probably lies somewhere in inter-process communication, or with the launching of external commands. | |||
Anyway, I found that grouping by `SYSLOG_IDENTIFIER` actually gives better results: | |||
* `SYSLOG_IDENTIFIER` names are shorter than `_SYSTEMD_UNIT` names, hence probably quicker to compare `:-p` | |||
* Several `_SYSTEMD_UNIT` names from generic units (like `unit-name@instance-name`) end up into the same `SYSLOG_IDENTIFIER`, which allows to occasionaly replace `filter_pcre` with `filter_equals`. | |||
* A single program often does several tasks, and `SYSLOG_IDENTIFIER` reflects this diversity, which makes writing rules much easier. | |||
For example, Pyruse sends emails using msmtp; I do not care about `msmtp`’s logs, but I do about `pyruse`’s. Filtering-out logs from the `msmtp` `SYSLOG_IDENTIFIER` is much easier to do than getting rid of email-related logs from the `pyruse.service` systemd unit. | |||
An [example based on the above statistics](../extra/examples/full_pyruse.json) is available in the `extra/examples/` source directory. |
@@ -4,7 +4,7 @@ | |||
Pyruse provides two actions, namely `action_dnatCapture` and `action_dnatReplace`, that work together towards a single goal: giving to Pyruse’s filters and actions the illusion that client connections come directly to the examined services, instead of going through a firewall Network Address Translation or a reverse-proxy. | |||
If for example you run a Prosody XMPP server (or anything that does not handle the [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt)) behind an HAProxy reverse-proxy, or if all your network connections go through a NAT’ing proxy in your DMZ, then the following happens: the services report your proxy as being the client, ie. usually `127.0.0.1` (same machine) or your LAN’s gateway IP. | |||
If for example you run a Prosody XMPP server (or anything that does not handle the [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt)) behind an HAProxy reverse-proxy, or if all your network connections go through a NAT’ing router in your DMZ, then the following happens: the services report your proxy as being the client, ie. usually `127.0.0.1` (same machine) or your LAN’s router IP. | |||
Here is a simplified illustration of the network configuration: | |||
@@ -5,7 +5,7 @@ The software requirements are: | |||
* a modern systemd-based Linux operating system (eg. [Archlinux](https://archlinux.org/)- or [Fedora](https://getfedora.org/)-based distributions); | |||
* python, at least version 3.4 (or [more, depending on the modules](intro_tech.md) being used); | |||
* [python-systemd](https://www.freedesktop.org/software/systemd/python-systemd/journal.html); | |||
* [nftables](http://wiki.nftables.org/) _if_ IP address bans are to be managed; | |||
* [nftables](http://wiki.nftables.org/) or [ipset](http://ipset.netfilter.org/) _if_ IP address bans are to be managed; | |||
* a sendmail-like program _if_ emails are wanted. | |||
Besides, getting the software requires [Git](http://git-scm.com/), and packaging it requires [python-setuptools](http://pypi.python.org/pypi/setuptools). | |||
@@ -45,4 +45,6 @@ To install Pyruse on the system, run these commands as root, in the root directo | |||
The `package` line is the one that actually alters the system. Until Pyruse is packaged for your operating system, you may want to change this line to `checkinstall package`. [Checkinstall](https://en.wikipedia.org/wiki/CheckInstall) should be able to turn your Pyruse installation into a native Linux package. | |||
Then, to run Pyruse, start (and enable) `pyruse.service`. | |||
If you use nftables bans, you should also start (and enable) `pyruse-boot@action_nftBan.service`. | |||
Likewise, if you use ipset bans, you should start (and enable) `pyruse-boot@action_ipsetBan.service`. |
@@ -2,7 +2,7 @@ | |||
Everyone knows [Fail2ban](http://www.fail2ban.org/). | |||
This program is excellent at what it does, which is matching patterns in a number of log files, and keeping track of counters in order to launch actions (ban an IP, send an email…) when thresholds are crossed. | |||
[Fail2ban is extremely customizable](http://yalis.fr/cms/index.php/post/2014/11/02/Migrate-from-DenyHosts-to-Fail2ban)… to a point; and to my knowledge it cannot benefit from all the features of modern-day logging with systemd. | |||
[Fail2ban is extremely customizable](http://yalis.fr/cms/index.php/post/2014/11/02/Migrate-from-DenyHosts-to-Fail2ban)… to a point; to my knowledge it cannot benefit from all the features of modern-day logging with systemd. | |||
Then, there is the less-known and aging [Epylog](http://freshmeat.sourceforge.net/projects/epylog/); several programs exist with the same features. | |||
Just like Fail2ban, this program continuously scans the logs to do pattern matching. |
@@ -31,8 +31,10 @@ Pyruse is split into several Python files: | |||
* `workflow.py`: This is where the configuration file get translated into actual execution chains, linked together into a single static workflow. | |||
* `module.py`: Whenever the workflow needs to add a filter or an action to an execution chain, this module finds it in the filesystem. | |||
* `base.py`: All actions and filters inherit from the `Action` and `Filter` classes defined in there; they act as an abstraction layer between the workflow and the modules. | |||
* `counter.py`: This utility class is parent to modules that manage a counter (Python supports multiple-inheritance). | |||
* `email.py`: This utility class is parent to modules that send emails (Python supports multiple-inheritance). | |||
* `ban.py`: This utility class is parent to modules that ban IP addresses using [Netfilter](https://netfilter.org/) (Python supports multiple-inheritance). | |||
* `counter.py`: This utility class is parent to modules that manage a counter. | |||
* `dnat.py`: This file contains utility parent classes for actions that try and restore the actual client IP addresses. | |||
* `email.py`: This utility class is parent to modules that send emails. | |||
All else is actions and filters… | |||
Some are delivered with Pyruse itself; [more can be added](customize.md). |
@@ -30,9 +30,22 @@ Here are some examples: | |||
## Ban IP addresses after they misbehaved | |||
Linux provides a number of firewall solutions: [iptables](http://www.netfilter.org/), its successor [nftables](http://wiki.nftables.org/), and many iptables frontends like [Shorewall](http://www.shorewall.net/) or RedHat’s [firewalld](http://www.firewalld.org/). | |||
For Pyruse, **nftables** was chosen, because it is modern and light-weight, and provides interesting features. | |||
Action `action_nftBan` takes three mandatory arguments: | |||
For Pyruse, **nftables** was initially chosen, because it is modern and light-weight, and provides interesting features. Besides, only nftables is actually tested in real use. | |||
Now, an **ipset**-based [alternative is also available](#iptables-with-ipset). | |||
### nftables | |||
Action `action_nftBan` requires that nftables is installed. | |||
In addition, the binary to run (and parameters if needed) should be set in the configuration file; here is the default value: | |||
```json | |||
"nftBan": { | |||
"nft": [ "/usr/bin/nft" ] | |||
} | |||
``` | |||
This action takes three mandatory arguments: | |||
* `nftSetIPv4` and `nftSetIPv6` are the nftables sets where IP addresses will be added. The IP address is considered to be IPv6 if it contains at least one colon (`:`) character, else it is supposed to be IPv4. | |||
* `IP` gives the name of the entry field to read, in order to get the IP address. | |||
@@ -88,7 +101,7 @@ Here are examples: | |||
} | |||
``` | |||
### List the currently banned addresses | |||
#### List the currently banned addresses | |||
To see what IP addresses are currently banned, here is the `nft` command: | |||
@@ -114,7 +127,7 @@ table ip Inet4 { | |||
_Note_: The un-rounded timeouts are post-reboot restored bans. | |||
### Un-ban an IP address | |||
#### Un-ban an IP address | |||
It is bound to happen some day: you will want to un-ban a banned IP address. | |||
@@ -129,7 +142,7 @@ To avoid that, also delete the corresponding record from the `action_nftBan.py.j | |||
To go further, you could tweak your configuration, so that your trusted IP addresses never reach `action_nftBan`. | |||
### Manual ban of an IP address | |||
#### Manual ban of an IP address | |||
To add a ban yourself, run a command like this: | |||
@@ -139,16 +152,99 @@ $ sudo nft 'add element ip Inet4 ssh_ban {192.168.1.1 timeout 5d} | |||
The `timeout …` part can be omitted to add a permanent ban. The timeout can be any combination of days (`d`), hours (`h`), minutes (`m`), and seconds (`s`), eg. “`3d31m16s`”. | |||
In order to make the ban persistent across reboots, a corresponding record should also be appended to the `action_nftBan.py.json` file in Pyruse’s [storage directory](conffile.md) (the IP address, nft Set, days, hours, minutes, seconds, and actual path to the file should be adapted to your needs): | |||
In order to make the ban persistent across reboots, a corresponding record should also be appended to the `action_nftBan.py.json` file in Pyruse’s [storage directory](conffile.md) (the IP address, Nftables Set, days, hours, minutes, seconds, and actual path to the file should be adapted to your needs): | |||
* either a time-limited ban: | |||
```bash | |||
$ sudo sed -i "\$s/.\$/$(date +', {"IP": "192.168.1.1", "nfSet": "ip Inet4 ssh_ban", "timestamp": %s.000000}' -d 'now +3day +31minute +16second')]/" /var/lib/pyruse/action_nftBan.py.json | |||
``` | |||
* or an unlimited ban: | |||
```bash | |||
$ sudo sed -i '$s/.$/, {"IP": "192.168.1.1", "nfSet": "ip Inet4 ssh_ban", "timestamp": 0}]/' /var/lib/pyruse/action_nftBan.py.json | |||
``` | |||
### iptables with ipset | |||
Action `action_ipsetBan` requires that ipset and iptables are installed. | |||
In addition, the ipset binary to run (and parameters if needed) should be set in the configuration file; here is the default value: | |||
```json | |||
"ipsetBan": { | |||
"ipset": [ "/usr/bin/ipset", "-exist", "-quiet" ] | |||
} | |||
``` | |||
This action works exactly [like `action_nftBan`](#nftables), except parameters `nftSetIPv4` and `nftSetIPv6` are named `ipSetIPv4` and `ipSetIPv6` instead. | |||
The name of the set in the `ipSetIPv4` parameter must have been created before running Pyruse, with: | |||
```bash | |||
$ sudo ipset create SET_NAME hash:ip family inet hashsize 1024 maxelem 65535 | |||
``` | |||
Likewise, the set given by `ipSetIPv6` must have been created before running Pyruse, with: | |||
```bash | |||
$ sudo ipset create SET_NAME hash:ip family inet6 hashsize 1024 maxelem 65535 | |||
``` | |||
Here are examples of usage for this action: | |||
```json | |||
{ | |||
"action": "action_ipsetBan", | |||
"args": { "IP": "thatIP", "banSeconds": 86400, "ipSetIPv4": "mail_ban4", "ipSetIPv6": "mail_ban6" } | |||
} | |||
{ | |||
"action": "action_ipsetBan", | |||
"args": { "IP": "thatIP", "ipSetIPv4": "sshd_ban4", "ipSetIPv6": "sshd_ban6" } | |||
} | |||
``` | |||
#### List the currently banned addresses | |||
To see what IP addresses are currently banned, here is the `ipset` command: | |||
```bash | |||
$ sudo ipset list mail_ban4' | |||
``` | |||
#### Un-ban an IP address | |||
To remove an IP address from a set, here is the `ipset` command: | |||
```bash | |||
$ sudo ipset del mail_ban4 10.0.0.10' | |||
``` | |||
To make the change persistent across reboots, also delete the corresponding record from the `action_ipsetBan.py.json` file in Pyruse’s [storage directory](conffile.md). | |||
To go further, you could tweak your configuration, so that your trusted IP addresses never reach `action_ipsetBan`. | |||
#### Manual ban of an IP address | |||
To add a ban yourself, run a command like this: | |||
```bash | |||
$ sudo ipset add ssh_ban4 192.168.1.1 timeout 261076 | |||
``` | |||
The `timeout …` part can be omitted to add a permanent ban; otherwise it is a number of seconds. | |||
In order to make the ban persistent across reboots, a corresponding record should also be appended to the `action_ipsetBan.py.json` file in Pyruse’s [storage directory](conffile.md) (the IP address, Nftables Set, days, hours, minutes, seconds, and actual path to the file should be adapted to your needs): | |||
* either a time-limited ban: | |||
```bash | |||
$ sudo sed -i "\$s/.\$/$(date +', {"IP": "192.168.1.1", "nftSet": "ip Inet4 ssh_ban", "timestamp": %s.000000}' -d 'now +3day +31minute +16second')]/" /var/lib/pyruse/action_nftBan.py.json | |||
$ sudo sed -i "\$s/.\$/$(date +', {"IP": "192.168.1.1", "nfSet": "ssh_ban4", "timestamp": %s.000000}' -d 'now +3day +31minute +16second')]/" /var/lib/pyruse/action_ipsetBan.py.json | |||
``` | |||
* or an unlimited ban: | |||
```bash | |||
$ sudo sed -i '$s/.$/, {"IP": "192.168.1.1", "nftSet": "ip Inet4 ssh_ban", "timestamp": 0}]/' /var/lib/pyruse/action_nftBan.py.json | |||
$ sudo sed -i '$s/.$/, {"IP": "192.168.1.1", "nfSet": "ssh_ban4", "timestamp": 0}]/' /var/lib/pyruse/action_ipsetBan.py.json | |||
``` |
@@ -1,11 +1,15 @@ | |||
#!/bin/bash | |||
# $1 (optional): systemd-nspawn machine name | |||
# $1 (optional): grouping criteria: SYSLOG_IDENTIFIER (default) or _SYSTEMD_UNIT | |||
# $+ (optional): journalctl options (-M machine, -S date…) | |||
CRIT=${1:-SYSLOG_IDENTIFIER} | |||
shift | |||
{ | |||
printf 'Units\tTotal\tP7\tP6\tP5\tP4\tP3\tP2\tP1\tP0\n' | |||
sudo journalctl ${1:+-M "$1"} -o json-pretty --output-fields=_SYSTEMD_UNIT,PRIORITY \ | |||
printf '%s\tTotal\tP7\tP6\tP5\tP4\tP3\tP2\tP1\tP0\n' $CRIT | |||
sudo journalctl "$@" -o json-pretty --output-fields=${CRIT},PRIORITY \ | |||
| tr -d $'"\t, ' \ | |||
| awk -F: -vOFS=: ' | |||
| awk -F: -vOFS=: -vCRIT=$CRIT ' | |||
/^\{/ { | |||
u = "" | |||
p = -1 | |||
@@ -13,7 +17,7 @@ | |||
$1 == "PRIORITY" { | |||
p = $2 | |||
} | |||
$1 == "_SYSTEMD_UNIT" { | |||
$1 == CRIT { | |||
u = gensub(\ | |||
"@.*(\\.[^.]*)$",\ | |||
"@*\\1",\ |
@@ -0,0 +1,3 @@ | |||
[Unit] | |||
Requires=iptables.service | |||
After=iptables.service |
@@ -0,0 +1,37 @@ | |||
# pyruse is intended as a replacement to both fail2ban and epylog | |||
# Copyright © 2017–2018 Y. Gablin | |||
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing. | |||
import os | |||
import subprocess | |||
from pyruse import ban, base, config | |||
class Action(base.Action, ban.NetfilterBan): | |||
_storage = config.Config().asMap().get("storage", "/var/lib/pyruse") \ | |||
+ "/" + os.path.basename(__file__) + ".json" | |||
_ipset = config.Config().asMap().get("ipsetBan", {}).get("ipset", ["/usr/bin/ipset", "-exist", "-quiet"]) | |||
def __init__(self, args): | |||
base.Action.__init__(self) | |||
ban.NetfilterBan.__init__(self, Action._storage) | |||
if args is None: | |||
return # on-boot configuration | |||
ipv4Set = args["ipSetIPv4"] | |||
ipv6Set = args["ipSetIPv6"] | |||
field = args["IP"] | |||
banSeconds = args.get("banSeconds", None) | |||
self.initSelf(ipv4Set, ipv6Set, field, banSeconds) | |||
def act(self, entry): | |||
ban.NetfilterBan.act(self, entry) | |||
def setBan(self, nfSet, ip, seconds): | |||
cmd = list(Action._ipset) | |||
cmd.extend(["add", nfSet, ip]) | |||
if seconds > 0: | |||
cmd.extend(["timeout", str(seconds)]) | |||
subprocess.run(cmd) | |||
def cancelBan(self, nfSet, ip): | |||
cmd = list(Action._ipset) | |||
cmd.extend(["del", nfSet, ip]) | |||
subprocess.run(cmd) |
@@ -1,97 +1,39 @@ | |||
# pyruse is intended as a replacement to both fail2ban and epylog | |||
# Copyright © 2017–2018 Y. Gablin | |||
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing. | |||
import datetime | |||
import json | |||
import os | |||
import subprocess | |||
from pyruse import base, config | |||
from pyruse import ban, base, config | |||
class Action(base.Action): | |||
class Action(base.Action, ban.NetfilterBan): | |||
_storage = config.Config().asMap().get("storage", "/var/lib/pyruse") \ | |||
+ "/" + os.path.basename(__file__) + ".json" | |||
_nft = config.Config().asMap().get("nftBan", {}).get("nft", ["/usr/bin/nft"]) | |||
def __init__(self, args): | |||
super().__init__() | |||
base.Action.__init__(self) | |||
ban.NetfilterBan.__init__(self, Action._storage) | |||
if args is None: | |||
return # on-boot configuration | |||
self.ipv4Set = args["nftSetIPv4"] | |||
self.ipv6Set = args["nftSetIPv6"] | |||
self.field = args["IP"] | |||
self.banSeconds = args.get("banSeconds", None) | |||
ipv4Set = args["nftSetIPv4"] | |||
ipv6Set = args["nftSetIPv6"] | |||
field = args["IP"] | |||
banSeconds = args.get("banSeconds", None) | |||
self.initSelf(ipv4Set, ipv6Set, field, banSeconds) | |||
def act(self, entry): | |||
ip = entry[self.field] | |||
nftSet = self.ipv6Set if ":" in ip else self.ipv4Set | |||
newBan = {"IP": ip, "nftSet": nftSet} | |||
ban.NetfilterBan.act(self, entry) | |||
now = datetime.datetime.utcnow() | |||
bans = [] | |||
previousTS = None | |||
try: | |||
with open(Action._storage) as dataFile: | |||
for ban in json.load(dataFile): | |||
if ban["timestamp"] > 0 and ban["timestamp"] <= now.timestamp(): | |||
continue | |||
elif {k: ban[k] for k in newBan.keys()} == newBan: | |||
# should not happen, since the IP is banned… | |||
previousTS = ban["timestamp"] | |||
else: | |||
bans.append(ban) | |||
except IOError: | |||
pass # new file | |||
if previousTS is not None: | |||
try: | |||
cmd = list(Action._nft) | |||
cmd.append("delete element %s {%s}" % (nftSet, ip)) | |||
subprocess.run(cmd) | |||
except Exception: | |||
pass # too late: not a problem | |||
if self.banSeconds: | |||
until = now + datetime.timedelta(seconds = self.banSeconds) | |||
newBan["timestamp"] = until.timestamp() | |||
timeout = self.banSeconds | |||
else: | |||
newBan["timestamp"] = 0 | |||
timeout = 0 | |||
self._doBan(timeout, ip, nftSet) | |||
bans.append(newBan) | |||
with open(Action._storage, "w") as dataFile: | |||
json.dump(bans, dataFile) | |||
def boot(self): | |||
now = datetime.datetime.utcnow() | |||
bans = [] | |||
try: | |||
with open(Action._storage) as dataFile: | |||
for ban in json.load(dataFile): | |||
if ban["timestamp"] == 0: | |||
self._doBan(0, ban["IP"], ban["nftSet"]) | |||
bans.append(ban) | |||
elif ban["timestamp"] <= now.timestamp(): | |||
continue | |||
else: | |||
until = datetime.datetime.utcfromtimestamp(ban["timestamp"]) | |||
timeout = (until - now).total_seconds() | |||
self._doBan(int(timeout), ban["IP"], ban["nftSet"]) | |||
bans.append(ban) | |||
except IOError: | |||
pass # no file | |||
with open(Action._storage, "w") as dataFile: | |||
json.dump(bans, dataFile) | |||
def _doBan(self, seconds, ip, nftSet): | |||
if seconds < 0: | |||
return # can happen when the threshold is crossed while computing the duration | |||
def setBan(self, nfSet, ip, seconds): | |||
if seconds == 0: | |||
timeout = "" | |||
else: | |||
timeout = " timeout %ss" % seconds | |||
cmd = list(Action._nft) | |||
cmd.append("add element %s {%s%s}" % (nftSet, ip, timeout)) | |||
cmd.append("add element %s {%s%s}" % (nfSet, ip, timeout)) | |||
subprocess.run(cmd) | |||
def cancelBan(self, nfSet, ip): | |||
cmd = list(Action._nft) | |||
cmd.append("delete element %s {%s}" % (nfSet, ip)) | |||
subprocess.run(cmd) |
@@ -0,0 +1,85 @@ | |||
# pyruse is intended as a replacement to both fail2ban and epylog | |||
# Copyright © 2017–2018 Y. Gablin | |||
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing. | |||
import abc | |||
import datetime | |||
import json | |||
class NetfilterBan(abc.ABC): | |||
def __init__(self, storage): | |||
self.storage = storage | |||
def initSelf(self, ipv4Set, ipv6Set, ipField, banSeconds): | |||
self.ipv4Set = ipv4Set | |||
self.ipv6Set = ipv6Set | |||
self.field = ipField | |||
self.banSeconds = banSeconds | |||
@abc.abstractmethod | |||
def setBan(self, nfSet, ip, seconds): | |||
pass | |||
@abc.abstractmethod | |||
def cancelBan(self, nfSet, ip): | |||
pass | |||
def act(self, entry): | |||
ip = entry[self.field] | |||
nfSet = self.ipv6Set if ":" in ip else self.ipv4Set | |||
newBan = {"IP": ip, "nfSet": nfSet} | |||
now = datetime.datetime.utcnow() | |||
bans = [] | |||
previousTS = None | |||
try: | |||
with open(self.storage) as dataFile: | |||
for ban in json.load(dataFile): | |||
if ban["timestamp"] > 0 and ban["timestamp"] <= now.timestamp(): | |||
continue | |||
elif {k: ban[k] for k in newBan.keys()} == newBan: | |||
# should not happen, since the IP is banned… | |||
previousTS = ban["timestamp"] | |||
else: | |||
bans.append(ban) | |||
except IOError: | |||
pass # new file | |||
if previousTS is not None: | |||
try: | |||
self.cancelBan(nfSet, ip) | |||
except Exception: | |||
pass # too late: not a problem | |||
if self.banSeconds: | |||
until = now + datetime.timedelta(seconds = self.banSeconds) | |||
newBan["timestamp"] = until.timestamp() | |||
timeout = self.banSeconds | |||
else: | |||
newBan["timestamp"] = 0 | |||
timeout = 0 | |||
self.setBan(nfSet, ip, timeout) | |||
bans.append(newBan) | |||
with open(self.storage, "w") as dataFile: | |||
json.dump(bans, dataFile) | |||
def boot(self): | |||
now = int(datetime.datetime.utcnow().timestamp()) | |||
bans = [] | |||
try: | |||
with open(self.storage) as dataFile: | |||
for ban in json.load(dataFile): | |||
if ban["timestamp"] == 0: | |||
self.setBan(ban["nfSet"], ban["IP"], 0) | |||
bans.append(ban) | |||
elif int(ban["timestamp"]) <= now: | |||
continue | |||
else: | |||
timeout = int(ban["timestamp"]) - now | |||
self.setBan(ban["nfSet"], ban["IP"], timeout) | |||
bans.append(ban) | |||
except IOError: | |||
pass # no file | |||
with open(self.storage, "w") as dataFile: | |||
json.dump(bans, dataFile) |
@@ -0,0 +1,147 @@ | |||
# pyruse is intended as a replacement to both fail2ban and epylog | |||
# Copyright © 2017–2018 Y. Gablin | |||
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing. | |||
import json | |||
import os | |||
import time | |||
from pyruse.actions.action_ipsetBan import Action | |||
ipBanCmd = "ipsetBan.cmd" | |||
ipBanState = "action_ipsetBan.py.json" | |||
def _clean(): | |||
if os.path.exists(ipBanCmd): | |||
os.remove(ipBanCmd) | |||
if os.path.exists(ipBanState): | |||
os.remove(ipBanState) | |||
def whenBanIPv4ThenAddToIPv4Set(): | |||
_clean() | |||
Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban"}).act({"ip": "10.0.0.1"}) | |||
assert os.path.exists(ipBanCmd) | |||
assert os.path.exists(ipBanState) | |||
nbLines = 0 | |||
with open(ipBanCmd, "rt") as c: | |||
for line in c: | |||
assert line == "add I4ban 10.0.0.1\n", line | |||
nbLines += 1 | |||
assert nbLines == 1, nbLines | |||
nbBans = 0 | |||
with open(ipBanState) as s: | |||
for ban in json.load(s): | |||
assert ban["IP"] == "10.0.0.1" and ban["nfSet"] == "I4ban", str(ban) | |||
nbBans += 1 | |||
assert nbBans == 1, nbBans | |||
_clean() | |||
def whenBanIPv6ThenAddToIPv6Set(): | |||
_clean() | |||
Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban"}).act({"ip": "::1"}) | |||
assert os.path.exists(ipBanCmd) | |||
assert os.path.exists(ipBanState) | |||
nbLines = 0 | |||
with open(ipBanCmd, "rt") as c: | |||
for line in c: | |||
assert line == "add I6ban ::1\n", line | |||
nbLines += 1 | |||
assert nbLines == 1, nbLines | |||
nbBans = 0 | |||
with open(ipBanState) as s: | |||
for ban in json.load(s): | |||
assert ban["IP"] == "::1" and ban["nfSet"] == "I6ban", str(ban) | |||
nbBans += 1 | |||
assert nbBans == 1, nbBans | |||
_clean() | |||
def whenBanTwoIPThenTwoLinesInState(): | |||
_clean() | |||
action = Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban"}) | |||
action.act({"ip": "10.0.0.1"}) | |||
action.act({"ip": "::1"}) | |||
action.act({"ip": "10.0.0.1"}) | |||
assert os.path.exists(ipBanState) | |||
nbBans = 0 | |||
with open(ipBanState) as s: | |||
for ban in json.load(s): | |||
if ban["IP"] == "10.0.0.1": | |||
assert ban["nfSet"] == "I4ban", str(ban) | |||
elif ban["IP"] == "::1": | |||
assert ban["nfSet"] == "I6ban", str(ban) | |||
else: | |||
assert false, str(ban) | |||
nbBans += 1 | |||
assert nbBans == 2, nbBans | |||
_clean() | |||
def whenBanAnewThenNoDuplicate(): | |||
_clean() | |||
action = Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban"}) | |||
action.act({"ip": "10.0.0.1"}) | |||
action.act({"ip": "10.0.0.1"}) | |||
assert os.path.exists(ipBanCmd) | |||
assert os.path.exists(ipBanState) | |||
lineCount = 0 | |||
with open(ipBanCmd, "rt") as c: | |||
for line in c: | |||
lineCount += 1 | |||
if lineCount == 1: | |||
assert line == "add I4ban 10.0.0.1\n", line | |||
elif lineCount == 2: | |||
assert line == "del I4ban 10.0.0.1\n", line | |||
elif lineCount == 3: | |||
assert line == "add I4ban 10.0.0.1\n", line | |||
assert lineCount == 3, lineCount | |||
nbBans = 0 | |||
with open(ipBanState) as s: | |||
for ban in json.load(s): | |||
if ban["IP"] == "10.0.0.1": | |||
assert ban["nfSet"] == "I4ban", str(ban) | |||
nbBans += 1 | |||
assert nbBans == 1, nbBans | |||
_clean() | |||
def whenFinishedBanThenAsIfNotThere(): | |||
_clean() | |||
action = Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban", "banSeconds": 1}) | |||
action.act({"ip": "10.0.0.1"}) | |||
time.sleep(1) | |||
action.act({"ip": "10.0.0.1"}) | |||
assert os.path.exists(ipBanCmd) | |||
lineCount = 0 | |||
with open(ipBanCmd, "rt") as c: | |||
for line in c: | |||
lineCount += 1 | |||
if lineCount == 1: | |||
assert line == "add I4ban 10.0.0.1 timeout 1\n", line | |||
elif lineCount == 2: | |||
assert line == "add I4ban 10.0.0.1 timeout 1\n", line | |||
assert lineCount == 2, lineCount | |||
_clean() | |||
def whenUnfinishedBanThenTimeoutReset(): | |||
_clean() | |||
action = Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban", "banSeconds": 2}) | |||
action.act({"ip": "10.0.0.1"}) | |||
time.sleep(1) | |||
action.act({"ip": "10.0.0.1"}) | |||
assert os.path.exists(ipBanCmd) | |||
lineCount = 0 | |||
with open(ipBanCmd, "rt") as c: | |||
for line in c: | |||
lineCount += 1 | |||
if lineCount == 1: | |||
assert line == "add I4ban 10.0.0.1 timeout 2\n", line | |||
elif lineCount == 2: | |||
assert line == "del I4ban 10.0.0.1\n", line | |||
elif lineCount == 3: | |||
assert line == "add I4ban 10.0.0.1 timeout 2\n", line | |||
assert lineCount == 3, lineCount | |||
_clean() | |||
def unitTests(): | |||
whenBanIPv4ThenAddToIPv4Set() | |||
whenBanIPv6ThenAddToIPv6Set() | |||
whenBanTwoIPThenTwoLinesInState() | |||
whenBanAnewThenNoDuplicate() | |||
whenFinishedBanThenAsIfNotThere() | |||
whenUnfinishedBanThenTimeoutReset() |
@@ -29,7 +29,7 @@ def whenBanIPv4ThenAddToIPv4Set(): | |||
nbBans = 0 | |||
with open(nftBanState) as s: | |||
for ban in json.load(s): | |||
assert ban["IP"] == "10.0.0.1" and ban["nftSet"] == "ip I4 ban", str(ban) | |||
assert ban["IP"] == "10.0.0.1" and ban["nfSet"] == "ip I4 ban", str(ban) | |||
nbBans += 1 | |||
assert nbBans == 1, nbBans | |||
_clean() | |||
@@ -48,7 +48,7 @@ def whenBanIPv6ThenAddToIPv6Set(): | |||
nbBans = 0 | |||
with open(nftBanState) as s: | |||
for ban in json.load(s): | |||
assert ban["IP"] == "::1" and ban["nftSet"] == "ip6 I6 ban", str(ban) | |||
assert ban["IP"] == "::1" and ban["nfSet"] == "ip6 I6 ban", str(ban) | |||
nbBans += 1 | |||
assert nbBans == 1, nbBans | |||
_clean() | |||
@@ -64,9 +64,9 @@ def whenBanTwoIPThenTwoLinesInState(): | |||
with open(nftBanState) as s: | |||
for ban in json.load(s): | |||
if ban["IP"] == "10.0.0.1": | |||
assert ban["nftSet"] == "ip I4 ban", str(ban) | |||
assert ban["nfSet"] == "ip I4 ban", str(ban) | |||
elif ban["IP"] == "::1": | |||
assert ban["nftSet"] == "ip6 I6 ban", str(ban) | |||
assert ban["nfSet"] == "ip6 I6 ban", str(ban) | |||
else: | |||
assert false, str(ban) | |||
nbBans += 1 | |||
@@ -95,7 +95,7 @@ def whenBanAnewThenNoDuplicate(): | |||
with open(nftBanState) as s: | |||
for ban in json.load(s): | |||
if ban["IP"] == "10.0.0.1": | |||
assert ban["nftSet"] == "ip I4 ban", str(ban) | |||
assert ban["nfSet"] == "ip I4 ban", str(ban) | |||
nbBans += 1 | |||
assert nbBans == 1, nbBans | |||
_clean() |
@@ -20,7 +20,7 @@ def main(): | |||
# Unit tests | |||
import filter_equals, filter_greaterOrEquals, filter_in, filter_inNetworks, filter_lowerOrEquals, filter_pcre, filter_pcreAny, filter_userExists | |||
import action_counterRaise, action_counterReset, action_dailyReport, action_dnatCapture, action_dnatReplace, action_email, action_log, action_nftBan | |||
import action_counterRaise, action_counterReset, action_dailyReport, action_dnatCapture, action_dnatReplace, action_email, action_ipsetBan, action_log, action_nftBan | |||
filter_equals.unitTests() | |||
filter_greaterOrEquals.unitTests() | |||
@@ -36,6 +36,7 @@ def main(): | |||
action_dnatCapture.unitTests() | |||
action_dnatReplace.unitTests() | |||
action_email.unitTests() | |||
action_ipsetBan.unitTests() | |||
action_log.unitTests() | |||
action_nftBan.unitTests() | |||
@@ -87,5 +87,8 @@ | |||
"nftBan": { | |||
"nft": [ "/bin/sh", "-c", "echo \"$0\" >>\"nftBan.cmd\"" ] | |||
}, | |||
"ipsetBan": { | |||
"ipset": [ "/bin/sh", "-c", "echo \"$0 $*\" >>\"ipsetBan.cmd\"" ] | |||
}, | |||
"storage": "." | |||
} |