yves
/
pyruse
Watch 1
Star 0
Fork 0
Code Issues 1 Pull Requests 0 Releases 3 Wiki Activity
Browse Source

ipset support; fixes #1

tags/2.0^0
Y 1 year ago
parent
f3b674ca26
commit
8aaa04389f
18 changed files with 956 additions and 460 deletions
Split View Show Diff Stats
  1. 12
    0
      Changelog.md
  2. 3
    1
      README.md
  3. 52
    29
      doc/configure.md
  4. 1
    1
      doc/dnat.md
  5. 3
    1
      doc/install.md
  6. 1
    1
      doc/intro_func.md
  7. 4
    2
      doc/intro_tech.md
  8. 104
    8
      doc/logandban.md
  9. 468
    331
      extra/examples/full_pyruse.json
  10. 9
    5
      extra/examples/get-systemd-stats.sh
  11. 3
    0
      extra/systemd/action_ipsetBan.conf
  12. 37
    0
      pyruse/actions/action_ipsetBan.py
  13. 17
    75
      pyruse/actions/action_nftBan.py
  14. 85
    0
      pyruse/ban.py
  15. 147
    0
      tests/action_ipsetBan.py
  16. 5
    5
      tests/action_nftBan.py
  17. 2
    1
      tests/main.py
  18. 3
    0
      tests/pyruse.json

+ 12
- 0
Changelog.md View File

@@ -0,0 +1,12 @@
# Changelog

This file is not intended as a dupplicate of Git logs.
Its purpose is to warn of important changes between version, that users should be aware of.

## v2.0

After this version is installed, the following command should be run on the `action_nftBan.py.json` file:

```bash
$ sudo sed -i s/nftSet/nfSet/g action_nftBan.py.json
```

+ 3
- 1
README.md View File

@@ -27,6 +27,8 @@ For example, to my knowledge, there is no equivalent in any tool of the same sca
Pyruse is [packaged for Archlinux](https://aur.archlinux.org/packages/pyruse/).
For other distributions, please [read the manual installation instructions](doc/install.md).

Whenever your upgrade Pyruse, make sure to check the [Changelog](Changelog.md).

## Configuration

The `/etc/pyruse` directory is where system-specific files are looked-for:
@@ -47,7 +49,7 @@ For more in-depth documentation, please refer to these pages:
- [the built-in filters](doc/builtinfilters.md)
- [the counter-based actions](doc/counters.md)
- [the DNAT-related actions](doc/dnat.md)
- [the actions that log and ban](doc/logandban.md)
- [the `action_noop` module](doc/noop.md)
- [the `action_email` module](doc/action_email.md)
- [the `action_dailyReport` module](doc/action_dailyReport.md)
- [the `action_nftBan` and `action_log` modules](doc/logandban.md)

+ 52
- 29
doc/configure.md View File

@@ -1,33 +1,46 @@
# Configuration tips

In contrast with legacy log parsers, Pyruse works with structured [systemd-journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html). This allows for better performance, since targeted comparisons become possible.

The general intent, when writing the configuration file, should be to handle the log entries that appear the most often first, in as few steps as possible. For example, I ran some stats on my server before writing my own configuration file; I got:

| systemd units | number of journal entries |
| ------------------------------- | -------------------------:|
| `prosody.service` | 518019 |
| `gitea.service` | 329389 |
| `uwsgi@nextcloud.service` | 217342 |
| `session-*.scope` | 89813 |
| `nginx.service` | 80762 |
| `dovecot.service` | 61898 |
| `exim.service` | 60743 |
| `init.scope` | 43021 |
| `nextcloud-maintenance.service` | 20775 |
| `haproxy.service` | 18445 |
| `user@*.service` | 7306 |
| `minidlna.service` | 6032 |
| `loolwsd.service` | 5797 |
| `sshd.service` | 4959 |
| `spamassassin-update.service` | 2383 |
| `systemd-nspawn@*.service` | 1497 |
| `nslcd.service` | 867 |
| `nfs-mountd.service` | 723 |
| `systemd-logind.service` | 696 |
| `nfs-server.service` | 293 |
| `systemd-networkd.service` | 121 |
| misc. units with < 100 entries | |
In contrast with legacy log parsers, Pyruse works with structured [systemd-journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html).
This allows for better performance, since targeted comparisons become possible.

The general intent, when writing the configuration file, should be to handle the log entries that appear the most often first, in as few steps as possible.
For example, I ran some stats on my server on the log entries of the past week; I got:

| SYSLOG_IDENTIFIER | number of journal entries |
| ----------------------- | -------------------------:|
| `uwsgi` (for Nextcloud) | 55930 |
| `gitea` | 38923 |
| `prosody` | 25596 |
| `haproxy` | 21877 |
| `postgres` | 12990 |
| `nginx` | 12808 |
| `dovecot` | 7062 |
| `exim` | 2540 |
| `systemd` | 1997 |
| `su` | 1458 |
| `ownCloud` (Nextcloud) | 1067 |
| `sshd` | 1051 |
| `mandb` | 953 |
| `spamd` | 855 |
| `pyruse` | 615 |
| `kernel` | 420 |
| `msmtp` | 295 |
| `sa-compile` | 255 |
| `ansible-*` | 103 |
| `systemd-logind` | 102 |
| `python` | 78 |
| `rpc.mountd` | 52 |
| `ldapwhoami` | 42 |
| `prosody_auth` | 42 |
| `minidlnad` | 39 |
| `kill` | 28 |
| `sudo` | 26 |
| `loolwsd` | 17 |
| `exportfs` | 15 |
| `dehydrated` | 6 |
| `sa-update` | 5 |
| `nslcd` | 4 |
| `rpc.idmapd` | 1 |

For reference, here is the command that gives these statistics:

@@ -37,6 +50,16 @@ $ bash ./extra/examples/get-systemd-stats.sh >~/systemd-units.stats.tsv

One should also remember, that numeric comparison are faster that string comparison, which in turn are faster than regular expression pattern-matching. Further more, some log entries are not worth checking for, because they are too rare: it costs more to grab them with filters (that most log entries will have to pass through), than letting them get caught by the catch-all last execution chain, which typically uses the `action_dailyReport` module.

An efficient way to organize the configuration file is by handling units from the most verbose to the least verbose, and for each unit, filter-out useless entries based on the `PRIORITY` (which is an integer number) whenever it is possible. In short, filtering on the actual message, while not entirely avoidable, is the last-resort operation.
An efficient way to organize the configuration file is by handling Syslog-identifiers from the most verbose to the least verbose, and for each one, filter-out useless entries based on the `PRIORITY` (which is an integer number) whenever it is possible.
In short, filtering on the actual message, while not entirely avoidable, is the last-resort operation.

NOTE: I used to group my log entries (and Pyruse execution chains) by `_SYSTEMD_UNIT`, which seemed logical at the time.
However, for some reason, there is some “leaking” of logs from some units to others; for example, I had Nginx logs appearing in the Exim `_SYSTEMD_UNIT`… The reason probably lies somewhere in inter-process communication, or with the launching of external commands.
Anyway, I found that grouping by `SYSLOG_IDENTIFIER` actually gives better results:

* `SYSLOG_IDENTIFIER` names are shorter than `_SYSTEMD_UNIT` names, hence probably quicker to compare `:-p`
* Several `_SYSTEMD_UNIT` names from generic units (like `unit-name@instance-name`) end up into the same `SYSLOG_IDENTIFIER`, which allows to occasionaly replace `filter_pcre` with `filter_equals`.
* A single program often does several tasks, and `SYSLOG_IDENTIFIER` reflects this diversity, which makes writing rules much easier.
For example, Pyruse sends emails using msmtp; I do not care about `msmtp`’s logs, but I do about `pyruse`’s. Filtering-out logs from the `msmtp` `SYSLOG_IDENTIFIER` is much easier to do than getting rid of email-related logs from the `pyruse.service` systemd unit.

An [example based on the above statistics](../extra/examples/full_pyruse.json) is available in the `extra/examples/` source directory.

+ 1
- 1
doc/dnat.md View File

@@ -4,7 +4,7 @@

Pyruse provides two actions, namely `action_dnatCapture` and `action_dnatReplace`, that work together towards a single goal: giving to Pyruse’s filters and actions the illusion that client connections come directly to the examined services, instead of going through a firewall Network Address Translation or a reverse-proxy.

If for example you run a Prosody XMPP server (or anything that does not handle the [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt)) behind an HAProxy reverse-proxy, or if all your network connections go through a NAT’ing proxy in your DMZ, then the following happens: the services report your proxy as being the client, ie. usually `127.0.0.1` (same machine) or your LAN’s gateway IP.
If for example you run a Prosody XMPP server (or anything that does not handle the [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt)) behind an HAProxy reverse-proxy, or if all your network connections go through a NAT’ing router in your DMZ, then the following happens: the services report your proxy as being the client, ie. usually `127.0.0.1` (same machine) or your LAN’s router IP.

Here is a simplified illustration of the network configuration:


+ 3
- 1
doc/install.md View File

@@ -5,7 +5,7 @@ The software requirements are:
* a modern systemd-based Linux operating system (eg. [Archlinux](https://archlinux.org/)- or [Fedora](https://getfedora.org/)-based distributions);
* python, at least version 3.4 (or [more, depending on the modules](intro_tech.md) being used);
* [python-systemd](https://www.freedesktop.org/software/systemd/python-systemd/journal.html);
* [nftables](http://wiki.nftables.org/) _if_ IP address bans are to be managed;
* [nftables](http://wiki.nftables.org/) or [ipset](http://ipset.netfilter.org/) _if_ IP address bans are to be managed;
* a sendmail-like program _if_ emails are wanted.

Besides, getting the software requires [Git](http://git-scm.com/), and packaging it requires [python-setuptools](http://pypi.python.org/pypi/setuptools).
@@ -45,4 +45,6 @@ To install Pyruse on the system, run these commands as root, in the root directo
The `package` line is the one that actually alters the system. Until Pyruse is packaged for your operating system, you may want to change this line to `checkinstall package`. [Checkinstall](https://en.wikipedia.org/wiki/CheckInstall) should be able to turn your Pyruse installation into a native Linux package.

Then, to run Pyruse, start (and enable) `pyruse.service`.

If you use nftables bans, you should also start (and enable) `pyruse-boot@action_nftBan.service`.
Likewise, if you use ipset bans, you should start (and enable) `pyruse-boot@action_ipsetBan.service`.

+ 1
- 1
doc/intro_func.md View File

@@ -2,7 +2,7 @@

Everyone knows [Fail2ban](http://www.fail2ban.org/).
This program is excellent at what it does, which is matching patterns in a number of log files, and keeping track of counters in order to launch actions (ban an IP, send an email…) when thresholds are crossed.
[Fail2ban is extremely customizable](http://yalis.fr/cms/index.php/post/2014/11/02/Migrate-from-DenyHosts-to-Fail2ban)… to a point; and to my knowledge it cannot benefit from all the features of modern-day logging with systemd.
[Fail2ban is extremely customizable](http://yalis.fr/cms/index.php/post/2014/11/02/Migrate-from-DenyHosts-to-Fail2ban)… to a point; to my knowledge it cannot benefit from all the features of modern-day logging with systemd.

Then, there is the less-known and aging [Epylog](http://freshmeat.sourceforge.net/projects/epylog/); several programs exist with the same features.
Just like Fail2ban, this program continuously scans the logs to do pattern matching.

+ 4
- 2
doc/intro_tech.md View File

@@ -31,8 +31,10 @@ Pyruse is split into several Python files:
* `workflow.py`: This is where the configuration file get translated into actual execution chains, linked together into a single static workflow.
* `module.py`: Whenever the workflow needs to add a filter or an action to an execution chain, this module finds it in the filesystem.
* `base.py`: All actions and filters inherit from the `Action` and `Filter` classes defined in there; they act as an abstraction layer between the workflow and the modules.
* `counter.py`: This utility class is parent to modules that manage a counter (Python supports multiple-inheritance).
* `email.py`: This utility class is parent to modules that send emails (Python supports multiple-inheritance).
* `ban.py`: This utility class is parent to modules that ban IP addresses using [Netfilter](https://netfilter.org/) (Python supports multiple-inheritance).
* `counter.py`: This utility class is parent to modules that manage a counter.
* `dnat.py`: This file contains utility parent classes for actions that try and restore the actual client IP addresses.
* `email.py`: This utility class is parent to modules that send emails.

All else is actions and filters…
Some are delivered with Pyruse itself; [more can be added](customize.md).

+ 104
- 8
doc/logandban.md View File

@@ -30,9 +30,22 @@ Here are some examples:
## Ban IP addresses after they misbehaved

Linux provides a number of firewall solutions: [iptables](http://www.netfilter.org/), its successor [nftables](http://wiki.nftables.org/), and many iptables frontends like [Shorewall](http://www.shorewall.net/) or RedHat’s [firewalld](http://www.firewalld.org/).
For Pyruse, **nftables** was chosen, because it is modern and light-weight, and provides interesting features.

Action `action_nftBan` takes three mandatory arguments:
For Pyruse, **nftables** was initially chosen, because it is modern and light-weight, and provides interesting features. Besides, only nftables is actually tested in real use.
Now, an **ipset**-based [alternative is also available](#iptables-with-ipset).

### nftables

Action `action_nftBan` requires that nftables is installed.
In addition, the binary to run (and parameters if needed) should be set in the configuration file; here is the default value:

```json
"nftBan": {
"nft": [ "/usr/bin/nft" ]
}
```

This action takes three mandatory arguments:

* `nftSetIPv4` and `nftSetIPv6` are the nftables sets where IP addresses will be added. The IP address is considered to be IPv6 if it contains at least one colon (`:`) character, else it is supposed to be IPv4.
* `IP` gives the name of the entry field to read, in order to get the IP address.
@@ -88,7 +101,7 @@ Here are examples:
}
```

### List the currently banned addresses
#### List the currently banned addresses

To see what IP addresses are currently banned, here is the `nft` command:

@@ -114,7 +127,7 @@ table ip Inet4 {

_Note_: The un-rounded timeouts are post-reboot restored bans.

### Un-ban an IP address
#### Un-ban an IP address

It is bound to happen some day: you will want to un-ban a banned IP address.

@@ -129,7 +142,7 @@ To avoid that, also delete the corresponding record from the `action_nftBan.py.j

To go further, you could tweak your configuration, so that your trusted IP addresses never reach `action_nftBan`.

### Manual ban of an IP address
#### Manual ban of an IP address

To add a ban yourself, run a command like this:

@@ -139,16 +152,99 @@ $ sudo nft 'add element ip Inet4 ssh_ban {192.168.1.1 timeout 5d}

The `timeout …` part can be omitted to add a permanent ban. The timeout can be any combination of days (`d`), hours (`h`), minutes (`m`), and seconds (`s`), eg. “`3d31m16s`”.

In order to make the ban persistent across reboots, a corresponding record should also be appended to the `action_nftBan.py.json` file in Pyruse’s [storage directory](conffile.md) (the IP address, nft Set, days, hours, minutes, seconds, and actual path to the file should be adapted to your needs):
In order to make the ban persistent across reboots, a corresponding record should also be appended to the `action_nftBan.py.json` file in Pyruse’s [storage directory](conffile.md) (the IP address, Nftables Set, days, hours, minutes, seconds, and actual path to the file should be adapted to your needs):

* either a time-limited ban:

```bash
$ sudo sed -i "\$s/.\$/$(date +', {"IP": "192.168.1.1", "nfSet": "ip Inet4 ssh_ban", "timestamp": %s.000000}' -d 'now +3day +31minute +16second')]/" /var/lib/pyruse/action_nftBan.py.json
```

* or an unlimited ban:

```bash
$ sudo sed -i '$s/.$/, {"IP": "192.168.1.1", "nfSet": "ip Inet4 ssh_ban", "timestamp": 0}]/' /var/lib/pyruse/action_nftBan.py.json
```

### iptables with ipset

Action `action_ipsetBan` requires that ipset and iptables are installed.
In addition, the ipset binary to run (and parameters if needed) should be set in the configuration file; here is the default value:

```json
"ipsetBan": {
"ipset": [ "/usr/bin/ipset", "-exist", "-quiet" ]
}
```

This action works exactly [like `action_nftBan`](#nftables), except parameters `nftSetIPv4` and `nftSetIPv6` are named `ipSetIPv4` and `ipSetIPv6` instead.

The name of the set in the `ipSetIPv4` parameter must have been created before running Pyruse, with:

```bash
$ sudo ipset create SET_NAME hash:ip family inet hashsize 1024 maxelem 65535
```

Likewise, the set given by `ipSetIPv6` must have been created before running Pyruse, with:

```bash
$ sudo ipset create SET_NAME hash:ip family inet6 hashsize 1024 maxelem 65535
```

Here are examples of usage for this action:

```json
{
"action": "action_ipsetBan",
"args": { "IP": "thatIP", "banSeconds": 86400, "ipSetIPv4": "mail_ban4", "ipSetIPv6": "mail_ban6" }
}

{
"action": "action_ipsetBan",
"args": { "IP": "thatIP", "ipSetIPv4": "sshd_ban4", "ipSetIPv6": "sshd_ban6" }
}
```

#### List the currently banned addresses

To see what IP addresses are currently banned, here is the `ipset` command:

```bash
$ sudo ipset list mail_ban4'
```

#### Un-ban an IP address

To remove an IP address from a set, here is the `ipset` command:

```bash
$ sudo ipset del mail_ban4 10.0.0.10'
```

To make the change persistent across reboots, also delete the corresponding record from the `action_ipsetBan.py.json` file in Pyruse’s [storage directory](conffile.md).

To go further, you could tweak your configuration, so that your trusted IP addresses never reach `action_ipsetBan`.

#### Manual ban of an IP address

To add a ban yourself, run a command like this:

```bash
$ sudo ipset add ssh_ban4 192.168.1.1 timeout 261076
```

The `timeout …` part can be omitted to add a permanent ban; otherwise it is a number of seconds.

In order to make the ban persistent across reboots, a corresponding record should also be appended to the `action_ipsetBan.py.json` file in Pyruse’s [storage directory](conffile.md) (the IP address, Nftables Set, days, hours, minutes, seconds, and actual path to the file should be adapted to your needs):

* either a time-limited ban:

```bash
$ sudo sed -i "\$s/.\$/$(date +', {"IP": "192.168.1.1", "nftSet": "ip Inet4 ssh_ban", "timestamp": %s.000000}' -d 'now +3day +31minute +16second')]/" /var/lib/pyruse/action_nftBan.py.json
$ sudo sed -i "\$s/.\$/$(date +', {"IP": "192.168.1.1", "nfSet": "ssh_ban4", "timestamp": %s.000000}' -d 'now +3day +31minute +16second')]/" /var/lib/pyruse/action_ipsetBan.py.json
```

* or an unlimited ban:

```bash
$ sudo sed -i '$s/.$/, {"IP": "192.168.1.1", "nftSet": "ip Inet4 ssh_ban", "timestamp": 0}]/' /var/lib/pyruse/action_nftBan.py.json
$ sudo sed -i '$s/.$/, {"IP": "192.168.1.1", "nfSet": "ssh_ban4", "timestamp": 0}]/' /var/lib/pyruse/action_ipsetBan.py.json
```

+ 468
- 331
extra/examples/full_pyruse.json
File diff suppressed because it is too large
View File


+ 9
- 5
extra/examples/get-systemd-stats.sh View File

@@ -1,11 +1,15 @@
#!/bin/bash
# $1 (optional): systemd-nspawn machine name
# $1 (optional): grouping criteria: SYSLOG_IDENTIFIER (default) or _SYSTEMD_UNIT
# $+ (optional): journalctl options (-M machine, -S date…)

CRIT=${1:-SYSLOG_IDENTIFIER}
shift

{
printf 'Units\tTotal\tP7\tP6\tP5\tP4\tP3\tP2\tP1\tP0\n'
sudo journalctl ${1:+-M "$1"} -o json-pretty --output-fields=_SYSTEMD_UNIT,PRIORITY \
printf '%s\tTotal\tP7\tP6\tP5\tP4\tP3\tP2\tP1\tP0\n' $CRIT
sudo journalctl "$@" -o json-pretty --output-fields=${CRIT},PRIORITY \
| tr -d $'"\t, ' \
| awk -F: -vOFS=: '
| awk -F: -vOFS=: -vCRIT=$CRIT '
/^\{/ {
u = ""
p = -1
@@ -13,7 +17,7 @@
$1 == "PRIORITY" {
p = $2
}
$1 == "_SYSTEMD_UNIT" {
$1 == CRIT {
u = gensub(\
"@.*(\\.[^.]*)$",\
"@*\\1",\

+ 3
- 0
extra/systemd/action_ipsetBan.conf View File

@@ -0,0 +1,3 @@
[Unit]
Requires=iptables.service
After=iptables.service

+ 37
- 0
pyruse/actions/action_ipsetBan.py View File

@@ -0,0 +1,37 @@
# pyruse is intended as a replacement to both fail2ban and epylog
# Copyright © 2017–2018 Y. Gablin
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
import os
import subprocess
from pyruse import ban, base, config

class Action(base.Action, ban.NetfilterBan):
_storage = config.Config().asMap().get("storage", "/var/lib/pyruse") \
+ "/" + os.path.basename(__file__) + ".json"
_ipset = config.Config().asMap().get("ipsetBan", {}).get("ipset", ["/usr/bin/ipset", "-exist", "-quiet"])

def __init__(self, args):
base.Action.__init__(self)
ban.NetfilterBan.__init__(self, Action._storage)
if args is None:
return # on-boot configuration
ipv4Set = args["ipSetIPv4"]
ipv6Set = args["ipSetIPv6"]
field = args["IP"]
banSeconds = args.get("banSeconds", None)
self.initSelf(ipv4Set, ipv6Set, field, banSeconds)

def act(self, entry):
ban.NetfilterBan.act(self, entry)

def setBan(self, nfSet, ip, seconds):
cmd = list(Action._ipset)
cmd.extend(["add", nfSet, ip])
if seconds > 0:
cmd.extend(["timeout", str(seconds)])
subprocess.run(cmd)

def cancelBan(self, nfSet, ip):
cmd = list(Action._ipset)
cmd.extend(["del", nfSet, ip])
subprocess.run(cmd)

+ 17
- 75
pyruse/actions/action_nftBan.py View File

@@ -1,97 +1,39 @@
# pyruse is intended as a replacement to both fail2ban and epylog
# Copyright © 2017–2018 Y. Gablin
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
import datetime
import json
import os
import subprocess
from pyruse import base, config
from pyruse import ban, base, config

class Action(base.Action):
class Action(base.Action, ban.NetfilterBan):
_storage = config.Config().asMap().get("storage", "/var/lib/pyruse") \
+ "/" + os.path.basename(__file__) + ".json"
_nft = config.Config().asMap().get("nftBan", {}).get("nft", ["/usr/bin/nft"])

def __init__(self, args):
super().__init__()
base.Action.__init__(self)
ban.NetfilterBan.__init__(self, Action._storage)
if args is None:
return # on-boot configuration
self.ipv4Set = args["nftSetIPv4"]
self.ipv6Set = args["nftSetIPv6"]
self.field = args["IP"]
self.banSeconds = args.get("banSeconds", None)
ipv4Set = args["nftSetIPv4"]
ipv6Set = args["nftSetIPv6"]
field = args["IP"]
banSeconds = args.get("banSeconds", None)
self.initSelf(ipv4Set, ipv6Set, field, banSeconds)

def act(self, entry):
ip = entry[self.field]
nftSet = self.ipv6Set if ":" in ip else self.ipv4Set
newBan = {"IP": ip, "nftSet": nftSet}
ban.NetfilterBan.act(self, entry)

now = datetime.datetime.utcnow()
bans = []
previousTS = None
try:
with open(Action._storage) as dataFile:
for ban in json.load(dataFile):
if ban["timestamp"] > 0 and ban["timestamp"] <= now.timestamp():
continue
elif {k: ban[k] for k in newBan.keys()} == newBan:
# should not happen, since the IP is banned…
previousTS = ban["timestamp"]
else:
bans.append(ban)
except IOError:
pass # new file

if previousTS is not None:
try:
cmd = list(Action._nft)
cmd.append("delete element %s {%s}" % (nftSet, ip))
subprocess.run(cmd)
except Exception:
pass # too late: not a problem

if self.banSeconds:
until = now + datetime.timedelta(seconds = self.banSeconds)
newBan["timestamp"] = until.timestamp()
timeout = self.banSeconds
else:
newBan["timestamp"] = 0
timeout = 0

self._doBan(timeout, ip, nftSet)
bans.append(newBan)
with open(Action._storage, "w") as dataFile:
json.dump(bans, dataFile)

def boot(self):
now = datetime.datetime.utcnow()
bans = []
try:
with open(Action._storage) as dataFile:
for ban in json.load(dataFile):
if ban["timestamp"] == 0:
self._doBan(0, ban["IP"], ban["nftSet"])
bans.append(ban)
elif ban["timestamp"] <= now.timestamp():
continue
else:
until = datetime.datetime.utcfromtimestamp(ban["timestamp"])
timeout = (until - now).total_seconds()
self._doBan(int(timeout), ban["IP"], ban["nftSet"])
bans.append(ban)
except IOError:
pass # no file

with open(Action._storage, "w") as dataFile:
json.dump(bans, dataFile)

def _doBan(self, seconds, ip, nftSet):
if seconds < 0:
return # can happen when the threshold is crossed while computing the duration
def setBan(self, nfSet, ip, seconds):
if seconds == 0:
timeout = ""
else:
timeout = " timeout %ss" % seconds
cmd = list(Action._nft)
cmd.append("add element %s {%s%s}" % (nftSet, ip, timeout))
cmd.append("add element %s {%s%s}" % (nfSet, ip, timeout))
subprocess.run(cmd)

def cancelBan(self, nfSet, ip):
cmd = list(Action._nft)
cmd.append("delete element %s {%s}" % (nfSet, ip))
subprocess.run(cmd)

+ 85
- 0
pyruse/ban.py View File

@@ -0,0 +1,85 @@
# pyruse is intended as a replacement to both fail2ban and epylog
# Copyright © 2017–2018 Y. Gablin
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
import abc
import datetime
import json

class NetfilterBan(abc.ABC):
def __init__(self, storage):
self.storage = storage

def initSelf(self, ipv4Set, ipv6Set, ipField, banSeconds):
self.ipv4Set = ipv4Set
self.ipv6Set = ipv6Set
self.field = ipField
self.banSeconds = banSeconds

@abc.abstractmethod
def setBan(self, nfSet, ip, seconds):
pass

@abc.abstractmethod
def cancelBan(self, nfSet, ip):
pass

def act(self, entry):
ip = entry[self.field]
nfSet = self.ipv6Set if ":" in ip else self.ipv4Set
newBan = {"IP": ip, "nfSet": nfSet}

now = datetime.datetime.utcnow()
bans = []
previousTS = None
try:
with open(self.storage) as dataFile:
for ban in json.load(dataFile):
if ban["timestamp"] > 0 and ban["timestamp"] <= now.timestamp():
continue
elif {k: ban[k] for k in newBan.keys()} == newBan:
# should not happen, since the IP is banned…
previousTS = ban["timestamp"]
else:
bans.append(ban)
except IOError:
pass # new file

if previousTS is not None:
try:
self.cancelBan(nfSet, ip)
except Exception:
pass # too late: not a problem

if self.banSeconds:
until = now + datetime.timedelta(seconds = self.banSeconds)
newBan["timestamp"] = until.timestamp()
timeout = self.banSeconds
else:
newBan["timestamp"] = 0
timeout = 0

self.setBan(nfSet, ip, timeout)
bans.append(newBan)
with open(self.storage, "w") as dataFile:
json.dump(bans, dataFile)

def boot(self):
now = int(datetime.datetime.utcnow().timestamp())
bans = []
try:
with open(self.storage) as dataFile:
for ban in json.load(dataFile):
if ban["timestamp"] == 0:
self.setBan(ban["nfSet"], ban["IP"], 0)
bans.append(ban)
elif int(ban["timestamp"]) <= now:
continue
else:
timeout = int(ban["timestamp"]) - now
self.setBan(ban["nfSet"], ban["IP"], timeout)
bans.append(ban)
except IOError:
pass # no file

with open(self.storage, "w") as dataFile:
json.dump(bans, dataFile)

+ 147
- 0
tests/action_ipsetBan.py View File

@@ -0,0 +1,147 @@
# pyruse is intended as a replacement to both fail2ban and epylog
# Copyright © 2017–2018 Y. Gablin
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
import json
import os
import time
from pyruse.actions.action_ipsetBan import Action

ipBanCmd = "ipsetBan.cmd"
ipBanState = "action_ipsetBan.py.json"

def _clean():
if os.path.exists(ipBanCmd):
os.remove(ipBanCmd)
if os.path.exists(ipBanState):
os.remove(ipBanState)

def whenBanIPv4ThenAddToIPv4Set():
_clean()
Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban"}).act({"ip": "10.0.0.1"})
assert os.path.exists(ipBanCmd)
assert os.path.exists(ipBanState)
nbLines = 0
with open(ipBanCmd, "rt") as c:
for line in c:
assert line == "add I4ban 10.0.0.1\n", line
nbLines += 1
assert nbLines == 1, nbLines
nbBans = 0
with open(ipBanState) as s:
for ban in json.load(s):
assert ban["IP"] == "10.0.0.1" and ban["nfSet"] == "I4ban", str(ban)
nbBans += 1
assert nbBans == 1, nbBans
_clean()

def whenBanIPv6ThenAddToIPv6Set():
_clean()
Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban"}).act({"ip": "::1"})
assert os.path.exists(ipBanCmd)
assert os.path.exists(ipBanState)
nbLines = 0
with open(ipBanCmd, "rt") as c:
for line in c:
assert line == "add I6ban ::1\n", line
nbLines += 1
assert nbLines == 1, nbLines
nbBans = 0
with open(ipBanState) as s:
for ban in json.load(s):
assert ban["IP"] == "::1" and ban["nfSet"] == "I6ban", str(ban)
nbBans += 1
assert nbBans == 1, nbBans
_clean()

def whenBanTwoIPThenTwoLinesInState():
_clean()
action = Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban"})
action.act({"ip": "10.0.0.1"})
action.act({"ip": "::1"})
action.act({"ip": "10.0.0.1"})
assert os.path.exists(ipBanState)
nbBans = 0
with open(ipBanState) as s:
for ban in json.load(s):
if ban["IP"] == "10.0.0.1":
assert ban["nfSet"] == "I4ban", str(ban)
elif ban["IP"] == "::1":
assert ban["nfSet"] == "I6ban", str(ban)
else:
assert false, str(ban)
nbBans += 1
assert nbBans == 2, nbBans
_clean()

def whenBanAnewThenNoDuplicate():
_clean()
action = Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban"})
action.act({"ip": "10.0.0.1"})
action.act({"ip": "10.0.0.1"})
assert os.path.exists(ipBanCmd)
assert os.path.exists(ipBanState)
lineCount = 0
with open(ipBanCmd, "rt") as c:
for line in c:
lineCount += 1
if lineCount == 1:
assert line == "add I4ban 10.0.0.1\n", line
elif lineCount == 2:
assert line == "del I4ban 10.0.0.1\n", line
elif lineCount == 3:
assert line == "add I4ban 10.0.0.1\n", line
assert lineCount == 3, lineCount
nbBans = 0
with open(ipBanState) as s:
for ban in json.load(s):
if ban["IP"] == "10.0.0.1":
assert ban["nfSet"] == "I4ban", str(ban)
nbBans += 1
assert nbBans == 1, nbBans
_clean()

def whenFinishedBanThenAsIfNotThere():
_clean()
action = Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban", "banSeconds": 1})
action.act({"ip": "10.0.0.1"})
time.sleep(1)
action.act({"ip": "10.0.0.1"})
assert os.path.exists(ipBanCmd)
lineCount = 0
with open(ipBanCmd, "rt") as c:
for line in c:
lineCount += 1
if lineCount == 1:
assert line == "add I4ban 10.0.0.1 timeout 1\n", line
elif lineCount == 2:
assert line == "add I4ban 10.0.0.1 timeout 1\n", line
assert lineCount == 2, lineCount
_clean()

def whenUnfinishedBanThenTimeoutReset():
_clean()
action = Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban", "banSeconds": 2})
action.act({"ip": "10.0.0.1"})
time.sleep(1)
action.act({"ip": "10.0.0.1"})
assert os.path.exists(ipBanCmd)
lineCount = 0
with open(ipBanCmd, "rt") as c:
for line in c:
lineCount += 1
if lineCount == 1:
assert line == "add I4ban 10.0.0.1 timeout 2\n", line
elif lineCount == 2:
assert line == "del I4ban 10.0.0.1\n", line
elif lineCount == 3:
assert line == "add I4ban 10.0.0.1 timeout 2\n", line
assert lineCount == 3, lineCount
_clean()

def unitTests():
whenBanIPv4ThenAddToIPv4Set()
whenBanIPv6ThenAddToIPv6Set()
whenBanTwoIPThenTwoLinesInState()
whenBanAnewThenNoDuplicate()
whenFinishedBanThenAsIfNotThere()
whenUnfinishedBanThenTimeoutReset()

+ 5
- 5
tests/action_nftBan.py View File

@@ -29,7 +29,7 @@ def whenBanIPv4ThenAddToIPv4Set():
nbBans = 0
with open(nftBanState) as s:
for ban in json.load(s):
assert ban["IP"] == "10.0.0.1" and ban["nftSet"] == "ip I4 ban", str(ban)
assert ban["IP"] == "10.0.0.1" and ban["nfSet"] == "ip I4 ban", str(ban)
nbBans += 1
assert nbBans == 1, nbBans
_clean()
@@ -48,7 +48,7 @@ def whenBanIPv6ThenAddToIPv6Set():
nbBans = 0
with open(nftBanState) as s:
for ban in json.load(s):
assert ban["IP"] == "::1" and ban["nftSet"] == "ip6 I6 ban", str(ban)
assert ban["IP"] == "::1" and ban["nfSet"] == "ip6 I6 ban", str(ban)
nbBans += 1
assert nbBans == 1, nbBans
_clean()
@@ -64,9 +64,9 @@ def whenBanTwoIPThenTwoLinesInState():
with open(nftBanState) as s:
for ban in json.load(s):
if ban["IP"] == "10.0.0.1":
assert ban["nftSet"] == "ip I4 ban", str(ban)
assert ban["nfSet"] == "ip I4 ban", str(ban)
elif ban["IP"] == "::1":
assert ban["nftSet"] == "ip6 I6 ban", str(ban)
assert ban["nfSet"] == "ip6 I6 ban", str(ban)
else:
assert false, str(ban)
nbBans += 1
@@ -95,7 +95,7 @@ def whenBanAnewThenNoDuplicate():
with open(nftBanState) as s:
for ban in json.load(s):
if ban["IP"] == "10.0.0.1":
assert ban["nftSet"] == "ip I4 ban", str(ban)
assert ban["nfSet"] == "ip I4 ban", str(ban)
nbBans += 1
assert nbBans == 1, nbBans
_clean()

+ 2
- 1
tests/main.py View File

@@ -20,7 +20,7 @@ def main():

# Unit tests
import filter_equals, filter_greaterOrEquals, filter_in, filter_inNetworks, filter_lowerOrEquals, filter_pcre, filter_pcreAny, filter_userExists
import action_counterRaise, action_counterReset, action_dailyReport, action_dnatCapture, action_dnatReplace, action_email, action_log, action_nftBan
import action_counterRaise, action_counterReset, action_dailyReport, action_dnatCapture, action_dnatReplace, action_email, action_ipsetBan, action_log, action_nftBan

filter_equals.unitTests()
filter_greaterOrEquals.unitTests()
@@ -36,6 +36,7 @@ def main():
action_dnatCapture.unitTests()
action_dnatReplace.unitTests()
action_email.unitTests()
action_ipsetBan.unitTests()
action_log.unitTests()
action_nftBan.unitTests()


+ 3
- 0
tests/pyruse.json View File

@@ -87,5 +87,8 @@
"nftBan": {
"nft": [ "/bin/sh", "-c", "echo \"$0\" >>\"nftBan.cmd\"" ]
},
"ipsetBan": {
"ipset": [ "/bin/sh", "-c", "echo \"$0 $*\" >>\"ipsetBan.cmd\"" ]
},
"storage": "."
}

Write Preview
Loading…
Cancel
Save