parent
f3b674ca26
commit
8aaa04389f
|
@ -0,0 +1,12 @@
|
|||
# Changelog
|
||||
|
||||
This file is not intended as a dupplicate of Git logs.
|
||||
Its purpose is to warn of important changes between version, that users should be aware of.
|
||||
|
||||
## v2.0
|
||||
|
||||
After this version is installed, the following command should be run on the `action_nftBan.py.json` file:
|
||||
|
||||
```bash
|
||||
$ sudo sed -i s/nftSet/nfSet/g action_nftBan.py.json
|
||||
```
|
|
@ -27,6 +27,8 @@ For example, to my knowledge, there is no equivalent in any tool of the same sca
|
|||
Pyruse is [packaged for Archlinux](https://aur.archlinux.org/packages/pyruse/).
|
||||
For other distributions, please [read the manual installation instructions](doc/install.md).
|
||||
|
||||
Whenever your upgrade Pyruse, make sure to check the [Changelog](Changelog.md).
|
||||
|
||||
## Configuration
|
||||
|
||||
The `/etc/pyruse` directory is where system-specific files are looked-for:
|
||||
|
@ -47,7 +49,7 @@ For more in-depth documentation, please refer to these pages:
|
|||
- [the built-in filters](doc/builtinfilters.md)
|
||||
- [the counter-based actions](doc/counters.md)
|
||||
- [the DNAT-related actions](doc/dnat.md)
|
||||
- [the actions that log and ban](doc/logandban.md)
|
||||
- [the `action_noop` module](doc/noop.md)
|
||||
- [the `action_email` module](doc/action_email.md)
|
||||
- [the `action_dailyReport` module](doc/action_dailyReport.md)
|
||||
- [the `action_nftBan` and `action_log` modules](doc/logandban.md)
|
||||
|
|
|
@ -1,33 +1,46 @@
|
|||
# Configuration tips
|
||||
|
||||
In contrast with legacy log parsers, Pyruse works with structured [systemd-journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html). This allows for better performance, since targeted comparisons become possible.
|
||||
In contrast with legacy log parsers, Pyruse works with structured [systemd-journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html).
|
||||
This allows for better performance, since targeted comparisons become possible.
|
||||
|
||||
The general intent, when writing the configuration file, should be to handle the log entries that appear the most often first, in as few steps as possible. For example, I ran some stats on my server before writing my own configuration file; I got:
|
||||
The general intent, when writing the configuration file, should be to handle the log entries that appear the most often first, in as few steps as possible.
|
||||
For example, I ran some stats on my server on the log entries of the past week; I got:
|
||||
|
||||
| systemd units | number of journal entries |
|
||||
| ------------------------------- | -------------------------:|
|
||||
| `prosody.service` | 518019 |
|
||||
| `gitea.service` | 329389 |
|
||||
| `uwsgi@nextcloud.service` | 217342 |
|
||||
| `session-*.scope` | 89813 |
|
||||
| `nginx.service` | 80762 |
|
||||
| `dovecot.service` | 61898 |
|
||||
| `exim.service` | 60743 |
|
||||
| `init.scope` | 43021 |
|
||||
| `nextcloud-maintenance.service` | 20775 |
|
||||
| `haproxy.service` | 18445 |
|
||||
| `user@*.service` | 7306 |
|
||||
| `minidlna.service` | 6032 |
|
||||
| `loolwsd.service` | 5797 |
|
||||
| `sshd.service` | 4959 |
|
||||
| `spamassassin-update.service` | 2383 |
|
||||
| `systemd-nspawn@*.service` | 1497 |
|
||||
| `nslcd.service` | 867 |
|
||||
| `nfs-mountd.service` | 723 |
|
||||
| `systemd-logind.service` | 696 |
|
||||
| `nfs-server.service` | 293 |
|
||||
| `systemd-networkd.service` | 121 |
|
||||
| misc. units with < 100 entries | |
|
||||
| SYSLOG_IDENTIFIER | number of journal entries |
|
||||
| ----------------------- | -------------------------:|
|
||||
| `uwsgi` (for Nextcloud) | 55930 |
|
||||
| `gitea` | 38923 |
|
||||
| `prosody` | 25596 |
|
||||
| `haproxy` | 21877 |
|
||||
| `postgres` | 12990 |
|
||||
| `nginx` | 12808 |
|
||||
| `dovecot` | 7062 |
|
||||
| `exim` | 2540 |
|
||||
| `systemd` | 1997 |
|
||||
| `su` | 1458 |
|
||||
| `ownCloud` (Nextcloud) | 1067 |
|
||||
| `sshd` | 1051 |
|
||||
| `mandb` | 953 |
|
||||
| `spamd` | 855 |
|
||||
| `pyruse` | 615 |
|
||||
| `kernel` | 420 |
|
||||
| `msmtp` | 295 |
|
||||
| `sa-compile` | 255 |
|
||||
| `ansible-*` | 103 |
|
||||
| `systemd-logind` | 102 |
|
||||
| `python` | 78 |
|
||||
| `rpc.mountd` | 52 |
|
||||
| `ldapwhoami` | 42 |
|
||||
| `prosody_auth` | 42 |
|
||||
| `minidlnad` | 39 |
|
||||
| `kill` | 28 |
|
||||
| `sudo` | 26 |
|
||||
| `loolwsd` | 17 |
|
||||
| `exportfs` | 15 |
|
||||
| `dehydrated` | 6 |
|
||||
| `sa-update` | 5 |
|
||||
| `nslcd` | 4 |
|
||||
| `rpc.idmapd` | 1 |
|
||||
|
||||
For reference, here is the command that gives these statistics:
|
||||
|
||||
|
@ -37,6 +50,16 @@ $ bash ./extra/examples/get-systemd-stats.sh >~/systemd-units.stats.tsv
|
|||
|
||||
One should also remember, that numeric comparison are faster that string comparison, which in turn are faster than regular expression pattern-matching. Further more, some log entries are not worth checking for, because they are too rare: it costs more to grab them with filters (that most log entries will have to pass through), than letting them get caught by the catch-all last execution chain, which typically uses the `action_dailyReport` module.
|
||||
|
||||
An efficient way to organize the configuration file is by handling units from the most verbose to the least verbose, and for each unit, filter-out useless entries based on the `PRIORITY` (which is an integer number) whenever it is possible. In short, filtering on the actual message, while not entirely avoidable, is the last-resort operation.
|
||||
An efficient way to organize the configuration file is by handling Syslog-identifiers from the most verbose to the least verbose, and for each one, filter-out useless entries based on the `PRIORITY` (which is an integer number) whenever it is possible.
|
||||
In short, filtering on the actual message, while not entirely avoidable, is the last-resort operation.
|
||||
|
||||
NOTE: I used to group my log entries (and Pyruse execution chains) by `_SYSTEMD_UNIT`, which seemed logical at the time.
|
||||
However, for some reason, there is some “leaking” of logs from some units to others; for example, I had Nginx logs appearing in the Exim `_SYSTEMD_UNIT`… The reason probably lies somewhere in inter-process communication, or with the launching of external commands.
|
||||
Anyway, I found that grouping by `SYSLOG_IDENTIFIER` actually gives better results:
|
||||
|
||||
* `SYSLOG_IDENTIFIER` names are shorter than `_SYSTEMD_UNIT` names, hence probably quicker to compare `:-p`
|
||||
* Several `_SYSTEMD_UNIT` names from generic units (like `unit-name@instance-name`) end up into the same `SYSLOG_IDENTIFIER`, which allows to occasionaly replace `filter_pcre` with `filter_equals`.
|
||||
* A single program often does several tasks, and `SYSLOG_IDENTIFIER` reflects this diversity, which makes writing rules much easier.
|
||||
For example, Pyruse sends emails using msmtp; I do not care about `msmtp`’s logs, but I do about `pyruse`’s. Filtering-out logs from the `msmtp` `SYSLOG_IDENTIFIER` is much easier to do than getting rid of email-related logs from the `pyruse.service` systemd unit.
|
||||
|
||||
An [example based on the above statistics](../extra/examples/full_pyruse.json) is available in the `extra/examples/` source directory.
|
||||
|
|
|
@ -4,7 +4,7 @@
|
|||
|
||||
Pyruse provides two actions, namely `action_dnatCapture` and `action_dnatReplace`, that work together towards a single goal: giving to Pyruse’s filters and actions the illusion that client connections come directly to the examined services, instead of going through a firewall Network Address Translation or a reverse-proxy.
|
||||
|
||||
If for example you run a Prosody XMPP server (or anything that does not handle the [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt)) behind an HAProxy reverse-proxy, or if all your network connections go through a NAT’ing proxy in your DMZ, then the following happens: the services report your proxy as being the client, ie. usually `127.0.0.1` (same machine) or your LAN’s gateway IP.
|
||||
If for example you run a Prosody XMPP server (or anything that does not handle the [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt)) behind an HAProxy reverse-proxy, or if all your network connections go through a NAT’ing router in your DMZ, then the following happens: the services report your proxy as being the client, ie. usually `127.0.0.1` (same machine) or your LAN’s router IP.
|
||||
|
||||
Here is a simplified illustration of the network configuration:
|
||||
|
||||
|
|
|
@ -5,7 +5,7 @@ The software requirements are:
|
|||
* a modern systemd-based Linux operating system (eg. [Archlinux](https://archlinux.org/)- or [Fedora](https://getfedora.org/)-based distributions);
|
||||
* python, at least version 3.4 (or [more, depending on the modules](intro_tech.md) being used);
|
||||
* [python-systemd](https://www.freedesktop.org/software/systemd/python-systemd/journal.html);
|
||||
* [nftables](http://wiki.nftables.org/) _if_ IP address bans are to be managed;
|
||||
* [nftables](http://wiki.nftables.org/) or [ipset](http://ipset.netfilter.org/) _if_ IP address bans are to be managed;
|
||||
* a sendmail-like program _if_ emails are wanted.
|
||||
|
||||
Besides, getting the software requires [Git](http://git-scm.com/), and packaging it requires [python-setuptools](http://pypi.python.org/pypi/setuptools).
|
||||
|
@ -45,4 +45,6 @@ To install Pyruse on the system, run these commands as root, in the root directo
|
|||
The `package` line is the one that actually alters the system. Until Pyruse is packaged for your operating system, you may want to change this line to `checkinstall package`. [Checkinstall](https://en.wikipedia.org/wiki/CheckInstall) should be able to turn your Pyruse installation into a native Linux package.
|
||||
|
||||
Then, to run Pyruse, start (and enable) `pyruse.service`.
|
||||
|
||||
If you use nftables bans, you should also start (and enable) `pyruse-boot@action_nftBan.service`.
|
||||
Likewise, if you use ipset bans, you should start (and enable) `pyruse-boot@action_ipsetBan.service`.
|
||||
|
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Everyone knows [Fail2ban](http://www.fail2ban.org/).
|
||||
This program is excellent at what it does, which is matching patterns in a number of log files, and keeping track of counters in order to launch actions (ban an IP, send an email…) when thresholds are crossed.
|
||||
[Fail2ban is extremely customizable](http://yalis.fr/cms/index.php/post/2014/11/02/Migrate-from-DenyHosts-to-Fail2ban)… to a point; and to my knowledge it cannot benefit from all the features of modern-day logging with systemd.
|
||||
[Fail2ban is extremely customizable](http://yalis.fr/cms/index.php/post/2014/11/02/Migrate-from-DenyHosts-to-Fail2ban)… to a point; to my knowledge it cannot benefit from all the features of modern-day logging with systemd.
|
||||
|
||||
Then, there is the less-known and aging [Epylog](http://freshmeat.sourceforge.net/projects/epylog/); several programs exist with the same features.
|
||||
Just like Fail2ban, this program continuously scans the logs to do pattern matching.
|
||||
|
|
|
@ -31,8 +31,10 @@ Pyruse is split into several Python files:
|
|||
* `workflow.py`: This is where the configuration file get translated into actual execution chains, linked together into a single static workflow.
|
||||
* `module.py`: Whenever the workflow needs to add a filter or an action to an execution chain, this module finds it in the filesystem.
|
||||
* `base.py`: All actions and filters inherit from the `Action` and `Filter` classes defined in there; they act as an abstraction layer between the workflow and the modules.
|
||||
* `counter.py`: This utility class is parent to modules that manage a counter (Python supports multiple-inheritance).
|
||||
* `email.py`: This utility class is parent to modules that send emails (Python supports multiple-inheritance).
|
||||
* `ban.py`: This utility class is parent to modules that ban IP addresses using [Netfilter](https://netfilter.org/) (Python supports multiple-inheritance).
|
||||
* `counter.py`: This utility class is parent to modules that manage a counter.
|
||||
* `dnat.py`: This file contains utility parent classes for actions that try and restore the actual client IP addresses.
|
||||
* `email.py`: This utility class is parent to modules that send emails.
|
||||
|
||||
All else is actions and filters…
|
||||
Some are delivered with Pyruse itself; [more can be added](customize.md).
|
||||
|
|
112
doc/logandban.md
112
doc/logandban.md
|
@ -30,9 +30,22 @@ Here are some examples:
|
|||
## Ban IP addresses after they misbehaved
|
||||
|
||||
Linux provides a number of firewall solutions: [iptables](http://www.netfilter.org/), its successor [nftables](http://wiki.nftables.org/), and many iptables frontends like [Shorewall](http://www.shorewall.net/) or RedHat’s [firewalld](http://www.firewalld.org/).
|
||||
For Pyruse, **nftables** was chosen, because it is modern and light-weight, and provides interesting features.
|
||||
|
||||
Action `action_nftBan` takes three mandatory arguments:
|
||||
For Pyruse, **nftables** was initially chosen, because it is modern and light-weight, and provides interesting features. Besides, only nftables is actually tested in real use.
|
||||
Now, an **ipset**-based [alternative is also available](#iptables-with-ipset).
|
||||
|
||||
### nftables
|
||||
|
||||
Action `action_nftBan` requires that nftables is installed.
|
||||
In addition, the binary to run (and parameters if needed) should be set in the configuration file; here is the default value:
|
||||
|
||||
```json
|
||||
"nftBan": {
|
||||
"nft": [ "/usr/bin/nft" ]
|
||||
}
|
||||
```
|
||||
|
||||
This action takes three mandatory arguments:
|
||||
|
||||
* `nftSetIPv4` and `nftSetIPv6` are the nftables sets where IP addresses will be added. The IP address is considered to be IPv6 if it contains at least one colon (`:`) character, else it is supposed to be IPv4.
|
||||
* `IP` gives the name of the entry field to read, in order to get the IP address.
|
||||
|
@ -88,7 +101,7 @@ Here are examples:
|
|||
}
|
||||
```
|
||||
|
||||
### List the currently banned addresses
|
||||
#### List the currently banned addresses
|
||||
|
||||
To see what IP addresses are currently banned, here is the `nft` command:
|
||||
|
||||
|
@ -114,7 +127,7 @@ table ip Inet4 {
|
|||
|
||||
_Note_: The un-rounded timeouts are post-reboot restored bans.
|
||||
|
||||
### Un-ban an IP address
|
||||
#### Un-ban an IP address
|
||||
|
||||
It is bound to happen some day: you will want to un-ban a banned IP address.
|
||||
|
||||
|
@ -129,7 +142,7 @@ To avoid that, also delete the corresponding record from the `action_nftBan.py.j
|
|||
|
||||
To go further, you could tweak your configuration, so that your trusted IP addresses never reach `action_nftBan`.
|
||||
|
||||
### Manual ban of an IP address
|
||||
#### Manual ban of an IP address
|
||||
|
||||
To add a ban yourself, run a command like this:
|
||||
|
||||
|
@ -139,16 +152,99 @@ $ sudo nft 'add element ip Inet4 ssh_ban {192.168.1.1 timeout 5d}
|
|||
|
||||
The `timeout …` part can be omitted to add a permanent ban. The timeout can be any combination of days (`d`), hours (`h`), minutes (`m`), and seconds (`s`), eg. “`3d31m16s`”.
|
||||
|
||||
In order to make the ban persistent across reboots, a corresponding record should also be appended to the `action_nftBan.py.json` file in Pyruse’s [storage directory](conffile.md) (the IP address, nft Set, days, hours, minutes, seconds, and actual path to the file should be adapted to your needs):
|
||||
In order to make the ban persistent across reboots, a corresponding record should also be appended to the `action_nftBan.py.json` file in Pyruse’s [storage directory](conffile.md) (the IP address, Nftables Set, days, hours, minutes, seconds, and actual path to the file should be adapted to your needs):
|
||||
|
||||
* either a time-limited ban:
|
||||
|
||||
```bash
|
||||
$ sudo sed -i "\$s/.\$/$(date +', {"IP": "192.168.1.1", "nftSet": "ip Inet4 ssh_ban", "timestamp": %s.000000}' -d 'now +3day +31minute +16second')]/" /var/lib/pyruse/action_nftBan.py.json
|
||||
$ sudo sed -i "\$s/.\$/$(date +', {"IP": "192.168.1.1", "nfSet": "ip Inet4 ssh_ban", "timestamp": %s.000000}' -d 'now +3day +31minute +16second')]/" /var/lib/pyruse/action_nftBan.py.json
|
||||
```
|
||||
|
||||
* or an unlimited ban:
|
||||
|
||||
```bash
|
||||
$ sudo sed -i '$s/.$/, {"IP": "192.168.1.1", "nftSet": "ip Inet4 ssh_ban", "timestamp": 0}]/' /var/lib/pyruse/action_nftBan.py.json
|
||||
$ sudo sed -i '$s/.$/, {"IP": "192.168.1.1", "nfSet": "ip Inet4 ssh_ban", "timestamp": 0}]/' /var/lib/pyruse/action_nftBan.py.json
|
||||
```
|
||||
|
||||
### iptables with ipset
|
||||
|
||||
Action `action_ipsetBan` requires that ipset and iptables are installed.
|
||||
In addition, the ipset binary to run (and parameters if needed) should be set in the configuration file; here is the default value:
|
||||
|
||||
```json
|
||||
"ipsetBan": {
|
||||
"ipset": [ "/usr/bin/ipset", "-exist", "-quiet" ]
|
||||
}
|
||||
```
|
||||
|
||||
This action works exactly [like `action_nftBan`](#nftables), except parameters `nftSetIPv4` and `nftSetIPv6` are named `ipSetIPv4` and `ipSetIPv6` instead.
|
||||
|
||||
The name of the set in the `ipSetIPv4` parameter must have been created before running Pyruse, with:
|
||||
|
||||
```bash
|
||||
$ sudo ipset create SET_NAME hash:ip family inet hashsize 1024 maxelem 65535
|
||||
```
|
||||
|
||||
Likewise, the set given by `ipSetIPv6` must have been created before running Pyruse, with:
|
||||
|
||||
```bash
|
||||
$ sudo ipset create SET_NAME hash:ip family inet6 hashsize 1024 maxelem 65535
|
||||
```
|
||||
|
||||
Here are examples of usage for this action:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "action_ipsetBan",
|
||||
"args": { "IP": "thatIP", "banSeconds": 86400, "ipSetIPv4": "mail_ban4", "ipSetIPv6": "mail_ban6" }
|
||||
}
|
||||
|
||||
{
|
||||
"action": "action_ipsetBan",
|
||||
"args": { "IP": "thatIP", "ipSetIPv4": "sshd_ban4", "ipSetIPv6": "sshd_ban6" }
|
||||
}
|
||||
```
|
||||
|
||||
#### List the currently banned addresses
|
||||
|
||||
To see what IP addresses are currently banned, here is the `ipset` command:
|
||||
|
||||
```bash
|
||||
$ sudo ipset list mail_ban4'
|
||||
```
|
||||
|
||||
#### Un-ban an IP address
|
||||
|
||||
To remove an IP address from a set, here is the `ipset` command:
|
||||
|
||||
```bash
|
||||
$ sudo ipset del mail_ban4 10.0.0.10'
|
||||
```
|
||||
|
||||
To make the change persistent across reboots, also delete the corresponding record from the `action_ipsetBan.py.json` file in Pyruse’s [storage directory](conffile.md).
|
||||
|
||||
To go further, you could tweak your configuration, so that your trusted IP addresses never reach `action_ipsetBan`.
|
||||
|
||||
#### Manual ban of an IP address
|
||||
|
||||
To add a ban yourself, run a command like this:
|
||||
|
||||
```bash
|
||||
$ sudo ipset add ssh_ban4 192.168.1.1 timeout 261076
|
||||
```
|
||||
|
||||
The `timeout …` part can be omitted to add a permanent ban; otherwise it is a number of seconds.
|
||||
|
||||
In order to make the ban persistent across reboots, a corresponding record should also be appended to the `action_ipsetBan.py.json` file in Pyruse’s [storage directory](conffile.md) (the IP address, Nftables Set, days, hours, minutes, seconds, and actual path to the file should be adapted to your needs):
|
||||
|
||||
* either a time-limited ban:
|
||||
|
||||
```bash
|
||||
$ sudo sed -i "\$s/.\$/$(date +', {"IP": "192.168.1.1", "nfSet": "ssh_ban4", "timestamp": %s.000000}' -d 'now +3day +31minute +16second')]/" /var/lib/pyruse/action_ipsetBan.py.json
|
||||
```
|
||||
|
||||
* or an unlimited ban:
|
||||
|
||||
```bash
|
||||
$ sudo sed -i '$s/.$/, {"IP": "192.168.1.1", "nfSet": "ssh_ban4", "timestamp": 0}]/' /var/lib/pyruse/action_ipsetBan.py.json
|
||||
```
|
||||
|
|
File diff suppressed because it is too large
Load Diff
|
@ -1,11 +1,15 @@
|
|||
#!/bin/bash
|
||||
# $1 (optional): systemd-nspawn machine name
|
||||
# $1 (optional): grouping criteria: SYSLOG_IDENTIFIER (default) or _SYSTEMD_UNIT
|
||||
# $+ (optional): journalctl options (-M machine, -S date…)
|
||||
|
||||
CRIT=${1:-SYSLOG_IDENTIFIER}
|
||||
shift
|
||||
|
||||
{
|
||||
printf 'Units\tTotal\tP7\tP6\tP5\tP4\tP3\tP2\tP1\tP0\n'
|
||||
sudo journalctl ${1:+-M "$1"} -o json-pretty --output-fields=_SYSTEMD_UNIT,PRIORITY \
|
||||
printf '%s\tTotal\tP7\tP6\tP5\tP4\tP3\tP2\tP1\tP0\n' $CRIT
|
||||
sudo journalctl "$@" -o json-pretty --output-fields=${CRIT},PRIORITY \
|
||||
| tr -d $'"\t, ' \
|
||||
| awk -F: -vOFS=: '
|
||||
| awk -F: -vOFS=: -vCRIT=$CRIT '
|
||||
/^\{/ {
|
||||
u = ""
|
||||
p = -1
|
||||
|
@ -13,7 +17,7 @@
|
|||
$1 == "PRIORITY" {
|
||||
p = $2
|
||||
}
|
||||
$1 == "_SYSTEMD_UNIT" {
|
||||
$1 == CRIT {
|
||||
u = gensub(\
|
||||
"@.*(\\.[^.]*)$",\
|
||||
"@*\\1",\
|
||||
|
|
|
@ -0,0 +1,3 @@
|
|||
[Unit]
|
||||
Requires=iptables.service
|
||||
After=iptables.service
|
|
@ -0,0 +1,37 @@
|
|||
# pyruse is intended as a replacement to both fail2ban and epylog
|
||||
# Copyright © 2017–2018 Y. Gablin
|
||||
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
|
||||
import os
|
||||
import subprocess
|
||||
from pyruse import ban, base, config
|
||||
|
||||
class Action(base.Action, ban.NetfilterBan):
|
||||
_storage = config.Config().asMap().get("storage", "/var/lib/pyruse") \
|
||||
+ "/" + os.path.basename(__file__) + ".json"
|
||||
_ipset = config.Config().asMap().get("ipsetBan", {}).get("ipset", ["/usr/bin/ipset", "-exist", "-quiet"])
|
||||
|
||||
def __init__(self, args):
|
||||
base.Action.__init__(self)
|
||||
ban.NetfilterBan.__init__(self, Action._storage)
|
||||
if args is None:
|
||||
return # on-boot configuration
|
||||
ipv4Set = args["ipSetIPv4"]
|
||||
ipv6Set = args["ipSetIPv6"]
|
||||
field = args["IP"]
|
||||
banSeconds = args.get("banSeconds", None)
|
||||
self.initSelf(ipv4Set, ipv6Set, field, banSeconds)
|
||||
|
||||
def act(self, entry):
|
||||
ban.NetfilterBan.act(self, entry)
|
||||
|
||||
def setBan(self, nfSet, ip, seconds):
|
||||
cmd = list(Action._ipset)
|
||||
cmd.extend(["add", nfSet, ip])
|
||||
if seconds > 0:
|
||||
cmd.extend(["timeout", str(seconds)])
|
||||
subprocess.run(cmd)
|
||||
|
||||
def cancelBan(self, nfSet, ip):
|
||||
cmd = list(Action._ipset)
|
||||
cmd.extend(["del", nfSet, ip])
|
||||
subprocess.run(cmd)
|
|
@ -1,97 +1,39 @@
|
|||
# pyruse is intended as a replacement to both fail2ban and epylog
|
||||
# Copyright © 2017–2018 Y. Gablin
|
||||
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
from pyruse import base, config
|
||||
from pyruse import ban, base, config
|
||||
|
||||
class Action(base.Action):
|
||||
class Action(base.Action, ban.NetfilterBan):
|
||||
_storage = config.Config().asMap().get("storage", "/var/lib/pyruse") \
|
||||
+ "/" + os.path.basename(__file__) + ".json"
|
||||
_nft = config.Config().asMap().get("nftBan", {}).get("nft", ["/usr/bin/nft"])
|
||||
|
||||
def __init__(self, args):
|
||||
super().__init__()
|
||||
base.Action.__init__(self)
|
||||
ban.NetfilterBan.__init__(self, Action._storage)
|
||||
if args is None:
|
||||
return # on-boot configuration
|
||||
self.ipv4Set = args["nftSetIPv4"]
|
||||
self.ipv6Set = args["nftSetIPv6"]
|
||||
self.field = args["IP"]
|
||||
self.banSeconds = args.get("banSeconds", None)
|
||||
ipv4Set = args["nftSetIPv4"]
|
||||
ipv6Set = args["nftSetIPv6"]
|
||||
field = args["IP"]
|
||||
banSeconds = args.get("banSeconds", None)
|
||||
self.initSelf(ipv4Set, ipv6Set, field, banSeconds)
|
||||
|
||||
def act(self, entry):
|
||||
ip = entry[self.field]
|
||||
nftSet = self.ipv6Set if ":" in ip else self.ipv4Set
|
||||
newBan = {"IP": ip, "nftSet": nftSet}
|
||||
ban.NetfilterBan.act(self, entry)
|
||||
|
||||
now = datetime.datetime.utcnow()
|
||||
bans = []
|
||||
previousTS = None
|
||||
try:
|
||||
with open(Action._storage) as dataFile:
|
||||
for ban in json.load(dataFile):
|
||||
if ban["timestamp"] > 0 and ban["timestamp"] <= now.timestamp():
|
||||
continue
|
||||
elif {k: ban[k] for k in newBan.keys()} == newBan:
|
||||
# should not happen, since the IP is banned…
|
||||
previousTS = ban["timestamp"]
|
||||
else:
|
||||
bans.append(ban)
|
||||
except IOError:
|
||||
pass # new file
|
||||
|
||||
if previousTS is not None:
|
||||
try:
|
||||
cmd = list(Action._nft)
|
||||
cmd.append("delete element %s {%s}" % (nftSet, ip))
|
||||
subprocess.run(cmd)
|
||||
except Exception:
|
||||
pass # too late: not a problem
|
||||
|
||||
if self.banSeconds:
|
||||
until = now + datetime.timedelta(seconds = self.banSeconds)
|
||||
newBan["timestamp"] = until.timestamp()
|
||||
timeout = self.banSeconds
|
||||
else:
|
||||
newBan["timestamp"] = 0
|
||||
timeout = 0
|
||||
|
||||
self._doBan(timeout, ip, nftSet)
|
||||
bans.append(newBan)
|
||||
with open(Action._storage, "w") as dataFile:
|
||||
json.dump(bans, dataFile)
|
||||
|
||||
def boot(self):
|
||||
now = datetime.datetime.utcnow()
|
||||
bans = []
|
||||
try:
|
||||
with open(Action._storage) as dataFile:
|
||||
for ban in json.load(dataFile):
|
||||
if ban["timestamp"] == 0:
|
||||
self._doBan(0, ban["IP"], ban["nftSet"])
|
||||
bans.append(ban)
|
||||
elif ban["timestamp"] <= now.timestamp():
|
||||
continue
|
||||
else:
|
||||
until = datetime.datetime.utcfromtimestamp(ban["timestamp"])
|
||||
timeout = (until - now).total_seconds()
|
||||
self._doBan(int(timeout), ban["IP"], ban["nftSet"])
|
||||
bans.append(ban)
|
||||
except IOError:
|
||||
pass # no file
|
||||
|
||||
with open(Action._storage, "w") as dataFile:
|
||||
json.dump(bans, dataFile)
|
||||
|
||||
def _doBan(self, seconds, ip, nftSet):
|
||||
if seconds < 0:
|
||||
return # can happen when the threshold is crossed while computing the duration
|
||||
def setBan(self, nfSet, ip, seconds):
|
||||
if seconds == 0:
|
||||
timeout = ""
|
||||
else:
|
||||
timeout = " timeout %ss" % seconds
|
||||
cmd = list(Action._nft)
|
||||
cmd.append("add element %s {%s%s}" % (nftSet, ip, timeout))
|
||||
cmd.append("add element %s {%s%s}" % (nfSet, ip, timeout))
|
||||
subprocess.run(cmd)
|
||||
|
||||
def cancelBan(self, nfSet, ip):
|
||||
cmd = list(Action._nft)
|
||||
cmd.append("delete element %s {%s}" % (nfSet, ip))
|
||||
subprocess.run(cmd)
|
||||
|
|
|
@ -0,0 +1,85 @@
|
|||
# pyruse is intended as a replacement to both fail2ban and epylog
|
||||
# Copyright © 2017–2018 Y. Gablin
|
||||
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
|
||||
import abc
|
||||
import datetime
|
||||
import json
|
||||
|
||||
class NetfilterBan(abc.ABC):
|
||||
def __init__(self, storage):
|
||||
self.storage = storage
|
||||
|
||||
def initSelf(self, ipv4Set, ipv6Set, ipField, banSeconds):
|
||||
self.ipv4Set = ipv4Set
|
||||
self.ipv6Set = ipv6Set
|
||||
self.field = ipField
|
||||
self.banSeconds = banSeconds
|
||||
|
||||
@abc.abstractmethod
|
||||
def setBan(self, nfSet, ip, seconds):
|
||||
pass
|
||||
|
||||
@abc.abstractmethod
|
||||
def cancelBan(self, nfSet, ip):
|
||||
pass
|
||||
|
||||
def act(self, entry):
|
||||
ip = entry[self.field]
|
||||
nfSet = self.ipv6Set if ":" in ip else self.ipv4Set
|
||||
newBan = {"IP": ip, "nfSet": nfSet}
|
||||
|
||||
now = datetime.datetime.utcnow()
|
||||
bans = []
|
||||
previousTS = None
|
||||
try:
|
||||
with open(self.storage) as dataFile:
|
||||
for ban in json.load(dataFile):
|
||||
if ban["timestamp"] > 0 and ban["timestamp"] <= now.timestamp():
|
||||
continue
|
||||
elif {k: ban[k] for k in newBan.keys()} == newBan:
|
||||
# should not happen, since the IP is banned…
|
||||
previousTS = ban["timestamp"]
|
||||
else:
|
||||
bans.append(ban)
|
||||
except IOError:
|
||||
pass # new file
|
||||
|
||||
if previousTS is not None:
|
||||
try:
|
||||
self.cancelBan(nfSet, ip)
|
||||
except Exception:
|
||||
pass # too late: not a problem
|
||||
|
||||
if self.banSeconds:
|
||||
until = now + datetime.timedelta(seconds = self.banSeconds)
|
||||
newBan["timestamp"] = until.timestamp()
|
||||
timeout = self.banSeconds
|
||||
else:
|
||||
newBan["timestamp"] = 0
|
||||
timeout = 0
|
||||
|
||||
self.setBan(nfSet, ip, timeout)
|
||||
bans.append(newBan)
|
||||
with open(self.storage, "w") as dataFile:
|
||||
json.dump(bans, dataFile)
|
||||
|
||||
def boot(self):
|
||||
now = int(datetime.datetime.utcnow().timestamp())
|
||||
bans = []
|
||||
try:
|
||||
with open(self.storage) as dataFile:
|
||||
for ban in json.load(dataFile):
|
||||
if ban["timestamp"] == 0:
|
||||
self.setBan(ban["nfSet"], ban["IP"], 0)
|
||||
bans.append(ban)
|
||||
elif int(ban["timestamp"]) <= now:
|
||||
continue
|
||||
else:
|
||||
timeout = int(ban["timestamp"]) - now
|
||||
self.setBan(ban["nfSet"], ban["IP"], timeout)
|
||||
bans.append(ban)
|
||||
except IOError:
|
||||
pass # no file
|
||||
|
||||
with open(self.storage, "w") as dataFile:
|
||||
json.dump(bans, dataFile)
|
|
@ -0,0 +1,147 @@
|
|||
# pyruse is intended as a replacement to both fail2ban and epylog
|
||||
# Copyright © 2017–2018 Y. Gablin
|
||||
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from pyruse.actions.action_ipsetBan import Action
|
||||
|
||||
ipBanCmd = "ipsetBan.cmd"
|
||||
ipBanState = "action_ipsetBan.py.json"
|
||||
|
||||
def _clean():
|
||||
if os.path.exists(ipBanCmd):
|
||||
os.remove(ipBanCmd)
|
||||
if os.path.exists(ipBanState):
|
||||
os.remove(ipBanState)
|
||||
|
||||
def whenBanIPv4ThenAddToIPv4Set():
|
||||
_clean()
|
||||
Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban"}).act({"ip": "10.0.0.1"})
|
||||
assert os.path.exists(ipBanCmd)
|
||||
assert os.path.exists(ipBanState)
|
||||
nbLines = 0
|
||||
with open(ipBanCmd, "rt") as c:
|
||||
for line in c:
|
||||
assert line == "add I4ban 10.0.0.1\n", line
|
||||
nbLines += 1
|
||||
assert nbLines == 1, nbLines
|
||||
nbBans = 0
|
||||
with open(ipBanState) as s:
|
||||
for ban in json.load(s):
|
||||
assert ban["IP"] == "10.0.0.1" and ban["nfSet"] == "I4ban", str(ban)
|
||||
nbBans += 1
|
||||
assert nbBans == 1, nbBans
|
||||
_clean()
|
||||
|
||||
def whenBanIPv6ThenAddToIPv6Set():
|
||||
_clean()
|
||||
Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban"}).act({"ip": "::1"})
|
||||
assert os.path.exists(ipBanCmd)
|
||||
assert os.path.exists(ipBanState)
|
||||
nbLines = 0
|
||||
with open(ipBanCmd, "rt") as c:
|
||||
for line in c:
|
||||
assert line == "add I6ban ::1\n", line
|
||||
nbLines += 1
|
||||
assert nbLines == 1, nbLines
|
||||
nbBans = 0
|
||||
with open(ipBanState) as s:
|
||||
for ban in json.load(s):
|
||||
assert ban["IP"] == "::1" and ban["nfSet"] == "I6ban", str(ban)
|
||||
nbBans += 1
|
||||
assert nbBans == 1, nbBans
|
||||
_clean()
|
||||
|
||||
def whenBanTwoIPThenTwoLinesInState():
|
||||
_clean()
|
||||
action = Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban"})
|
||||
action.act({"ip": "10.0.0.1"})
|
||||
action.act({"ip": "::1"})
|
||||
action.act({"ip": "10.0.0.1"})
|
||||
assert os.path.exists(ipBanState)
|
||||
nbBans = 0
|
||||
with open(ipBanState) as s:
|
||||
for ban in json.load(s):
|
||||
if ban["IP"] == "10.0.0.1":
|
||||
assert ban["nfSet"] == "I4ban", str(ban)
|
||||
elif ban["IP"] == "::1":
|
||||
assert ban["nfSet"] == "I6ban", str(ban)
|
||||
else:
|
||||
assert false, str(ban)
|
||||
nbBans += 1
|
||||
assert nbBans == 2, nbBans
|
||||
_clean()
|
||||
|
||||
def whenBanAnewThenNoDuplicate():
|
||||
_clean()
|
||||
action = Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban"})
|
||||
action.act({"ip": "10.0.0.1"})
|
||||
action.act({"ip": "10.0.0.1"})
|
||||
assert os.path.exists(ipBanCmd)
|
||||
assert os.path.exists(ipBanState)
|
||||
lineCount = 0
|
||||
with open(ipBanCmd, "rt") as c:
|
||||
for line in c:
|
||||
lineCount += 1
|
||||
if lineCount == 1:
|
||||
assert line == "add I4ban 10.0.0.1\n", line
|
||||
elif lineCount == 2:
|
||||
assert line == "del I4ban 10.0.0.1\n", line
|
||||
elif lineCount == 3:
|
||||
assert line == "add I4ban 10.0.0.1\n", line
|
||||
assert lineCount == 3, lineCount
|
||||
nbBans = 0
|
||||
with open(ipBanState) as s:
|
||||
for ban in json.load(s):
|
||||
if ban["IP"] == "10.0.0.1":
|
||||
assert ban["nfSet"] == "I4ban", str(ban)
|
||||
nbBans += 1
|
||||
assert nbBans == 1, nbBans
|
||||
_clean()
|
||||
|
||||
def whenFinishedBanThenAsIfNotThere():
|
||||
_clean()
|
||||
action = Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban", "banSeconds": 1})
|
||||
action.act({"ip": "10.0.0.1"})
|
||||
time.sleep(1)
|
||||
action.act({"ip": "10.0.0.1"})
|
||||
assert os.path.exists(ipBanCmd)
|
||||
lineCount = 0
|
||||
with open(ipBanCmd, "rt") as c:
|
||||
for line in c:
|
||||
lineCount += 1
|
||||
if lineCount == 1:
|
||||
assert line == "add I4ban 10.0.0.1 timeout 1\n", line
|
||||
elif lineCount == 2:
|
||||
assert line == "add I4ban 10.0.0.1 timeout 1\n", line
|
||||
assert lineCount == 2, lineCount
|
||||
_clean()
|
||||
|
||||
def whenUnfinishedBanThenTimeoutReset():
|
||||
_clean()
|
||||
action = Action({"IP": "ip", "ipSetIPv4": "I4ban", "ipSetIPv6": "I6ban", "banSeconds": 2})
|
||||
action.act({"ip": "10.0.0.1"})
|
||||
time.sleep(1)
|
||||
action.act({"ip": "10.0.0.1"})
|
||||
assert os.path.exists(ipBanCmd)
|
||||
lineCount = 0
|
||||
with open(ipBanCmd, "rt") as c:
|
||||
for line in c:
|
||||
lineCount += 1
|
||||
if lineCount == 1:
|
||||
assert line == "add I4ban 10.0.0.1 timeout 2\n", line
|
||||
elif lineCount == 2:
|
||||
assert line == "del I4ban 10.0.0.1\n", line
|
||||
elif lineCount == 3:
|
||||
assert line == "add I4ban 10.0.0.1 timeout 2\n", line
|
||||
assert lineCount == 3, lineCount
|
||||
_clean()
|
||||
|
||||
def unitTests():
|
||||
whenBanIPv4ThenAddToIPv4Set()
|
||||
whenBanIPv6ThenAddToIPv6Set()
|
||||
whenBanTwoIPThenTwoLinesInState()
|
||||
whenBanAnewThenNoDuplicate()
|
||||
whenFinishedBanThenAsIfNotThere()
|
||||
whenUnfinishedBanThenTimeoutReset()
|
|
@ -29,7 +29,7 @@ def whenBanIPv4ThenAddToIPv4Set():
|
|||
nbBans = 0
|
||||
with open(nftBanState) as s:
|
||||
for ban in json.load(s):
|
||||
assert ban["IP"] == "10.0.0.1" and ban["nftSet"] == "ip I4 ban", str(ban)
|
||||
assert ban["IP"] == "10.0.0.1" and ban["nfSet"] == "ip I4 ban", str(ban)
|
||||
nbBans += 1
|
||||
assert nbBans == 1, nbBans
|
||||
_clean()
|
||||
|
@ -48,7 +48,7 @@ def whenBanIPv6ThenAddToIPv6Set():
|
|||
nbBans = 0
|
||||
with open(nftBanState) as s:
|
||||
for ban in json.load(s):
|
||||
assert ban["IP"] == "::1" and ban["nftSet"] == "ip6 I6 ban", str(ban)
|
||||
assert ban["IP"] == "::1" and ban["nfSet"] == "ip6 I6 ban", str(ban)
|
||||
nbBans += 1
|
||||
assert nbBans == 1, nbBans
|
||||
_clean()
|
||||
|
@ -64,9 +64,9 @@ def whenBanTwoIPThenTwoLinesInState():
|
|||
with open(nftBanState) as s:
|
||||
for ban in json.load(s):
|
||||
if ban["IP"] == "10.0.0.1":
|
||||
assert ban["nftSet"] == "ip I4 ban", str(ban)
|
||||
assert ban["nfSet"] == "ip I4 ban", str(ban)
|
||||
elif ban["IP"] == "::1":
|
||||
assert ban["nftSet"] == "ip6 I6 ban", str(ban)
|
||||
assert ban["nfSet"] == "ip6 I6 ban", str(ban)
|
||||
else:
|
||||
assert false, str(ban)
|
||||
nbBans += 1
|
||||
|
@ -95,7 +95,7 @@ def whenBanAnewThenNoDuplicate():
|
|||
with open(nftBanState) as s:
|
||||
for ban in json.load(s):
|
||||
if ban["IP"] == "10.0.0.1":
|
||||
assert ban["nftSet"] == "ip I4 ban", str(ban)
|
||||
assert ban["nfSet"] == "ip I4 ban", str(ban)
|
||||
nbBans += 1
|
||||
assert nbBans == 1, nbBans
|
||||
_clean()
|
||||
|
|
|
@ -20,7 +20,7 @@ def main():
|
|||
|
||||
# Unit tests
|
||||
import filter_equals, filter_greaterOrEquals, filter_in, filter_inNetworks, filter_lowerOrEquals, filter_pcre, filter_pcreAny, filter_userExists
|
||||
import action_counterRaise, action_counterReset, action_dailyReport, action_dnatCapture, action_dnatReplace, action_email, action_log, action_nftBan
|
||||
import action_counterRaise, action_counterReset, action_dailyReport, action_dnatCapture, action_dnatReplace, action_email, action_ipsetBan, action_log, action_nftBan
|
||||
|
||||
filter_equals.unitTests()
|
||||
filter_greaterOrEquals.unitTests()
|
||||
|
@ -36,6 +36,7 @@ def main():
|
|||
action_dnatCapture.unitTests()
|
||||
action_dnatReplace.unitTests()
|
||||
action_email.unitTests()
|
||||
action_ipsetBan.unitTests()
|
||||
action_log.unitTests()
|
||||
action_nftBan.unitTests()
|
||||
|
||||
|
|
|
@ -87,5 +87,8 @@
|
|||
"nftBan": {
|
||||
"nft": [ "/bin/sh", "-c", "echo \"$0\" >>\"nftBan.cmd\"" ]
|
||||
},
|
||||
"ipsetBan": {
|
||||
"ipset": [ "/bin/sh", "-c", "echo \"$0 $*\" >>\"ipsetBan.cmd\"" ]
|
||||
},
|
||||
"storage": "."
|
||||
}
|
||||
|
|
Loading…
Reference in New Issue