Browse Source

dnat actions: simulate a transparent proxy, where logs are concerned

master
Yves G 3 years ago
parent
commit
f3b674ca26
8 changed files with 501 additions and 1 deletions
  1. +23
    -0
      README.md
  2. +175
    -0
      doc/dnat.md
  3. +18
    -0
      pyruse/actions/action_dnatCapture.py
  4. +18
    -0
      pyruse/actions/action_dnatReplace.py
  5. +97
    -0
      pyruse/dnat.py
  6. +93
    -0
      tests/action_dnatCapture.py
  7. +74
    -0
      tests/action_dnatReplace.py
  8. +3
    -1
      tests/main.py

+ 23
- 0
README.md View File

@ -1,14 +1,34 @@
# Python peruser of systemd-journal
## Summary
This program is intended to be used as a lightweight replacement for both epylog and fail2ban.
Its purpose is to peruse the system log entries, warn of important situations, report daily on the latest events, and act on specific patterns (IP address bans…).
* [Functional overview](doc/intro_func.md)
* [Technical overview](doc/intro_tech.md)
The benefits of Pyruse over products of the same kind are:
* **Optimization brought by systemd**
[systemd-journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html) play an important role in Pyruse: instead of matching log entries against message patterns only, the whole range of systemd’s journal fields is available. This allows for the much faster integer comparisons (`PRIORITY`, `_UID`…), or even faster comparisons with short strings like the `SYSLOG_IDENTIFIER`, `_SYSTEMD_UNIT`, or `_HOSTNAME`, with the opportunity to test more often for equality, and less for regular expressions.
* **Optimization brought by context**
Programs that peruse the system logs usually apply a set of rules on each log entry, rule after rule, regardless of what can be deduced by the already-applied rules.
In contrast, each fact learnt by applying a rule in Pyruse can be taken into account so that rules that do not apply are not even considered.
For example, after matching the `SYSLOG_IDENTIFIER` of a journal entry to the value `sshd`, only SSH-related rules are applied, not Nginx-related rules, nor Prosody-related rules.
* **Modularity**
Each filter (ie. a matching step) or action (eg. a ban, an email, etc.) is a Python module with a very simple API. As soon as a new need arises, a module can be written for it.
For example, to my knowledge, there is no equivalent in any tool of the same scale, for the [DNAT-correcting actions](doc/dnat.md) now included with Pyruse.
## Get Pyruse
Pyruse is [packaged for Archlinux](https://aur.archlinux.org/packages/pyruse/).
For other distributions, please [read the manual installation instructions](doc/install.md).
## Configuration
The `/etc/pyruse` directory is where system-specific files are looked-for:
* the `pyruse.json` file that contains the [configuration](doc/conffile.md),
@ -16,6 +36,8 @@ The `/etc/pyruse` directory is where system-specific files are looked-for:
Instead of using `/etc/pyruse`, an alternate directory may be specified with the `PYRUSE_EXTRA` environment variable.
## Documentation
For more in-depth documentation, please refer to these pages:
* [General structure of the `pyruse.json` file](doc/conffile.md)
@ -24,6 +46,7 @@ For more in-depth documentation, please refer to these pages:
* More information about:
- [the built-in filters](doc/builtinfilters.md)
- [the counter-based actions](doc/counters.md)
- [the DNAT-related actions](doc/dnat.md)
- [the `action_noop` module](doc/noop.md)
- [the `action_email` module](doc/action_email.md)
- [the `action_dailyReport` module](doc/action_dailyReport.md)


+ 175
- 0
doc/dnat.md View File

@ -0,0 +1,175 @@
# DNAT-correcting actions
## Introduction
Pyruse provides two actions, namely `action_dnatCapture` and `action_dnatReplace`, that work together towards a single goal: giving to Pyruse’s filters and actions the illusion that client connections come directly to the examined services, instead of going through a firewall Network Address Translation or a reverse-proxy.
If for example you run a Prosody XMPP server (or anything that does not handle the [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt)) behind an HAProxy reverse-proxy, or if all your network connections go through a NAT’ing proxy in your DMZ, then the following happens: the services report your proxy as being the client, ie. usually `127.0.0.1` (same machine) or your LAN’s gateway IP.
Here is a simplified illustration of the network configuration:
```ditaa
/-------------------------------\
| +---------------------+
| Client | ClientIP:ClientPort +---\==\
| +---------------------+ | :
\-------------------------------/ | :
(1)| :(2)
/-------------------------------\ | :
| +--++-------------------+ | :
| | PublicIP:PublicPort +<--/ :
| +-----------------------+ : /---------------------------------\
| Proxy | : +-----------------------+ |
| +-----------------------+ \===>+ | Service |
| | ProxyLanIP:RandomPort +---------->+ ServiceIP:ServicePort | |
| +-----------------------+ (1) +-----------------------+ |
\-------------------------------/ \---------------------------------/
```
The circuit number `(1)` is the real one, which is why the service sees `ProxyLanIP:RandomPort` as the client.
The circuit number `(2)` is what Pyruse will fake, using the fore-mentionned actions.
First some “vocabulary”. In `action_dnatCapture` and `action_dnatReplace`:
* `ClientIP` and `ClientPort` are called `saddr` and `sport` (`s` for **s**ource)
* `ProxyLanIP` and `RandomPort` are called `addr` and `port`
* `ServiceIP` and `ServicePort` are called `daddr` and `dport` (`d` for **d**estination)
Pyruse’s actions work by storing the link between these 6 values in memory, and later replacing `addr` with `saddr`, and optionaly `port` with `sport`.
## Action `action_dnatCapture`
For Pyruse to be able to capture the wanted values, the proxy software must first be configured to provide them.
### HAProxy configuration
Here is an example configuration that reproduces the default `tcplog` format, simply adding the missing information between square braquets:
```haproxy
global
log /dev/log local0 info
… misc. other options …
defaults
mode tcp
log global
option log-separate-errors
log-format "%ci:%cp [%t] %ft %b[%bi:%bp]/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"
… misc. other options …
```
The above configuration would produce log lines like this one:
```log
12.34.56.78:54321 [dd/MM/yyyy:HH:mm:ss.…] tls~ xmpp[10.0.0.1:43210]/xmpp …/…/… … -- …/…/…/…/… …/…
```
### nftables configuration
Here is an example rule for nftables on the proxy:
```nftables
tcp dport 22 log prefix "DNAT/ssh: " dnat to 10.0.0.2
```
Having an easily recognizable log prefix helps.
The above would result in a line like this one:
```log
DNAT/ssh: IN=… OUT=… MAC=… SRC=12.34.56.78 DST=10.0.0.1 LEN=… … PROTO=… SPT=43210 DPT=22 WINDOW=… …
```
Besides, Netfilter logs (part of the kernel logs) must be enabled for those log lines to actually appear in the logs.
For example, it [may be required](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2851940ffee313e0ff12540a8e11a8c54dea9c65) to run this:
```bash
sysctl net.netfilter.nf_log_all_netns=1
```
### Pyruse configuration
Action `action_dnatCapture` must tell Pyruse what fields in the current log entry match the 6 parameters described in the introduction.
If some values are constant and known (or marker values are wished for), but these values are in no available field in the log entry, then those values may be given in the parameters `addrValue`, `portValue`, `daddrValue`, and `dportValue`.
When an information is given by both a field reference and a plain value, the field reference is used first, and the plain value is used as a default value if the referenced field is not found.
The `saddr` parameter is mandatory (else there is no point), and either `addr` or `addrValue` (or both) must be given as well.
In addition to all these parameters, a `keepSeconds` parameter may be given to indicate how many seconds the detected correspondance should be kept in memory (default value is 63).
NOTE: For performance reasons, the `keepSeconds` value is rounded up to the _next_ power of 2 — eg. values 4 to 7 are rounded up to 8 —, and the retention countdown only begins at the next occurrence of that power of two in the current time, expressed as a Unix timestamp.
As a consequence, the actual length of time a correspondance is kept in memory, varies between 1× and 4× the length of time given by the parameter, depending on the chosen value, and depending on the current time-of-the-day when the correspondance is found.
Here is an example configuration that would work fine with log lines as produced by nftables (see above):
```json
{ "filter": "filter_pcre",
"args": {
"field": "MESSAGE",
"re": "^DNAT/ssh:.* SRC=([^ ]+) DST=([^ ]+) .* SPT=([^ ]+) DPT=([^ ]+) ",
"save": [ "dnatSaddr", "dnatAddr", "dnatPort", "dnatDport" ]
}
},
{ "action": "action_dnatCapture",
"args": {
"saddr": "dnatSaddr",
"addr": "dnatAddr", "addrValue": "127.0.0.1",
"port": "dnatPort",
"dport": "dnatDport", "dportValue": "22"
}
}
```
## Action `action_dnatReplace`
Action `action_dnatReplace` should be inserted whenever there is a chance that the values stored in a log entry’s fields for a client IP address (and possibly the port as well) are those of a proxy instead.
Properties `addr`, `port`, `daddr`, and `dport` are used to match against a correspondance currently held in memory; at least one of these properties must be given.
Each property corresponds to the name of a log entry field in which to read the corresponding value.
Properties `saddrInto` and `sportInto` indicate the log entry fields in which to store the corrected source IP address or port; at least one of those properties must be given.
For example, consider the following (simplified) log entries:
```json
{ '_HOSTNAME': 'dmz',
'SYSLOG_IDENTIFIER': 'kernel',
'MESSAGE': 'DNAT/ssh: … SRC=12.34.56.78 DST=10.0.0.1 … SPT=43210 …'
}
{ '_HOSTNAME': 'sshserv',
'SYSLOG_IDENTIFIER': 'sshd',
'_SYSTEMD_UNIT': 'sshd.service',
'MESSAGE': 'Failed password for ME from 10.0.0.1 port 43210 ssh2'
}
{ '_HOSTNAME': 'dmz',
'SYSLOG_IDENTIFIER': 'sshd',
'_SYSTEMD_UNIT': 'sshd.service',
'MESSAGE': 'Failed password for KeyUser from 87.65.43.21 port 24680 ssh2'
}
```
Assuming the first log entry is correctly handled by `action_dnatCapture`, a good configuration to handle SSH failed logins could be:
```json
{ "filter": "filter_equals",
"args": { "field": "SYSLOG_IDENTIFIER", "value": "sshd" }
},
{ "filter": "filter_pcre",
"args": {
"field": "MESSAGE",
"re": "^Failed password for (.*) from ([^ ]+) port ([^ ]+) ssh2$",
"save": [ "sshUser", "clientIP", "clientPort" ]
},
{ "action": "action_dnatReplace",
"args": { "addr": "clientIP", "port": "clientPort", "saddrInto": "clientIP" }
},
{ "action": "action_email",
"args": { "message": "SSH attack from {clientIP} on {sshUser}@{_HOSTNAME}." }
}
```
As a result, two emails would be sent, with these messages:
```
SSH attack from 12.34.56.78 on ME@sshserv.
SSH attack from 87.65.43.21 on KeyUser@dmz.
```

+ 18
- 0
pyruse/actions/action_dnatCapture.py View File

@ -0,0 +1,18 @@
# pyruse is intended as a replacement to both fail2ban and epylog
# Copyright © 2017–2018 Y. Gablin
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
from pyruse import base, dnat
class Action(base.Action, dnat.Mapper):
def __init__(self, args):
base.Action.__init__(self)
sa = (args ["saddr"], None )
sp = (args.get("sport", None), None )
a = (args.get("addr", None), args.get("addrValue", None))
p = (args.get("port", None), args.get("portValue", None))
da = (args.get("daddr", None), args.get("daddrValue", None))
dp = (args.get("dport", None), args.get("dportValue", None))
dnat.Mapper.__init__(self, sa, sp, a, p, da, dp, args.get("keepSeconds", 63))
def act(self, entry):
self.map(entry)

+ 18
- 0
pyruse/actions/action_dnatReplace.py View File

@ -0,0 +1,18 @@
# pyruse is intended as a replacement to both fail2ban and epylog
# Copyright © 2017–2018 Y. Gablin
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
from pyruse import base, dnat
class Action(base.Action, dnat.Matcher):
def __init__(self, args):
base.Action.__init__(self)
sa = args ["saddrInto"]
sp = args.get("sportInto", None)
a = args.get("addr", None)
p = args.get("port", None)
da = args.get("daddr", None)
dp = args.get("dport", None)
dnat.Matcher.__init__(self, a, p, da, dp, sa, sp)
def act(self, entry):
self.replace(entry)

+ 97
- 0
pyruse/dnat.py View File

@ -0,0 +1,97 @@
# pyruse is intended as a replacement to both fail2ban and epylog
# Copyright © 2017–2018 Y. Gablin
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
from datetime import datetime
_mappings = []
def _cleanMappings():
global _mappings
now = int(datetime.today().timestamp())
_mappings = [m for m in _mappings if (now >> m["bits"]) <= m["time"]]
def putMapping(mapping):
global _mappings
_cleanMappings()
_mappings.append(mapping)
def getMappings():
global _mappings
_cleanMappings()
return _mappings
def periodBits(keepSeconds):
seconds, bits = keepSeconds, 0
while seconds:
bits += 1
seconds = seconds >> 1
return bits # number of significant bits in keepSeconds
def valueFor(spec, entry):
return spec[1] if spec[0] is None else entry.get(spec[0], spec[1])
class Mapper():
def __init__(self, saddr, sport, addr, port, daddr, dport, keepSeconds):
for spec in [saddr, addr]:
if spec[0] is None and spec[1] is None:
raise ValueError("Neither field nor value was specified for address")
self.saddr = saddr
self.sport = sport
self.addr = addr
self.port = port
self.daddr = daddr
self.dport = dport
self.keepBits = periodBits(keepSeconds)
def map(self, entry):
saddr = valueFor(self.saddr, entry)
addr = valueFor(self.addr, entry)
if saddr is None or addr is None:
return
sport = valueFor(self.sport, entry)
port = valueFor(self.port, entry)
daddr = valueFor(self.daddr, entry)
dport = valueFor(self.dport, entry)
putMapping(dict(
bits = self.keepBits,
time = 1 + (int(entry["__REALTIME_TIMESTAMP"].timestamp()) >> self.keepBits),
saddr = saddr, sport = sport,
addr = addr, port = port,
daddr = daddr, dport = dport
))
class Matcher():
def __init__(self, addr, port, daddr, dport, saddr, sport):
if addr is None and port is None and daddr is None and dport is None:
raise ValueError("No field was provided on which to do the matching")
if saddr is None and sport is None:
raise ValueError("No field was provided in which to store the translated values")
matchers = []
updaters = []
if addr is not None:
matchers.append((addr, "addr"))
if port is not None:
matchers.append((port, "port"))
if daddr is not None:
matchers.append((daddr, "daddr"))
if dport is not None:
matchers.append((dport, "dport"))
if saddr is not None:
updaters.append((saddr, "saddr"))
if sport is not None:
updaters.append((sport, "sport"))
self.matchers = matchers
self.updaters = updaters
def replace(self, entry):
for field, _void in self.matchers:
if field not in entry:
return
for mapping in getMappings():
for field, mapEntry in self.matchers:
if entry[field] != mapping[mapEntry]:
break
else:
for field, mapEntry in self.updaters:
entry[field] = mapping[mapEntry]
return

+ 93
- 0
tests/action_dnatCapture.py View File

@ -0,0 +1,93 @@
# pyruse is intended as a replacement to both fail2ban and epylog
# Copyright © 2017–2018 Y. Gablin
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
from datetime import datetime
from pyruse import dnat
from pyruse.actions.action_dnatCapture import Action
def whenNoSaddrThenError():
try:
Action(dict(addr=1))
except Exception:
return
assert False, "An exception should be raised when saddr is absent"
def whenNoAddrNorAddrvalueThenError():
try:
Action(dict(saddr=1))
except Exception:
return
assert False, "An exception should be raised when addr and addrValue are absent"
def whenNoAddrButAddrvalueThenNoError():
Action(dict(saddr=1, addrValue=1))
def whenNoAddrvalueButAddrThenNoError():
Action(dict(saddr=1, addr=1))
def whenNoKeepsecondsThen6bits():
a = Action(dict(saddr=1, addr=1))
assert a.keepBits == 6, "Default keepSeconds (63) should be on 6 bits, not " + str(a.keepBits)
def whenKeepsecondsIs150Then8bits():
a = Action(dict(saddr=1, addr=1, keepSeconds=150))
assert a.keepBits == 8, "150 for keepSeconds should be on 8 bits, not " + str(a.keepBits)
def whenInsufficientEntryThenNoMapping():
dnat._mappings = []
Action({"saddr": "sa", "addrValue": "x"}).act({"__REALTIME_TIMESTAMP": datetime(2018,1,1)})
assert dnat._mappings == [], "Got:\n" + str(dnat._mappings) + "\ninstead of []"
def whenFieldAndOrValueThenCheckMapping(spec, entryWithAddr, entryWithDAddr, expect):
dnat._mappings = []
# specify the Action
spec.update({"saddr": "sa"})
# prepare the entry
entry = {
"__REALTIME_TIMESTAMP": datetime(2018,1,1),
"sa": "vsa", "sp": "vsp"}
if entryWithAddr:
entry.update({"a": "va", "p": "vp"})
if entryWithDAddr:
entry.update({"da": "vda", "dp": "vdp"})
# run
Action(spec).act(entry)
# check the result
expect.update({"bits": 6, "time": 23668144, "saddr": "vsa"})
assert dnat._mappings == [expect], "Got:\n" + str(dnat._mappings) + "\ninstead of:\n" + str([expect])
def unitTests():
whenNoSaddrThenError()
whenNoAddrNorAddrvalueThenError()
whenNoAddrButAddrvalueThenNoError()
whenNoAddrvalueButAddrThenNoError()
whenNoKeepsecondsThen6bits()
whenKeepsecondsIs150Then8bits()
whenInsufficientEntryThenNoMapping()
whenFieldAndOrValueThenCheckMapping({"addr": "a"}, True, True,
{"sport": None, "addr": "va", "port": None, "daddr": None, "dport": None})
whenFieldAndOrValueThenCheckMapping({"addrValue": "x"}, True, True,
{"sport": None, "addr": "x", "port": None, "daddr": None, "dport": None})
whenFieldAndOrValueThenCheckMapping({"addr": "a", "addrValue": "x"}, True, True,
{"sport": None, "addr": "va", "port": None, "daddr": None, "dport": None})
whenFieldAndOrValueThenCheckMapping({"addr": "a", "addrValue": "x"}, False, True,
{"sport": None, "addr": "x", "port": None, "daddr": None, "dport": None})
whenFieldAndOrValueThenCheckMapping({"addr": "a", "daddr": "da"}, True, True,
{"sport": None, "addr": "va", "port": None, "daddr": "vda", "dport": None})
whenFieldAndOrValueThenCheckMapping({"addr": "a", "daddrValue": "x"}, True, True,
{"sport": None, "addr": "va", "port": None, "daddr": "x", "dport": None})
whenFieldAndOrValueThenCheckMapping({"addr": "a", "daddr": "da", "daddrValue": "x"}, True, True,
{"sport": None, "addr": "va", "port": None, "daddr": "vda", "dport": None})
whenFieldAndOrValueThenCheckMapping({"addr": "a", "daddr": "da", "daddrValue": "x"}, True, False,
{"sport": None, "addr": "va", "port": None, "daddr": "x", "dport": None})
whenFieldAndOrValueThenCheckMapping({"addr": "a", "port": "p"}, True, True,
{"sport": None, "addr": "va", "port": "vp", "daddr": None, "dport": None})
whenFieldAndOrValueThenCheckMapping({"addr": "a", "dport": "dp"}, True, True,
{"sport": None, "addr": "va", "port": None, "daddr": None, "dport": "vdp"})

+ 74
- 0
tests/action_dnatReplace.py View File

@ -0,0 +1,74 @@
# pyruse is intended as a replacement to both fail2ban and epylog
# Copyright © 2017–2018 Y. Gablin
# Full licensing information in the LICENSE file, or gnu.org/licences/gpl-3.0.txt if the file is missing.
from datetime import datetime
from pyruse import dnat
from pyruse.actions.action_dnatReplace import Action
def whenNoSaddrintoThenError():
try:
Action(dict(addr=1))
except Exception:
return
assert False, "An exception should be raised when saddrInto is absent"
def whenNoMatchFieldThenError():
try:
Action(dict(saddrInto=1))
except Exception:
return
assert False, "An exception should be raised when no match-field is present"
def whenSaddrintoAndAtLeastOneMatchFieldThenNoError():
a = Action(dict(saddrInto=1, dport=1))
assert a.matchers == [(1, "dport")], "Got:\n" + str(a.matchers) + "\ninstead of:\n" + str([(1, "dport")])
assert a.updaters == [(1, "saddr")], "Got:\n" + str(a.updaters) + "\ninstead of:\n" + str([(1, "saddr")])
def whenNoMatchingEntryThenNoChange():
dnat._mappings = [{
"bits": 7, "time": 1183407200,
"saddr": "bad", "sport": None,
"addr": "prox", "port": 12345,
"daddr": "serv", "dport": None}]
a = Action(dict(saddrInto="sa", port="sp"))
entryIn = dict(sa = "prox", da = "serv")
entryOut = entryIn.copy()
a.act(entryOut)
assert entryIn == entryOut, "Got:\n" + str(entryOut) + "\ninstead of:\n" + str(entryIn)
def whenNoMatchingValueThenNoChange():
dnat._mappings = [{
"bits": 7, "time": 1183407200,
"saddr": "bad", "sport": None,
"addr": "prox", "port": 12345,
"daddr": "serv", "dport": None}]
a = Action(dict(saddrInto="sa", port="sp"))
entryIn = dict(sa = "prox", sp = 1234, da = "serv")
entryOut = entryIn.copy()
a.act(entryOut)
assert entryIn == entryOut, "Got:\n" + str(entryOut) + "\ninstead of:\n" + str(entryIn)
def whenMatchingEntryThenChange():
dnat._mappings = [{
"bits": 7, "time": 1183407200,
"saddr": "bad", "sport": None,
"addr": "prox", "port": 12345,
"daddr": "serv", "dport": None}]
a = Action(dict(saddrInto="sa", port="sp"))
entryIn = dict(sa = "prox", sp = 12345, da = "serv")
expect = entryIn.copy()
expect.update({"sa": "bad"})
entryOut = entryIn.copy()
a.act(entryOut)
assert expect == entryOut, "Got:\n" + str(entryOut) + "\ninstead of:\n" + str(expect)
def unitTests():
whenNoSaddrintoThenError()
whenNoMatchFieldThenError()
whenSaddrintoAndAtLeastOneMatchFieldThenNoError()
whenNoMatchingEntryThenNoChange()
whenNoMatchingValueThenNoChange()
whenMatchingEntryThenChange()

+ 3
- 1
tests/main.py View File

@ -20,7 +20,7 @@ def main():
# Unit tests
import filter_equals, filter_greaterOrEquals, filter_in, filter_inNetworks, filter_lowerOrEquals, filter_pcre, filter_pcreAny, filter_userExists
import action_counterRaise, action_counterReset, action_dailyReport, action_email, action_log, action_nftBan
import action_counterRaise, action_counterReset, action_dailyReport, action_dnatCapture, action_dnatReplace, action_email, action_log, action_nftBan
filter_equals.unitTests()
filter_greaterOrEquals.unitTests()
@ -33,6 +33,8 @@ def main():
action_counterRaise.unitTests()
action_counterReset.unitTests()
action_dailyReport.unitTests()
action_dnatCapture.unitTests()
action_dnatReplace.unitTests()
action_email.unitTests()
action_log.unitTests()
action_nftBan.unitTests()


Loading…
Cancel
Save