Browse Source

daily report: see all, none, first, last, or first+last times

tags/1.1
Y 1 year ago
parent
commit
71a9ba321f
5 changed files with 144 additions and 24 deletions
  1. 1
    0
      TODO.md
  2. 17
    6
      doc/action_dailyReport.md
  3. 3
    2
      doc/intro_tech.md
  4. 44
    11
      pyruse/actions/action_dailyReport.py
  5. 79
    5
      tests/action_dailyReport.py

+ 1
- 0
TODO.md View File

@@ -1,6 +1,7 @@
# Backlog

* Switch the `dailyReport` from a static layout to a light-weight template system.
* Change `e.get("D", Details.ALL.name)` to `e["D"]` in `action_dailyReport.py` at release next+1, when backward compatibility with the running Pyruse will not be an issue any more.
* Maybe persist counters; they are currently lost on restart.
* Maybe switch from storing the daily journal in a file, to storing it in a database.
* Eventually make the code more elegant, as I learn more about Python…

+ 17
- 6
doc/action_dailyReport.md View File

@@ -35,9 +35,20 @@ When an `action_dailyReport` is used, there are two mandatory parameters:
- this means that any key in the current entry may be referrenced by its name between curly braces;
- and that literal curly braces must be doubled, lest they are read as the start of a template placeholder.

In the `WARN` and `INFO` sections, there is one table row by unique message, and the messages are sorted in alphabetical order.
On each row, the table cells contain first the number of times the message was added to the section, then the message itself, and finally all the dates and times of occurrence.
_Note_: As a consequence, it is useless to put the date and time of occurrence in the message.
Additionally, the `details` parameter may be used to fine-tune the rendering of the times at which events occur (see below).

In the `WARN` and `INFO` sections, there is one table row per unique message, and the messages are sorted in alphabetical order.
On each row, the table cells contain first the number of times the message was added to the section, then the message itself, and finally the dates and times of occurrence:

* If `details` is “`ALL`” or unspecified, each occurrence is mentionned.
* If `details` is “`NONE`”, no occurrence is mentionned.
* If `details` is “`FIRST`”, only the first occurrence is mentionned, prepended by “`From :`”.
* If `details` is “`LAST`”, only the last occurrence is mentionned, prepended by “`Until:`”.
* “`FIRSTLAST`” is a combination of “`FIRST`” and “`LAST`”, although it falls back to “`ALL`” when there are fewer than 2 occurrences.

_Notes_:
* As a consequence, it is useless to put the date and time of occurrence in the message.
* If the same message is added to a section with different levels of details, each level of details gets its own paragraph in the third column.

In the `OTHER` section, the messages are kept in chronological order, and prepended by their date and time of occurrence: “`date+time: message`”. It is thus useless to put the date and time of occurrence in the message.

@@ -46,7 +57,7 @@ Here are examples for each of the sections:
```json
{
"action": "action_dailyReport",
"args": { "level": "WARN", "message": "Nextcloud query failed because the buffer-size was too low" }
"args": { "level": "WARN", "message": "Nextcloud query failed because the buffer-size was too low", "details": "NONE" }
}

{
@@ -60,11 +71,11 @@ Here are examples for each of the sections:
}
```

I chose the `WARN` level for the first situation because, although there is no immediate security risk associated with this fact, I know that some users will experience a loss of functionality.
I chose the `WARN` level for the first situation because, although there is no immediate security risk associated with this fact, I know that some users will experience a loss of functionality. However, the exact times of occurrence are of no use; this is just a situation I need to be aware of.

I chose the `INFO` level for the second situation because all is well with my server; however, depending on who the remote `xmppServer` is, I might want to add it to a whitelist of allowed unsecured peers.

As for the last example, it is the catch-all action, that will report unexpected log lines.

_Tip_: System administrators should know that the contents of the next daily report can always be read in Pyruse’s [storage directory](conffile.md), in the file named `action_dailyReport.py.journal`.
In this file, `L` is the section (aka. level: `1` for `WARN`, `2` for `INFO`, and `0` for `OTHER`), `T` is the Unix timestamp, and `M` is the message.
In this file, `L` is the section (aka. level: `1` for `WARN`, `2` for `INFO`, and `0` for `OTHER`), `T` is the Unix timestamp, `M` is the message, and `D` is the level of details regarding the times of occurrence.

+ 3
- 2
doc/intro_tech.md View File

@@ -15,8 +15,9 @@ It should be noted, that modern Python API are used. Thus:
* Python version ≥ 3.1 is required for managing modules (`importlib`);
* Python version ≥ 3.1 is required for loading the configuration (json’s `object_pairs_hook`);
* Python version ≥ 3.2 is required for the daily report and emails (string’s `format_map`);
* Python version ≥ 3.5 is required for IP address bans and emails (subprocess’ `run`);
* Python version ≥ 3.6 is required for sending emails (`headerregistry`, `EmailMessage`).
* Python version ≥ 3.4 is required for the daily report (`enum`);
* Python version ≥ 3.5 is required for IP address bans and emails, thus also the daily report (subprocess’ `run`);
* Python version ≥ 3.6 is required for emails, thus also the daily report (`headerregistry`, `EmailMessage`).

In order to be fast, this program avoids dynamic decisions while running.
To this end, a static workflow of filters and actions is built upon start, based on the configuration file.

+ 44
- 11
pyruse/actions/action_dailyReport.py View File

@@ -6,8 +6,24 @@ import os
import string
from collections import OrderedDict
from datetime import datetime
from enum import Enum, unique
from pyruse import base, config, email

@unique
class Details(Enum):
NONE = [ lambda l: [] ]
FIRST = [ lambda l: ["From : " + str(t) for t in l[:1]] ]
LAST = [ lambda l: ["Until: " + str(t) for t in l[-1:]] ]
FIRSTLAST = [ lambda l: ["From : " + str(l[0]), "Until: " + str(l[-1])] if len(l) > 1 else [str(t) for t in l] ]
ALL = [ lambda l: [str(t) for t in l] ]

def __init__(self, wrapper):
self.fn = wrapper[0]
def toAdoc(self, times):
return " +\n ".join(str(t) for t in self.fn(times))
def toHtml(self, times):
return "<br/>".join(str(t) for t in self.fn(times))

class Action(base.Action):
_storage = config.Config().asMap().get("storage", "/var/lib/pyruse") \
+ "/" + os.path.basename(__file__) + ".journal"
@@ -47,6 +63,7 @@ class Action(base.Action):

def __init__(self, args):
super().__init__()

l = args["level"]
if l == "WARN":
self.level = 1
@@ -54,6 +71,7 @@ class Action(base.Action):
self.level = 2
else:
self.level = 0

self.template = args["message"]
values = {}
for (_void, name, _void, _void) in string.Formatter().parse(self.template):
@@ -61,12 +79,20 @@ class Action(base.Action):
values[name] = None
self.values = values

ts = args.get("details", Details.ALL.name)
for e in Details:
if ts == e.name:
self.details = e
break
else:
self.details = Details.ALL

def act(self, entry):
for (name, _void) in self.values.items():
self.values[name] = entry.get(name, None)
msg = self.template.format_map(self.values)
json.dump(
OrderedDict(L = self.level, T = entry["__REALTIME_TIMESTAMP"].timestamp(), M = msg),
OrderedDict(L = self.level, T = entry["__REALTIME_TIMESTAMP"].timestamp(), M = msg, D = self.details.name),
Action._out
)
Action._out.write(",\n")
@@ -81,27 +107,34 @@ class Action(base.Action):
return text.replace('&', '&amp;').replace('<', '&lt;').replace('>', '&gt;')

def _toAdoc(self, msg, times):
return "\n|{count:^5d}|{text}\n |{times}\n".format_map(
{"count": len(times), "text": msg, "times": " +\n ".join(str(t) for t in times)}
)
return "\n|{count:^5d}|{text}\n |{times}\n".format_map({
"count": sum(len(t) for (_void, t) in times.items()),
"text": msg,
"times": "\n +\n ".join(d.toAdoc(t) for (d,t) in times.items())
})

def _toHtml(self, msg, times):
return "<tr><td>{count}</td><td>{text}</td><td>{times}</td></tr>\n".format_map(
{"count": len(times), "text": self._encode(msg), "times": "<br/>".join(str(t) for t in times)}
)
return "<tr><td>{count}</td><td>{text}</td><td>{times}</td></tr>\n".format_map({
"count": sum(len(t) for (_void, t) in times.items()),
"text": self._encode(msg),
"times": "<br/><br/>".join(d.toHtml(t) for (d,t) in times.items())
})

def _sendReport(self):
messages = [[], {}, {}]
with open(Action._storage) as journal:
for e in json.load(journal):
if e != {}:
(L, T, M) = (e["L"], datetime.fromtimestamp(e["T"]), e["M"])
(L, T, M, D) = (e["L"], datetime.fromtimestamp(e["T"]), e["M"], e.get("D", Details.ALL.name))
if L == 0:
messages[0].append((T, M))
elif M in messages[L]:
messages[L][M].append(T)
else:
messages[L][M] = [T]
dd = Details[D]
if M not in messages[L]:
messages[L][M] = {}
if dd not in messages[L][M]:
messages[L][M][dd] = []
messages[L][M][dd].append(T)
os.remove(Action._storage)

html = Action._htmDocStart + Action._htmHeadWarn

+ 79
- 5
tests/action_dailyReport.py View File

@@ -11,6 +11,10 @@ mail_filename = "email.dump"
wAction = Action({"level": "WARN", "message": "WarnMsg {m}"})
iAction = Action({"level": "INFO", "message": "InfoMsg {m}"})
oAction = Action({"level": "OTHER", "message": "MiscMsg {m}"})
wActFirst = Action({"level": "WARN", "message": "WarnMsg {m}", "details": "FIRST"})
wActLast = Action({"level": "WARN", "message": "WarnMsg {m}", "details": "LAST"})
wActFL = Action({"level": "WARN", "message": "WarnMsg {m}", "details": "FIRSTLAST"})
wActNone = Action({"level": "WARN", "message": "WarnMsg {m}", "details": "NONE"})

def newEntry(m):
return {"__REALTIME_TIMESTAMP": datetime.utcnow(), "m": m}
@@ -18,7 +22,6 @@ def newEntry(m):
def whenNewDayThenReport():
if os.path.exists(mail_filename):
os.remove(mail_filename)
Action._hour = 0
oAction.act(newEntry("message1"))
assert not os.path.exists(mail_filename)
Action._hour = 25
@@ -26,10 +29,9 @@ def whenNewDayThenReport():
assert os.path.exists(mail_filename)
os.remove(mail_filename)

def whenEmailThenCheckContents():
def whenEmailThenCheck3Sections():
if os.path.exists(mail_filename):
os.remove(mail_filename)
Action._hour = 0
wAction.act(newEntry("messageW"))
iAction.act(newEntry("messageI"))
Action._hour = 25
@@ -70,6 +72,73 @@ def whenEmailThenCheckContents():
assert nbMisc == 2
os.remove(mail_filename)

def _compareEmailWithExpected(expected):
assert os.path.exists(mail_filename)
reTime = re.compile(r"\d{4}(?:[- :.]\d{2}){6}\d{4}")
warnSeen = False
nbTimes = 0
nbFirst = 0
nbLast = 0
line = ""
with open(mail_filename, 'rt') as m:
for l in m:
if l != "" and l[-1:] == "=":
line += l[:-1]
continue
elif l == "" and warnSeen:
break
line += l
if "WarnMsg" in line:
warnSeen = True
elif not warnSeen:
line = ""
continue
nbTimes += len(reTime.findall(line))
if "From=C2=A0:" in line:
nbFirst += 1
if "Until:" in line:
nbLast += 1
if "</tr>" in line:
break
line = ""
seen = dict(warn = warnSeen, times = nbTimes, first = nbFirst, last = nbLast)
assert seen == expected, "Expected=" + str(expected) + " ≠ Seen=" + str(seen)
os.remove(mail_filename)

def whenEmailThenCheckTimes(warnAction, expected):
if os.path.exists(mail_filename):
os.remove(mail_filename)
warnAction.act(newEntry("messageW"))
warnAction.act(newEntry("messageW"))
Action._hour = 25
warnAction.act(newEntry("messageW"))
_compareEmailWithExpected(expected)

def whenSeveralDetailsModesThenOnlyOneWarn():
if os.path.exists(mail_filename):
os.remove(mail_filename)
wAction.act(newEntry("messageW"))
wAction.act(newEntry("messageW"))
wAction.act(newEntry("messageW"))
wAction.act(newEntry("messageW"))
wAction.act(newEntry("messageW"))
wActFirst.act(newEntry("messageW"))
wActFirst.act(newEntry("messageW"))
wActFirst.act(newEntry("messageW"))
wActLast.act(newEntry("messageW"))
wActLast.act(newEntry("messageW"))
wActLast.act(newEntry("messageW"))
wActFL.act(newEntry("messageW"))
wActFL.act(newEntry("messageW"))
wActFL.act(newEntry("messageW"))
wActFL.act(newEntry("messageW"))
wActNone.act(newEntry("messageW"))
wActNone.act(newEntry("messageW"))
wActNone.act(newEntry("messageW"))
Action._hour = 25
wActNone.act(newEntry("messageW"))
_compareEmailWithExpected(dict(warn = True, times = 9, first = 2, last = 2))

def whenReportThenNewSetOfMessages():
if os.path.exists(mail_filename):
os.remove(mail_filename)
@@ -77,9 +146,14 @@ def whenReportThenNewSetOfMessages():
oAction.act(newEntry("message3"))
assert os.path.exists(mail_filename)
os.remove(mail_filename)
whenEmailThenCheckContents()
whenEmailThenCheck3Sections()

def unitTests():
whenNewDayThenReport()
whenEmailThenCheckContents()
whenEmailThenCheck3Sections()
whenEmailThenCheckTimes(wActFirst, dict(warn = True, times = 1, first = 1, last = 0))
whenEmailThenCheckTimes(wActLast, dict(warn = True, times = 1, first = 0, last = 1))
whenEmailThenCheckTimes(wActFL, dict(warn = True, times = 2, first = 1, last = 1))
whenEmailThenCheckTimes(wActNone, dict(warn = True, times = 0, first = 0, last = 0))
whenSeveralDetailsModesThenOnlyOneWarn()
whenReportThenNewSetOfMessages()

Loading…
Cancel
Save