diff --git a/TODO.md b/TODO.md index d3a0d35..ab88688 100644 --- a/TODO.md +++ b/TODO.md @@ -1,6 +1,7 @@ # Backlog * Switch the `dailyReport` from a static layout to a light-weight template system. +* Change `e.get("D", Details.ALL.name)` to `e["D"]` in `action_dailyReport.py` at release next+1, when backward compatibility with the running Pyruse will not be an issue any more. * Maybe persist counters; they are currently lost on restart. * Maybe switch from storing the daily journal in a file, to storing it in a database. * Eventually make the code more elegant, as I learn more about Python… diff --git a/doc/action_dailyReport.md b/doc/action_dailyReport.md index fa76360..c3fda55 100644 --- a/doc/action_dailyReport.md +++ b/doc/action_dailyReport.md @@ -35,9 +35,20 @@ When an `action_dailyReport` is used, there are two mandatory parameters: - this means that any key in the current entry may be referrenced by its name between curly braces; - and that literal curly braces must be doubled, lest they are read as the start of a template placeholder. -In the `WARN` and `INFO` sections, there is one table row by unique message, and the messages are sorted in alphabetical order. -On each row, the table cells contain first the number of times the message was added to the section, then the message itself, and finally all the dates and times of occurrence. -_Note_: As a consequence, it is useless to put the date and time of occurrence in the message. +Additionally, the `details` parameter may be used to fine-tune the rendering of the times at which events occur (see below). + +In the `WARN` and `INFO` sections, there is one table row per unique message, and the messages are sorted in alphabetical order. +On each row, the table cells contain first the number of times the message was added to the section, then the message itself, and finally the dates and times of occurrence: + +* If `details` is “`ALL`” or unspecified, each occurrence is mentionned. +* If `details` is “`NONE`”, no occurrence is mentionned. +* If `details` is “`FIRST`”, only the first occurrence is mentionned, prepended by “`From :`”. +* If `details` is “`LAST`”, only the last occurrence is mentionned, prepended by “`Until:`”. +* “`FIRSTLAST`” is a combination of “`FIRST`” and “`LAST`”, although it falls back to “`ALL`” when there are fewer than 2 occurrences. + +_Notes_: +* As a consequence, it is useless to put the date and time of occurrence in the message. +* If the same message is added to a section with different levels of details, each level of details gets its own paragraph in the third column. In the `OTHER` section, the messages are kept in chronological order, and prepended by their date and time of occurrence: “`date+time: message`”. It is thus useless to put the date and time of occurrence in the message. @@ -46,7 +57,7 @@ Here are examples for each of the sections: ```json { "action": "action_dailyReport", - "args": { "level": "WARN", "message": "Nextcloud query failed because the buffer-size was too low" } + "args": { "level": "WARN", "message": "Nextcloud query failed because the buffer-size was too low", "details": "NONE" } } { @@ -60,11 +71,11 @@ Here are examples for each of the sections: } ``` -I chose the `WARN` level for the first situation because, although there is no immediate security risk associated with this fact, I know that some users will experience a loss of functionality. +I chose the `WARN` level for the first situation because, although there is no immediate security risk associated with this fact, I know that some users will experience a loss of functionality. However, the exact times of occurrence are of no use; this is just a situation I need to be aware of. I chose the `INFO` level for the second situation because all is well with my server; however, depending on who the remote `xmppServer` is, I might want to add it to a whitelist of allowed unsecured peers. As for the last example, it is the catch-all action, that will report unexpected log lines. _Tip_: System administrators should know that the contents of the next daily report can always be read in Pyruse’s [storage directory](conffile.md), in the file named `action_dailyReport.py.journal`. -In this file, `L` is the section (aka. level: `1` for `WARN`, `2` for `INFO`, and `0` for `OTHER`), `T` is the Unix timestamp, and `M` is the message. +In this file, `L` is the section (aka. level: `1` for `WARN`, `2` for `INFO`, and `0` for `OTHER`), `T` is the Unix timestamp, `M` is the message, and `D` is the level of details regarding the times of occurrence. diff --git a/doc/intro_tech.md b/doc/intro_tech.md index a9bc7a2..4591763 100644 --- a/doc/intro_tech.md +++ b/doc/intro_tech.md @@ -15,8 +15,9 @@ It should be noted, that modern Python API are used. Thus: * Python version ≥ 3.1 is required for managing modules (`importlib`); * Python version ≥ 3.1 is required for loading the configuration (json’s `object_pairs_hook`); * Python version ≥ 3.2 is required for the daily report and emails (string’s `format_map`); -* Python version ≥ 3.5 is required for IP address bans and emails (subprocess’ `run`); -* Python version ≥ 3.6 is required for sending emails (`headerregistry`, `EmailMessage`). +* Python version ≥ 3.4 is required for the daily report (`enum`); +* Python version ≥ 3.5 is required for IP address bans and emails, thus also the daily report (subprocess’ `run`); +* Python version ≥ 3.6 is required for emails, thus also the daily report (`headerregistry`, `EmailMessage`). In order to be fast, this program avoids dynamic decisions while running. To this end, a static workflow of filters and actions is built upon start, based on the configuration file. diff --git a/pyruse/actions/action_dailyReport.py b/pyruse/actions/action_dailyReport.py index 325afd4..2b4e20e 100644 --- a/pyruse/actions/action_dailyReport.py +++ b/pyruse/actions/action_dailyReport.py @@ -6,8 +6,24 @@ import os import string from collections import OrderedDict from datetime import datetime +from enum import Enum, unique from pyruse import base, config, email +@unique +class Details(Enum): + NONE = [ lambda l: [] ] + FIRST = [ lambda l: ["From : " + str(t) for t in l[:1]] ] + LAST = [ lambda l: ["Until: " + str(t) for t in l[-1:]] ] + FIRSTLAST = [ lambda l: ["From : " + str(l[0]), "Until: " + str(l[-1])] if len(l) > 1 else [str(t) for t in l] ] + ALL = [ lambda l: [str(t) for t in l] ] + + def __init__(self, wrapper): + self.fn = wrapper[0] + def toAdoc(self, times): + return " +\n ".join(str(t) for t in self.fn(times)) + def toHtml(self, times): + return "
".join(str(t) for t in self.fn(times)) + class Action(base.Action): _storage = config.Config().asMap().get("storage", "/var/lib/pyruse") \ + "/" + os.path.basename(__file__) + ".journal" @@ -47,6 +63,7 @@ class Action(base.Action): def __init__(self, args): super().__init__() + l = args["level"] if l == "WARN": self.level = 1 @@ -54,6 +71,7 @@ class Action(base.Action): self.level = 2 else: self.level = 0 + self.template = args["message"] values = {} for (_void, name, _void, _void) in string.Formatter().parse(self.template): @@ -61,12 +79,20 @@ class Action(base.Action): values[name] = None self.values = values + ts = args.get("details", Details.ALL.name) + for e in Details: + if ts == e.name: + self.details = e + break + else: + self.details = Details.ALL + def act(self, entry): for (name, _void) in self.values.items(): self.values[name] = entry.get(name, None) msg = self.template.format_map(self.values) json.dump( - OrderedDict(L = self.level, T = entry["__REALTIME_TIMESTAMP"].timestamp(), M = msg), + OrderedDict(L = self.level, T = entry["__REALTIME_TIMESTAMP"].timestamp(), M = msg, D = self.details.name), Action._out ) Action._out.write(",\n") @@ -81,27 +107,34 @@ class Action(base.Action): return text.replace('&', '&').replace('<', '<').replace('>', '>') def _toAdoc(self, msg, times): - return "\n|{count:^5d}|{text}\n |{times}\n".format_map( - {"count": len(times), "text": msg, "times": " +\n ".join(str(t) for t in times)} - ) + return "\n|{count:^5d}|{text}\n |{times}\n".format_map({ + "count": sum(len(t) for (_void, t) in times.items()), + "text": msg, + "times": "\n +\n ".join(d.toAdoc(t) for (d,t) in times.items()) + }) def _toHtml(self, msg, times): - return "{count}{text}{times}\n".format_map( - {"count": len(times), "text": self._encode(msg), "times": "
".join(str(t) for t in times)} - ) + return "{count}{text}{times}\n".format_map({ + "count": sum(len(t) for (_void, t) in times.items()), + "text": self._encode(msg), + "times": "

".join(d.toHtml(t) for (d,t) in times.items()) + }) def _sendReport(self): messages = [[], {}, {}] with open(Action._storage) as journal: for e in json.load(journal): if e != {}: - (L, T, M) = (e["L"], datetime.fromtimestamp(e["T"]), e["M"]) + (L, T, M, D) = (e["L"], datetime.fromtimestamp(e["T"]), e["M"], e.get("D", Details.ALL.name)) if L == 0: messages[0].append((T, M)) - elif M in messages[L]: - messages[L][M].append(T) else: - messages[L][M] = [T] + dd = Details[D] + if M not in messages[L]: + messages[L][M] = {} + if dd not in messages[L][M]: + messages[L][M][dd] = [] + messages[L][M][dd].append(T) os.remove(Action._storage) html = Action._htmDocStart + Action._htmHeadWarn diff --git a/tests/action_dailyReport.py b/tests/action_dailyReport.py index 279dc79..01ee63a 100644 --- a/tests/action_dailyReport.py +++ b/tests/action_dailyReport.py @@ -11,6 +11,10 @@ mail_filename = "email.dump" wAction = Action({"level": "WARN", "message": "WarnMsg {m}"}) iAction = Action({"level": "INFO", "message": "InfoMsg {m}"}) oAction = Action({"level": "OTHER", "message": "MiscMsg {m}"}) +wActFirst = Action({"level": "WARN", "message": "WarnMsg {m}", "details": "FIRST"}) +wActLast = Action({"level": "WARN", "message": "WarnMsg {m}", "details": "LAST"}) +wActFL = Action({"level": "WARN", "message": "WarnMsg {m}", "details": "FIRSTLAST"}) +wActNone = Action({"level": "WARN", "message": "WarnMsg {m}", "details": "NONE"}) def newEntry(m): return {"__REALTIME_TIMESTAMP": datetime.utcnow(), "m": m} @@ -18,7 +22,6 @@ def newEntry(m): def whenNewDayThenReport(): if os.path.exists(mail_filename): os.remove(mail_filename) - Action._hour = 0 oAction.act(newEntry("message1")) assert not os.path.exists(mail_filename) Action._hour = 25 @@ -26,10 +29,9 @@ def whenNewDayThenReport(): assert os.path.exists(mail_filename) os.remove(mail_filename) -def whenEmailThenCheckContents(): +def whenEmailThenCheck3Sections(): if os.path.exists(mail_filename): os.remove(mail_filename) - Action._hour = 0 wAction.act(newEntry("messageW")) iAction.act(newEntry("messageI")) Action._hour = 25 @@ -70,6 +72,73 @@ def whenEmailThenCheckContents(): assert nbMisc == 2 os.remove(mail_filename) +def _compareEmailWithExpected(expected): + assert os.path.exists(mail_filename) + reTime = re.compile(r"\d{4}(?:[- :.]\d{2}){6}\d{4}") + warnSeen = False + nbTimes = 0 + nbFirst = 0 + nbLast = 0 + line = "" + with open(mail_filename, 'rt') as m: + for l in m: + if l != "" and l[-1:] == "=": + line += l[:-1] + continue + elif l == "" and warnSeen: + break + line += l + if "WarnMsg" in line: + warnSeen = True + elif not warnSeen: + line = "" + continue + nbTimes += len(reTime.findall(line)) + if "From=C2=A0:" in line: + nbFirst += 1 + if "Until:" in line: + nbLast += 1 + if "" in line: + break + line = "" + seen = dict(warn = warnSeen, times = nbTimes, first = nbFirst, last = nbLast) + assert seen == expected, "Expected=" + str(expected) + " ≠ Seen=" + str(seen) + os.remove(mail_filename) + +def whenEmailThenCheckTimes(warnAction, expected): + if os.path.exists(mail_filename): + os.remove(mail_filename) + warnAction.act(newEntry("messageW")) + warnAction.act(newEntry("messageW")) + Action._hour = 25 + warnAction.act(newEntry("messageW")) + _compareEmailWithExpected(expected) + +def whenSeveralDetailsModesThenOnlyOneWarn(): + if os.path.exists(mail_filename): + os.remove(mail_filename) + wAction.act(newEntry("messageW")) + wAction.act(newEntry("messageW")) + wAction.act(newEntry("messageW")) + wAction.act(newEntry("messageW")) + wAction.act(newEntry("messageW")) + wActFirst.act(newEntry("messageW")) + wActFirst.act(newEntry("messageW")) + wActFirst.act(newEntry("messageW")) + wActLast.act(newEntry("messageW")) + wActLast.act(newEntry("messageW")) + wActLast.act(newEntry("messageW")) + wActFL.act(newEntry("messageW")) + wActFL.act(newEntry("messageW")) + wActFL.act(newEntry("messageW")) + wActFL.act(newEntry("messageW")) + wActNone.act(newEntry("messageW")) + wActNone.act(newEntry("messageW")) + wActNone.act(newEntry("messageW")) + Action._hour = 25 + wActNone.act(newEntry("messageW")) + _compareEmailWithExpected(dict(warn = True, times = 9, first = 2, last = 2)) + def whenReportThenNewSetOfMessages(): if os.path.exists(mail_filename): os.remove(mail_filename) @@ -77,9 +146,14 @@ def whenReportThenNewSetOfMessages(): oAction.act(newEntry("message3")) assert os.path.exists(mail_filename) os.remove(mail_filename) - whenEmailThenCheckContents() + whenEmailThenCheck3Sections() def unitTests(): whenNewDayThenReport() - whenEmailThenCheckContents() + whenEmailThenCheck3Sections() + whenEmailThenCheckTimes(wActFirst, dict(warn = True, times = 1, first = 1, last = 0)) + whenEmailThenCheckTimes(wActLast, dict(warn = True, times = 1, first = 0, last = 1)) + whenEmailThenCheckTimes(wActFL, dict(warn = True, times = 2, first = 1, last = 1)) + whenEmailThenCheckTimes(wActNone, dict(warn = True, times = 0, first = 0, last = 0)) + whenSeveralDetailsModesThenOnlyOneWarn() whenReportThenNewSetOfMessages()