Mentat

Poznámky ze schůzky 2024-07-23:

• Hlavičky v Idea _Mentat: vedle EventClass, EventSeverity, ResolvedAbuses použijeme TargetClass, TargetSeverity, TargetAbuses
• V rozhraní Source groups, Target groups (kde to dává smysl zbavujeme se 'abuse')
• Subclassy, Tresholding a relapse, filtry - bylo by vhodné zachovat, zvlášť pokud to povede k znovupoužití kódu

Traceback (most recent call last):
  File "/var/mentat/venv/bin/mentat-reporter.py", line 51, in <module>
    MentatReporterScript().run()
  File "/var/mentat/venv/lib/python3.11/site-packages/pyzenkit/baseapp.py", line 1560, in run
    self._stage_evaluate()
  File "/var/mentat/venv/lib/python3.11/site-packages/pyzenkit/baseapp.py", line 1495, in _stage_evaluate
    analysis = self.runlog_analyze(self._prepare_runlog())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mentat/venv/lib/python3.11/site-packages/pyzenkit/baseapp.py", line 1748, in runlog_analyze
    return self._sub_runlog_analyze(runlog, analysis)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mentat/venv/lib/python3.11/site-packages/mentat/module/reporter.py", line 549, in _sub_runlog_analyze
    source_stats = severity_stats['Source']
                   ~~~~~~~~~~~~~~^^^^^^^^^^
KeyError: 'Source'

I tried hot patching mentat-dev based on 19101ef5 and another issue occurred:

Traceback (most recent call last):
  File "/var/mentat/venv/bin/mentat-reporter.py", line 48, in <module>
    MentatReporterScript().run()
  File "/var/mentat/venv/lib/python3.11/site-packages/pyzenkit/baseapp.py", line 1559, in run
    self._stage_process()
  File "/var/mentat/venv/lib/python3.11/site-packages/pyzenkit/baseapp.py", line 1472, in _stage_process
    self._sub_stage_process()
  File "/var/mentat/venv/lib/python3.11/site-packages/pyzenkit/zenscript.py", line 353, in _sub_stage_process
    self.execute_script_command(cmdname)
  File "/var/mentat/venv/lib/python3.11/site-packages/pyzenkit/zenscript.py", line 410, in execute_script_command
    self.runlog[command_name] = cbk()  # pylint: disable=locally-disabled,not-callable
                                ^^^^^
  File "/var/mentat/venv/lib/python3.11/site-packages/mentat/module/reporter.py", line 389, in cbk_command_report
    result['reports'][group.name] = self._report_for_group(
                                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mentat/venv/lib/python3.11/site-packages/mentat/module/reporter.py", line 444, in _report_for_group
    result[severity] = reporter.report(
                       ^^^^^^^^^^^^^^^^
  File "/var/mentat/venv/lib/python3.11/site-packages/mentat/reports/event.py", line 187, in report
    events_passed_filters, aggregated_events, fltlog, passed_cnt = self.filter_events(group.name, events_fetched, is_target)
                                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mentat/venv/lib/python3.11/site-packages/mentat/reports/event.py", line 696, in filter_events
    filtered_groups, fallback_groups, fltlog = self.filter_one_event(src, event_copy, main_group, fltlog, is_target)
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mentat/venv/lib/python3.11/site-packages/mentat/reports/event.py", line 621, in filter_one_event
    filtered_groups, fltlog = self._filter_groups(groups, event, fltlog, is_target)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mentat/venv/lib/python3.11/site-packages/mentat/reports/event.py", line 578, in _filter_groups
    filter_list = self.settings_dict[group].setup_filters(self.filter_parser, self.filter_compiler, is_target)
                  ~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: 'abuse@xxxxxxx'

The actual name of the group is redacted. It specifically fails on this group though, the error occurred also on the second run.

Some reports contain wrong average packet sizes. For example https://mentat-dev.cesnet.cz/mentat/reports/M20241014TL-E7CXD/unauth - reported with 1, which does not compute from original data.

Rajmund Hruška wrote in #note-9:

I tried hot patching mentat-dev based on 19101ef5 and another issue occurred:

[...]

The actual name of the group is redacted. It specifically fails on this group though, the error occurred also on the second run.

This may have something to do with the fact, that this specific group is disabled on mentat-dev. There are 9 disabled groups total, but we probably do not hit them all (or at least the offending one is hit first):
https://mentat-dev.cesnet.cz/mentat/groups/list?state=disabled&submit=Search

Radko Krkoš wrote in #note-10:

Some reports contain wrong average packet sizes. For example https://mentat-dev.cesnet.cz/mentat/reports/M20241014TL-E7CXD/unauth - reported with 1, which does not compute from original data.

Missing +. Should be fixed now.

Radko Krkoš wrote in #note-11:

Rajmund Hruška wrote in #note-9:

I tried hot patching mentat-dev based on 19101ef5 and another issue occurred:

[...]

The actual name of the group is redacted. It specifically fails on this group though, the error occurred also on the second run.

This may have something to do with the fact, that this specific group is disabled on mentat-dev. There are 9 disabled groups total, but we probably do not hit them all (or at least the offending one is hit first):
https://mentat-dev.cesnet.cz/mentat/groups/list?state=disabled&submit=Search

WHOIS mapped IP to a network even though the network was from a disabled group. This should also be fixed now.