Feature #3378

Task #3374: Migrate all core modules from legacy Mentat

Implement mentat-reporter.py module

Added by Jan Mach over 2 years ago. Updated 12 months ago.

Status:ClosedStart date:03/22/2017
Priority:NormalDue date:
Assignee:Jan Mach% Done:

100%

Category:Development - Core
Target version:2.0

Description

Migrate, redesign and implement periodical reporting module. Make sure to implement the support for multiple languages.

  • Should support multiple classes in _CESNET.EventClass (for example bot & cc)
  • Should strip _CESNET namespace and legacy keys (ts, ts_u and similar) for Idea events, attached to reports

Associated revisions

Revision 716d198a
Added by Jan Mach over 1 year ago

Implemented the prototype of mentat-reporter.py module.

This commit introduces the Pythonic reimplementation of mentat-reporter-ng module. It is complete from roughly 70 percent, there are many features that stil need to be implemented as well as some necessary polishing.

The most important next implementation steps:
  • thresholding and relapse
  • email template improvements
  • report section legends with translations
  • report web interface improvements
  • unauthorized access to reports
  • unit testing and bulletproofing

(Redmine issue: #3378)

Revision eca46762
Added by Jan Mach over 1 year ago

Enhanced mentat-reporter.py module with source filtering feature.

Previous implementation of reporter did not correctly filtered out sources that did not belong to abuge group`s network address space. It was posssible for external sources to be listed in the report as well as internal ones. This was caused by the fact, that enricher precalculates the ResolvedAbuses attribute, but this field gets calculated from the concatenated list of all sources, that are found in the event and the mapping information between the source and abuse gets lost. For correct functionality it was necessary to perform additional source based filtering of the events fetched from database for particular abuse group.

(Redmine issue: #3378)

Revision 340d5b9b
Added by Jan Mach over 1 year ago

Implemented the “relapse” feature into mentat-reporter.py.

The reporting module is now able to detect relapses of thresholded events and resend them in separate section of next report. (Redmine issue: #3378)

Revision 2b9bbb09
Added by Jan Mach over 1 year ago

Fix: Fixed invalid method definitions and failing unit tests.

The TTL argument was missing from definition of some methods related to relapse feature. (Redmine issue: #3378)

Revision a10f56a1
Added by Jan Mach over 1 year ago

Finished implmenting missing legacy features into mentat-reporter.py.

  • Implemented export into flat CSV report attachments.
  • Implemented support for compressed report attachments.
  • Implemented support for report redirection.
  • Implemented support for choosing report locale into reporting settings web interface.

(Redmine issue: #3378)

Revision d8a6d57a
Added by Jan Mach about 1 year ago

Improvements in mentat-reporter.py.

  • Reporter no longer pre-fetches list of groups, that have any events to report. This simplifies the reporting algorithm and fixes the bug with relapsed events not being reported.
  • Revised and improved lines written to log files.
  • Removed the period argument from functions, where it was not necessary.
  • Activated the enabled feature for groups. Only groups with enabled attribute set to True will be processed.
  • Improved code comments.

(Redmine issue: #3378)

Revision 764d7345
Added by Jan Mach about 1 year ago

Improvements in periodical event report emails.

  • Each report section representing an event class now have localized human readable label and link pointing to more resources about that particualr event class.
  • Maintained full report translations into czech language.
  • Started reporting documentation page.
  • Improvements in both summary and extra report templates.

(Redmine issue: #3378)

Revision 1f664121
Added by Jan Mach about 1 year ago

Improved mentat-reporter.py module to be capable of reporting events tagged with 'Test’ category.

This feature can be usefull for development and testing purposes. (Redmine issue: #3378)

Revision b1d367dd
Added by Jan Mach about 1 year ago

Improved documentation of mentat-reporter.py module.

(Redmine issue: #3378,#3361)

Revision 572d964b
Added by Jan Mach about 1 year ago

Improved report templates for mentat-reporter.py module.

Added application logger reference to Jinja2 report templates. The logger is now used to inform administrator about missign event class labels and references. Additional minor tweaks in report template design. (Redmine issue: #3378)

Revision b50a86ad
Added by Jan Mach about 1 year ago

Implemented proper report target email propagation to database and then further to web interface.

The mentat-reporter.py module now correctly recognizes the 'mute’ reporting setting and in case email is really sent it registers that fact into database record. This information can be then displayed to the user in web interface. (Redmine issue: #3378)

Revision d6b49f24
Added by Jan Mach about 1 year ago

Improved output of runlog-view and runlogs-evaluate actions of mentat-reporter.py module.

(Redmine issue: #3378)

Revision f046f617
Added by Jan Mach about 1 year ago

Fixed invalid module names in mentat-reporter.py configuration file.

(Redmine issue: #3378)

Revision 56503806
Added by Jan Mach about 1 year ago

Fixes and improvements in mentat-reporter.py module.

There was a bug in reporter module, that caused the events_thresholded cache to be entirely stripped off of any records after each reporting cycle. The cleanup algorithm was all wrong, so the relapse mechanism could never work. This issue is fixed now, but it was necessary to alter SQL schema and make some additional API changes to make everything work. Additionally to have more control over the reporting process the logging features of the module were improved. (Redmine issue: #3378)

Revision d75925cf
Added by Jan Mach about 1 year ago

Removed extra space in event report templates.

(Redmine issue: #3378)

Revision f35f7da7
Added by Jan Mach about 1 year ago

Improvements in mentat-reporter.py module.

  • Report emails contain additional email headers for more automated processing options.
  • Added legacy report email headers for backwards compatibility purposes.
  • Enabled more verbose logging of thresholding and relapse cache.

(Redmine issue: #3378)

Revision 000f743a
Added by Jan Mach about 1 year ago

Improved reporting documentation page.

(Redmine issue: #3378,#3361)

Revision c5e0a14a
Added by Jan Mach about 1 year ago

Changed timestamps to be labeled as in UTC instead of local time in reports.

(Redmine issue: #3378)

Revision c7dba53c
Added by Jan Mach about 1 year ago

Added additional extented header to extra reports containing source IP.

(Redmine issue: #3378)

Revision 488e10b2
Added by Jan Mach about 1 year ago

This commit features big redesign of reporting configuration features.

  • Enhanced data model for reporting settings to contain timezone selection for generated reports.
  • Changed data model for reporting settings to allow Null values for many of the settings. In this case system default will be used.
  • Updated utility class for configuring reporting settings to enable use of new features mentioned above.
  • Updated utility class for configuring reporting settings to enable enforcing most of the values by administrator launching the reporting script.
  • Implemented new base form radio field that accepts empty option as valid. This enables implementing for example three state choices like True, False, None, which is great for some reporting settings.
  • Updated all Hawat views for working with reporting settings to use new features.
  • Enhanced reporter template built-in features, mainly for working with timezones.
  • Changed default reporter templates to output dates in chosen timezone.
  • Translated all new features to czech localization.

The main result of this commit is, that each group can configure in which timezone it wants to have the dates in the report to be displayed.

(Redmine issue: #3378)

Revision da327d97
Added by Jan Mach about 1 year ago

Improved presentation of event time window in reports.

Time window is now separated into two columns with fixed width to preserve the table layout with different localizations. (Redmine issue: #3378)

Revision a4de025b
Added by Jan Mach about 1 year ago

Further improvements in report templates.

  • Implemented Jinja2 macro for calculating value sets.
  • Displayed set of relevant protocols to each event in report.
  • Displayed event count using localized format_decimal macro.
  • Changed table border to more prettier UTF8 character.

(Redmine issue: #3378)

Revision 643a3ca9
Added by Jan Mach 12 months ago

Fix: Made time window columns in report one character wider to stop breaking the table layout.

(Redmine issue: #3378)

Revision 023f002f
Added by Jan Mach 12 months ago

Minor fixes in report related features.

  • Tweaked report templates: made Count column more narrow, aligned Count title to right to correspond with column value.
  • Fixed translation typo.
  • Fixed displaying report messages in web interface (message content must be escaped first to display characters like < and >.

(Redmine issue: #3378)

Revision 9252b125
Added by Jan Mach 12 months ago

Improved reporter to sort report attachments into subdirectories according to the report timestamp.

On busy server the report storage can easily contain hundreds of thousands of files and things can start to get really slow when working with that particular directory. This patch will solve that problem by sorting reports according to the timestamp within the report file name. (Redmine issue: #3378)

Revision 99bad520
Added by Jan Mach 12 months ago

Fix: Added separate constants for event report types.

(Redmine issue: #3378)

Revision 57f93b42
Added by Jan Mach 12 months ago

Added support for Return-Path email header to base report email class.

(Redmine issue: #3377,#3378)

Revision f91e1151
Added by Jan Mach 12 months ago

Fix: Fixed example configuration for force_attachments in mentat-reporter.py configuration file.

(Redmine issue: #3378)

Revision b0882ff5
Added by Jan Mach 12 months ago

Fix: Used correct constants for setting report type in mentat-reporter.py module.

(Redmine issue: #3378)

History

#1 Updated by Pavel Kácha over 2 years ago

  • Description updated (diff)

#2 Updated by Pavel Kácha over 2 years ago

  • Description updated (diff)

#3 Updated by Jan Mach 12 months ago

  • Subject changed from Implement mentat-reporter-ng.py module to Implement mentat-reporter.py module
  • Status changed from New to Closed
  • % Done changed from 0 to 100

Current state of this module is sufficient for production environment. We are finally releasing 2.0 version of Mentat system, so the period of frantic coding and implementation chaos is over. Any further improvements of this module will be done as they should in separate Redmine issues.

Also available in: Atom PDF