Mentat

Feature: Prototype implementation of next generation web user interface for Mentat system.

This commit represents huge amount of work and introduces new Python-based version of Hawat web user interface for Mentat system. It is built on top of the great Flask framework. Currently it is still work in progress and many features are implemented only as dummy stubs, however the core functionality is almost finished. Following is a list of current key features:

• Focus on modular architecture using Flask`s blueprint feature.
• Multi-layered configurability (defaults, multiple file overrides).
• Plugin system.
• Internationalization support.
• Support for multiple authentication systems.

This commit provides working albeit hollow starting point for future web interface development. IT IS NOT READY FOR PRODUCTION USAGE!!!

(Redmine issue: #3734,#3361)

Added some initial CSS styles to geoip and whois search view results.

(Redmine issue: #3734)

Added new helper methods into Jinja global namespace.

Added following helpers into Jinja global namespace to be used in templates:

• babel_format_datetime ⇒ flask_babel.format_datetime
• babel_format_timedelta ⇒ flask_babel.format_timedelta
• get_current_utc ⇒ datetime.datetime.utcnow

These helpers can be used for localized formating of timestamps in template views.

(Redmine issue: #3734)

Implemented framework for easy development of item CRUD views.

Application now provides easy to extend View classes for quick development of create,update,view,delete,list views. (Redmine issue: #3734)

Translation work into czech language. Full coverage of all currently existing strings.

(Redmine issue: #3734)

Improved and unified output of form errors using predefined site macro.

(Redmine issue: #3734)

Revision of the item CRUDL framework in Hawat.

The item manipulation framework (create,read,update,delete,list) was revised, simplified and better documented. All changes were implemented and tested on existing user management pluggable module. (Redmine issue: #3734,#3361)

Moved string output conversion from hawat.models.user.GuiUserModel to the base class.

(Redmine issue: #3734)

Implemented prototype pluggable module for network record management.

The module contains all CRUDL views, forms and templates to enable network record management. (Redmine issue: #3734)

Added post-action hooks to support additional tasks after successfull item create/update/delete action.

This will allow performing additional tasks like sending email about newly created user accounts, etc. (Redmine issue: #3734)

Implemented prefered locale selection feature.

Users may now choose prefered locale, which will be used for application language localization and datetime formating. (Redmine issue: #3734)

Refactoring: Renamed get_current_utc and get_current_local Jinja template helpers to get_datetime_utc and get_datetime_local respectively.

(Redmine issue: #3734)

Greatly improved translation coverage of existing web interface content.

(Redmine issue: #3734)

Added user account role management feature for web interface.

(Redmine issue: #3734)

Removed locale and timezone from database GroupModel.

This feature would introduce unnecessary complexity and will not be implemented. User might be member of multiple groups and it would not be clear which settings to use. (Redmine issue: #3734)

Added link to user account update to the detail view.

(Redmine issue: #3734)

Added link to network record update to the detail view.

(Redmine issue: #3734)

Added link to “My account” to main menu for quick access to the profile of currently logged in user.

(Redmine issue: #3734)

Implemented user membership and user managers feature into groups.

This feature is a reverse direction to the previous commit. User accounts can now be added into groups and can be designated as group managers. (Redmine issue: #3734)

Implemented last login time remembering for user accounts.

(Redmine issue: #3734)

Implemented parent - children relationship for groups.

(Redmine issue: #3734)

Implemented very simple skeleton of a script for quick SQL database initialization.

This script is a very simple helper capable of creating appropriate user accounts and databases. (Redmine issue: #3734)

Implemented prototype of reporting settings management.

(Redmine issue: #3734)

Design improvevent in list, show and creatupdate views.

Added delete action to item detail view, moved actions to toolbar in item list view, changed layout and page width in item list view. (Redmine issue: #3734)

Implemented skeleton of environment based user account registration page.

For the purposes of prototyping. (Redmine issue: #3734)

Implemented prototype of environment based registration process.

Work in progress, needs polishing. (Redmine issue: #3734)

Switched auth_env login process to request environment variable.

So far the auth_env module worked in development mode and performed login for hardcoded user. This is no longer necessary and this feature was removed. (Redmine issue: #3734)

Updated default configurations for Hawat.

Added better default value for default email sender, commented local Hawat configuration file. (Redmine issue: #3734)

Moved flask_login bootstrapping and logout feature to application core.

This will simplify the creation of additional authentication modules, because they have to take care only of custom login. As a side-effect the flask_login is now mandatory authentication provider, but this step brings an order into otherwise potentially chaotic environment. (Redmine issue: #3734)

Fix: Fixed broken action cancelation button.

Up to this point it was not possible to cancel action without valid form. Check for button press was after form validation, which made the whole feature useless. (Redmine issue: #3734)

Implemented the feature for redirection back to original page.

After performing certain actions (item create/update/delete, login, locale change) an attempt will be made to SECURELY redirect user back to original page beforefalling back to default redirection target. (Redmine issue: #3734)

Removed now unnecessary and obsolete methods that were supplying redirection targets in item manipulation views.

(Redmine issue: #3734)

Improved base item create/update/delete views.

Moved common features into common base class. Tweaked action status messages. (Redmine issue: #3734)

Added menu link for easy switching users with auth_dev authentication module.

(Redmine issue: #3734)

Fix: Fixed typo in dictionary key name.

(Redmine issue: #3734)

Improved user experience with geoip and whois database lookup pluggable modules.

Improved feedback messages and result templates for better user experience, unified module code. (Redmine issue: #3734)

Fix: Prevent redirection to item detail page after item deletion.

(Redmine issue: #3734)

Removed the 'abuses’ attribute from network record model.

Target emails will be stored in settings_reporting object. (Redmine issue: #3734)

Added developer role protected printout of raw item data to creatupdate views.

The printout is implemented using Jinja macro for reusability. (Redmine issue: #3734)

Moved global jinja variable to application core instead of bae view class.

(Redmine issue: #3734)

Included template context into dobefore() and doafter() method arguments.

(Redmine issue: #3734)

Enhanced implementation of filter management module.

The filter management plugable module is now fully implemented. (Redmine issue: #3734)

Small tweaks in a design of devtools module view.

(Redmine issue: #3734)

Changed default size of sparkchart images (made it smaller).

(Redmine issue: #3734)

Implemented Jinja template helper for detecting existence of arbitrary file.

(Redmine issue: #3734)

Implemented prototype of performance statistics module for Hawat.

This module is capable of displaying system performace statistic charts based on RRD library. Charts are already pregenerated by the mentat-statistician.py module, this module only renders the result page. Currently the location of chart directory is hardcoded and not configurable. This should be fixed in a similar way to for example geoip service configuration, so that there is only single configuration for chart location. (Redmine issue: #3734)

Implemented data model and migration script for event statistics.

The model represents PostgreSQL representation of MongoDB based event statistics. The current migration script was enhanced to migrate event statistics objects. (Redmine issue: #3734)

Implemented internal and SQL model for representing event reports.

This commit includes updates in MongoDB2SQL migration script. (Redmine issue: #3734)

Improved interface translations.

(Redmine issue: #3734)

Changed datatype of _CESNET.StorageTime attribute of IDEA message from integer to timestamp.

The numeric representation was kept in the schema for legacy purposes, but now it will be unified with other datetime attributes. As a result, the mentat.idea.internal or other compatible library is a required base class for handling IDEA messages in Mentat framework (because the idea.lite and idea.base do not properly define the custom _CESNET.StorageTime attribute).

(Redmine issue: #3734)

Fix: Fixed unit tests after changing mentat.idea.mongodb data model.

The _CESNET.StorageTime attribute is now a timestamp instead of integer. (Redmine issue: #3734)

Big improvements in IDEA event module for Hawat.

• Event search form is almost complete. The only thing missing is implementation of logical operation 'or’ and perhaps ability to show and hide unnecessary form elements.
• Event search result list was improved, but interlinking back to search form is in prototype state. There are many ways how this feature can be implemented and this will be polished after discussion and more testing.
• Event detail page design was improved, but interlinking back to search form is missing and waiting for resolution from previous paragraph to be consistent.
• Ability to search according to protocols was added to event search form.
• Created translations for the whole module.

• Event delete action
• Event download action

This commit introduces a lot of new work and not everything is yet in final state. However the live demonstration and hands-on experience is needed to move things forward.

(Redmine issue: #3734)

Implemented working prototype of Mentat system status module for Hawat.

This module is based on the same library as mentat-controller.py and can provide access to following information (non exhaustive list):

• status of real-time modules (runnning, not running, ...)
• status of cronjob modules (enabled, disabled)
• PID file analysis
• log file analysis
• ... and many more

The idea is to provide administrator access to information about the state of the whole system. Current implementation is only a skeleton and fraction of possible features, they will be implemented in future versions.

(Redmine issue: #3734)

Improved web interface translations.

(Redmine issue: #3734)

Implemented working prototype of Mentat database status module for Hawat.

The idea is to provide administrator access to information about the state of the whole database. Current implementation is only a proof of concept and fraction of possible features, they will be implemented in future versions.

(Redmine issue: #3734)

Minor improvements in Hawat menus.

• Made certain texts in the menu hidden on smaller displays, so that the menu does not take two rows.
• Menu reorganization - moved status overviews from MORE section to ADMIN section of menu.
• Minor menu translation tweaks.

(Redmine issue: #3734)

Next iteration in IDEA event search form design.

• Added the ability to search according to the 'source|target|detector types’ and 'description’.
• Added the ability to search according to the host address|ports|types. Host is an alias for Source OR Target.
• Minor design improvements in main menu and other page widgets.
• Updated mentat-precache.py according to changes mentioned above.
• Added new 'allow_empty’ feature to mentat-precache.py.

(Redmine issue: #3752,#3734)

Improvements in 'dbstatus’ Hawat pluggable module.

Improved implementation, design, output and coding style of 'dbstatus’ pluggable module. The view page now display basic characteristics of event table (number of rows, oldest and newest record, table size, ...) as well as the values of most important PostgreSQL configurations. (Redmine issue: #3734)

Fix: Fixed minor typo in translation.

(Redmine issue: #3734)

Improvements in 'status’ Hawat pluggable module.

Improved coding style of the blueprint. Improved output of the status view page. The page now provides more detailed information about module and cronjob status as well as detailed information about discovered log files, pid files and processes. This should still be considered as a draft, there are many more improvements yet to be done, especialy the regarding the design of the page. (Redmine issue: #3734)

Implemented actual resource authorization in Hawat web interface.

So far the access to various resources in Hawat web interface was protected only by authentication. This was to simplify the development process. As the first stable release approaches, it was time to harden the security. (Redmine issue: #3734)

Improved czech web interface translations.

(Redmine issue: #3734)

Upgraded FontAwesome from major version 4 to major version 5.

(Redmine issue: #3734)

Fix: Fixed the bug in searching for port numbers using event search form.

The PostgreSQL database complained, that it is unable to serach for integers using list of strings. Added explicit conversion from list of strings (received by user form) to list of integers. (Redmine issue: #3752,#3734)

Improved web interface translations.

(Redmine issue: #3734)

Added list of reporting filters to group detail view.

(Redmine issue: #3734)

Improvements related to account registration process using auth_env pluggable module for Hawat.

• Changed validator for user login from email to string containing restricted set of characters.
• Renamed field 'Login (eppn)' to just 'Login’. It made no sense to be used that was for example with Basic or certificate based authentication service.
• Implemented sendind of confirmation emails after successfull registration to system admins and to the user that performed the registration. The emails are generated using Jinja2 templates.
• Implemented sending of confirmation emails after successfull account activation to the user and as BCC to system admins.
• Added missing rollback() calls to base View implementations to resolve SQLAlchemy errors after unsuccessfull database operation.

(Redmine issue: #3734)

Translated informational emails, that are sent to the user and admins after account registration and activation.

• Registration email sent to the user is localized into the locale selected by the user during registration.
• Registration email sent to the admins is localized into the default application locale.
• Activation email sent to the user is localized into the user`s locale.

(Redmine issue: #3734)

Implemented new views for creating new networks and filters for particular group.

Based on previous commit new feature was implemented, that will enable group maintainers to add new filters and networks to that particular group (so far administration privileges were necessary). (Redmine issue: #3734)

Template design improvements for better presentation of user/group/settings detail page.

(Redmine issue: #3734)

Huge overhaul of Hawat`s Flask-based framework and all it`s modules.

This commit introduces huge amount of work that went into revision of existing Flask-based framework, on top of which the web interface and all of it`s modules are implemented. The process of registering modules and views into the application was greatly simplified (this will be used for further improvements of application menu). Code quality was improved using Pylint and Pyflakes tools. Documentation was improved. Many code features were refactored for performance, optimization and code style reasons. (Redmine issue: #3734,#3443)

Small tweaks in main menu entry organization.

(Redmine issue: #3734)

Implemented views for enabling/disabling reporting filters.

(Redmine issue: #3734)

Localized HTTP error pages.

(Redmine issue: #3734)

Improved Czech web interface translations.

(Redmine issue: #3734)

Fix: Fixed two minor bugs - invalid icon name and incorrect Jinja macro that failed in certain conditions.

(Redmine issue: #3734)

Fix: Fixed invalid icon name.

(Redmine issue: #3734)

Disabled broken and unfinished Javascript feature.

(Redmine issue: #3734)

Added mechanism for propagating information from build system to web interface.

For now we want to be able to display information about current build version in the footer of the web page. (Redmine issue: #3734)

Minor tweaks in Hawat`s top menu bar.

Removed unnecesary title. Added tooltip to locale switcher. (Redmine issue: #3734)

Fix: The IP4R PostgreSQL extension does not recognize certain network addressess.

User was presented with HTTP 500 error page without knowing what is actually wrong. Because the error is encountered at database level and the value passess form validation, it was necessary to analyze the exception thrown by database and present user with meaningfull message. For example 195.113.134.0/14 is not a valid network address from the point of view of IP4R extension, because there are more 1s than necessary and the extension does no adjusting on its own. This pathc solves only the error presentation to the user, actual solution stil has to be implemented. (Redmine issue: #3752,#3734)

Moved do_before_action hook to more appropriate place.

This hook should execute after each successfull form submission (except cancel action), otherwise it is not much usefull. (Redmine issue: #3734)

Improved network management module.

• more information is displayed in network list view
• fixed invalid permissions for network create view

(Redmine issue: #3734)

Implemented percentage displays in statistical tables.

(Redmine issue: #3734)

Improved report show and search view visuals.

Added new data displays, widgets and tooltips. (Redmine issue: #3734)

More web interface UI and UX improvements.

This patch focuses more on improving look and feel when working with event search form and event detail view. The interlinking with other parts of the system was improved when working from event detail page. Event action search dropdown functionality was also improved. (Redmine issue: #3734)

Tweaked event severity and classification display in event search and show views.

(Redmine issue: #3734)

Improved czech web interface translations.

(Redmine issue: #3734)

Improvements in the dbstatus pluggable module for Hawat.

The module is now capable of displaying information about following tables within the events database: events, thresholds, events_thresholded. (Redmine issue: #3734)

Improved czech web interface translations.

(Redmine issue: #3734)

Fix: Fixed link to incorrect dashboard view.

Overall event dashboards can not be calculated for particular group, link was meant to pount to reporting dashboard. (Redmine issue: #3734)

Improved/fixed czech web interface and report translations.

(Redmine issue: #3734)

Huge improvements in Hawat web interface.

This patch introduces huge amount of work related to Hawat web interface. I was just clicking through the interface and trying to improve everything that popped out at the moment. There are following improvements:

• Enabled DataTable localication plugins.
• Implemented common layout for object management views (list views).
• Redesigned application menu implementation.
• Enhanced application menu to support contextual item action menus in listing tables. All administration list views now use it to generate contextual action toolbars.
• Menu object is now used even to generate breadcrumb menus.
• Simplified working with view icons and their propagation into action menus.
• Implemented AJAX spinner and used AJAX to fetch raw report data to web interface. There is stil a lot of room for improvements, the idea is for the user to be able to inteligently browse through the loaded data in some way.
• Redesigned object administration views (users, groups, filters, networks).
• Cleaned up list of built-in icons.
• Improved web interface translations after the changes in this commit.
• Fixed the bug in serving report data files.

(Redmine issue: #3734,#3443)

Added new Jinja2 function for getting reference to view class.

This method can be usefull for accessing view metadata stored as class properties like icons, titles, etc. (Redmine issue: #3734)

Fix: Improved _layout_list.html base template to be more resistant to nonexistence of data for breadcrumbs and action menu widgets.

(Redmine issue: #3734)

Fix: Minor fixes and improvements.

(Redmine issue: #3734)

Added logging of directly served files.

(Redmine issue: #3734)

Added new action button for searching related events from report detail view.

(Redmine issue: #3734)

Fix: Fixed broken event dashboard view.

(Redmine issue: #3734)