Task #3734

Task #3374: Migrate all core modules from legacy Mentat

Migrate Hawat web user inteface from Perl-base to Python-based Mentat framework

Added by Jan Mach almost 2 years ago. Updated about 1 year ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:Jan Mach% Done:

100%

Category:Development - GUI
Target version:2.0

Description

The web interface for Mentat system called Hawat must be rewritten and migrated to Python-based Mentat framework.


Related issues

Related to Mentat - Task #3362: Migrate Mentat system to new hardware Closed 03/21/2017

Associated revisions

Revision 0a724ce9
Added by Jan Mach almost 2 years ago

Feature: Prototype implementation of next generation web user interface for Mentat system.

This commit represents huge amount of work and introduces new Python-based version of Hawat web user interface for Mentat system. It is built on top of the great Flask framework. Currently it is still work in progress and many features are implemented only as dummy stubs, however the core functionality is almost finished. Following is a list of current key features:

  • Focus on modular architecture using Flask`s blueprint feature.
  • Multi-layered configurability (defaults, multiple file overrides).
  • Plugin system.
  • Internationalization support.
  • Support for multiple authentication systems.

This commit provides working albeit hollow starting point for future web interface development. IT IS NOT READY FOR PRODUCTION USAGE!!!

(Redmine issue: #3734,#3361)

Revision e9950c3b
Added by Jan Mach almost 2 years ago

Implemented demonstration whois and geoip lookup functionality in Hawat web user interface.

The lookup results are not styled yet, only proof of concept. (Redmine issue: #3734)

Revision fb5104a6
Added by Jan Mach almost 2 years ago

Added some initial CSS styles to geoip and whois search view results.

(Redmine issue: #3734)

Revision 6d147b6f
Added by Jan Mach almost 2 years ago

Implemented generic view endpoint protection mechanism for Jinja templates.

Objects in Jinja templates can now be protected with permission protection mechanism. This can be used for hiding links, that are inaccessible to current user etc. As a proof of concept raw results output of geoip and whois search are now restricted only to users with developer role. (Redmine issue: #3734)

Revision 1089a2a3
Added by Jan Mach almost 2 years ago

Added new helper methods into Jinja global namespace.

Added following helpers into Jinja global namespace to be used in templates:

  • babel_format_datetime ⇒ flask_babel.format_datetime
  • babel_format_timedelta ⇒ flask_babel.format_timedelta
  • get_current_utc ⇒ datetime.datetime.utcnow

These helpers can be used for localized formating of timestamps in template views.

(Redmine issue: #3734)

Revision cbb270ef
Added by Jan Mach almost 2 years ago

The hawat application factory now returns production level interface by default instead of development level.

The default configuration object for Hawat application factory is now hawat.config.ProductionConfig. (Redmine issue: #3734)

Revision 09e07ef9
Added by Jan Mach almost 2 years ago

Implemented framework for easy development of item CRUD views.

Application now provides easy to extend View classes for quick development of create,update,view,delete,list views. (Redmine issue: #3734)

Revision 8c1f9ce5
Added by Jan Mach almost 2 years ago

Implemented prototype CRUDL views for user account management.

(Redmine issue: #3734)

Revision 71dbaac3
Added by Jan Mach almost 2 years ago

Translation work into czech language. Full coverage of all currently existing strings.

(Redmine issue: #3734)

Revision b7ffa45a
Added by Jan Mach almost 2 years ago

Improved documentation of hawat.config module.

(Redmine issue: #3734,#3361)

Revision 2e337722
Added by Jan Mach almost 2 years ago

Improved and unified output of form errors using predefined site macro.

(Redmine issue: #3734)

Revision 7de3ad42
Added by Jan Mach almost 2 years ago

PostgreSQL migration: Reimplemented DBAL, authentication layer and user management modules in Hawat.

Base on the previous work the DBAL in Hawat was reimplemented to use the new mentat.services.sqlstorage module and sqlalchemy backend. The authentication layer was rewritten to use the new dabase interface. The user management plugin was rewritten to use new database interface.

(Redmine issue: #3752,#3734)

Revision c65c3126
Added by Jan Mach almost 2 years ago

Revision of the item CRUDL framework in Hawat.

The item manipulation framework (create,read,update,delete,list) was revised, simplified and better documented. All changes were implemented and tested on existing user management pluggable module. (Redmine issue: #3734,#3361)

Revision ddf2c570
Added by Jan Mach almost 2 years ago

Implemented prototype pluggable module for group management.

The module contains all CRUDL views, forms and templates to enable group management. (Redmine issue: #3734)

Revision 46b873a5
Added by Jan Mach almost 2 years ago

Moved string output conversion from hawat.models.user.GuiUserModel to the base class.

(Redmine issue: #3734)

Revision e697ee1a
Added by Jan Mach almost 2 years ago

Implemented prototype pluggable module for reporting filter management.

The module contains all CRUDL views, forms and templates to enable reporting filter management. (Redmine issue: #3734)

Revision 30b64d8c
Added by Jan Mach almost 2 years ago

Implemented prototype pluggable module for network record management.

The module contains all CRUDL views, forms and templates to enable network record management. (Redmine issue: #3734)

Revision 6d9aacd0
Added by Jan Mach almost 2 years ago

Fix: Fixed minor bug in attribute name.

(Redmine issue: #3734)

Revision 3a5bb058
Added by Jan Mach almost 2 years ago

Added post-action hooks to support additional tasks after successfull item create/update/delete action.

This will allow performing additional tasks like sending email about newly created user accounts, etc. (Redmine issue: #3734)

Revision 6934f0ff
Added by Jan Mach almost 2 years ago

Implemented prefered timezone selection feature.

Users may now choose prefered timezone, which will be used when displaying time data. (Redmine issue: #3734)

Revision 8d8ee5ef
Added by Jan Mach almost 2 years ago

Implemented prefered locale selection feature.

Users may now choose prefered locale, which will be used for application language localization and datetime formating. (Redmine issue: #3734)

Revision a25373b6
Added by Jan Mach almost 2 years ago

Documentation improvements.

(Redmine issue: #3361,#3734)

Revision b6b9be8d
Added by Jan Mach almost 2 years ago

Refactoring: Renamed get_current_utc and get_current_local Jinja template helpers to get_datetime_utc and get_datetime_local respectively.

(Redmine issue: #3734)

Revision c3c54d42
Added by Jan Mach almost 2 years ago

Added current datetime timestamp to global Jinja template variables.

This timestamp more or less represents the generation time of the page and may serve as common baseline for multiple time operations. (Redmine issue: #3734)

Revision 16c4cc89
Added by Jan Mach almost 2 years ago

Greatly improved translation coverage of existing web interface content.

(Redmine issue: #3734)

Revision 5a05f74a
Added by Jan Mach almost 2 years ago

Added limits to the size of selectpicker widget.

(Redmine issue: #3734)

Revision 65bcc277
Added by Jan Mach almost 2 years ago

Added user account role management feature for web interface.

(Redmine issue: #3734)

Revision d3a56194
Added by Jan Mach almost 2 years ago

Created macros for rendering form items and modified all existing forms to use them.

This will enable very easy modification of rendering of all form items. (Redmine issue: #3734)

Revision 15f27f8b
Added by Jan Mach almost 2 years ago

Removed locale and timezone from database GroupModel.

This feature would introduce unnecessary complexity and will not be implemented. User might be member of multiple groups and it would not be clear which settings to use. (Redmine issue: #3734)

Revision 3ef06827
Added by Jan Mach almost 2 years ago

Minor style improvements in user account detail view.

(Redmine issue: #3734)

Revision 902dd56a
Added by Jan Mach almost 2 years ago

Added link to user account update to the detail view.

(Redmine issue: #3734)

Revision c93e53ad
Added by Jan Mach almost 2 years ago

Added link to group update to the detail view.

(Redmine issue: #3734)

Revision b6eec5b6
Added by Jan Mach almost 2 years ago

Added link to network record update to the detail view.

(Redmine issue: #3734)

Revision 5813eb16
Added by Jan Mach almost 2 years ago

Added link to reporting filter update to the detail view.

(Redmine issue: #3734)

Revision ca14e80b
Added by Jan Mach almost 2 years ago

Added link to “My account” to main menu for quick access to the profile of currently logged in user.

(Redmine issue: #3734)

Revision de571bc4
Added by Jan Mach almost 2 years ago

Implemented group membership and group management feature into user accounts.

User accounts can now be added into groups and can be designated as group managers. (Redmine issue: #3734)

Revision f85abe62
Added by Jan Mach almost 2 years ago

Implemented user membership and user managers feature into groups.

This feature is a reverse direction to the previous commit. User accounts can now be added into groups and can be designated as group managers. (Redmine issue: #3734)

Revision 48421e0c
Added by Jan Mach almost 2 years ago

Limited number of values to be displayed in multi variant of bootstrap-select.

(Redmine issue: #3734)

Revision 56d1deb6
Added by Jan Mach almost 2 years ago

Implemented last login time remembering for user accounts.

(Redmine issue: #3734)

Revision 8dc2e453
Added by Jan Mach almost 2 years ago

Translated the “before” string in item detail view.

(Redmine issue: #3734)

Revision d502cd1c
Added by Jan Mach almost 2 years ago

Implemented parent - children relationship for groups.

(Redmine issue: #3734)

Revision d54b32c2
Added by Jan Mach almost 2 years ago

Improved implementation of network record management.

  • Implemented network specific abuse contacts.
  • Implemented linking between groups and asociated networks.
  • Hardened network record form.

(Redmine issue: #3734)

Revision 9e550a9d
Added by Jan Mach almost 2 years ago

Implemented very simple skeleton of a script for quick SQL database initialization.

This script is a very simple helper capable of creating appropriate user accounts and databases. (Redmine issue: #3734)

Revision b864a68e
Added by Jan Mach almost 2 years ago

Added the dobefore() hook to item create/update/delete base views.

(Redmine issue: #3734)

Revision 87c3d6e6
Added by Jan Mach almost 2 years ago

Implemented prototype of reporting settings management.

(Redmine issue: #3734)

Revision 1b63de61
Added by Jan Mach almost 2 years ago

Improved breadcrumbs navigation for object management views.

(Redmine issue: #3734)

Revision fc1952e6
Added by Jan Mach almost 2 years ago

Design improvevent in list, show and creatupdate views.

Added delete action to item detail view, moved actions to toolbar in item list view, changed layout and page width in item list view. (Redmine issue: #3734)

Revision 6e5fedaa
Added by Jan Mach almost 2 years ago

Design improvevents in list, show and creatupdate views.

Added delete action to item detail view, moved actions to toolbar in item list view, changed layout and page width in item list view. (Redmine issue: #3734)

Revision 9ae94cb8
Added by Jan Mach almost 2 years ago

Implemented skeleton of environment based user account registration page.

For the purposes of prototyping. (Redmine issue: #3734)

Revision 68ecb42c
Added by Jan Mach almost 2 years ago

Added separate side-menus for authenticated and anonymous users.

The original menu was dependent on auth_env blueprint and did not allow customizations. New menus allow any authentication modules to append any number of menu entries. Additionally, the menu rendering macro was moved to design blueprint and all macros were commented. ((Redmine issue: #3734)

Revision 14144a7d
Added by Jan Mach almost 2 years ago

Implemented prototype of environment based registration process.

Work in progress, needs polishing. (Redmine issue: #3734)

Revision 1d060229
Added by Jan Mach almost 2 years ago

Activated account existence checking feature.

(Redmine issue: #3734)

Revision 589b3f05
Added by Jan Mach almost 2 years ago

Switched auth_env login process to request environment variable.

So far the auth_env module worked in development mode and performed login for hardcoded user. This is no longer necessary and this feature was removed. (Redmine issue: #3734)

Revision b31d70ac
Added by Jan Mach almost 2 years ago

Fix: Added missig return statement.

(Redmine issue: #3734)

Revision 0d36bfa1
Added by Jan Mach almost 2 years ago

Updated default configurations for Hawat.

Added better default value for default email sender, commented local Hawat configuration file. (Redmine issue: #3734)

Revision c9d37d39
Added by Jan Mach almost 2 years ago

Implemented basic sending of account registration information emails.

(Redmine issue: #3734)

Revision a8ab5385
Added by Jan Mach almost 2 years ago

Moved flask_login bootstrapping and logout feature to application core.

This will simplify the creation of additional authentication modules, because they have to take care only of custom login. As a side-effect the flask_login is now mandatory authentication provider, but this step brings an order into otherwise potentially chaotic environment. (Redmine issue: #3734)

Revision 3002c284
Added by Jan Mach almost 2 years ago

Rerouted auth_env module from auth to auth_env namespace, made default login view configurable via application configuration.

(Redmine issue: #3734)

Revision 32777627
Added by Jan Mach almost 2 years ago

Fix: Fixed broken action cancelation button.

Up to this point it was not possible to cancel action without valid form. Check for button press was after form validation, which made the whole feature useless. (Redmine issue: #3734)

Revision 4b9dcc3a
Added by Jan Mach almost 2 years ago

Changed class of flash messages produced by flask_login to 'info’.

(Redmine issue: #3734)

Revision 50e8a7d6
Added by Jan Mach almost 2 years ago

Implemented the feature for redirection back to original page.

After performing certain actions (item create/update/delete, login, locale change) an attempt will be made to SECURELY redirect user back to original page beforefalling back to default redirection target. (Redmine issue: #3734)

Revision d2de5ca0
Added by Jan Mach almost 2 years ago

Refactoring: Renamed HawatCRUDLView to HawatDbmodelView.

(Redmine issue: #3734)

Revision fc0fc6bc
Added by Jan Mach almost 2 years ago

Removed now unnecessary and obsolete methods that were supplying redirection targets in item manipulation views.

(Redmine issue: #3734)

Revision cfb3b53f
Added by Jan Mach almost 2 years ago

Removed forgotten test printouts in redirection feature.

(Redmine issue: #3734)

Revision 128319ce
Added by Jan Mach almost 2 years ago

Improved base item create/update/delete views.

Moved common features into common base class. Tweaked action status messages. (Redmine issue: #3734)

Revision 0ada7d53
Added by Jan Mach almost 2 years ago

Rewritten the inheritance tree for user account forms to move common fields into parent classes.

(Redmine issue: #3734)

Revision 7487e4df
Added by Jan Mach almost 2 years ago

Added menu link for easy switching users with auth_dev authentication module.

(Redmine issue: #3734)

Revision 2a60ee00
Added by Jan Mach almost 2 years ago

Added specific view for displaying user account of currently logged-in user.

(Redmine issue: #3734)

Revision adaef13d
Added by Jan Mach almost 2 years ago

Fix: Fixed typo in dictionary key name.

(Redmine issue: #3734)

Revision a1bc0d91
Added by Jan Mach almost 2 years ago

Added support for Bootstrap tooltips.

(Redmine issue: #3734)

Revision 0614dd24
Added by Jan Mach almost 2 years ago

Improved user experience with geoip and whois database lookup pluggable modules.

Improved feedback messages and result templates for better user experience, unified module code. (Redmine issue: #3734)

Revision 7e364fbf
Added by Jan Mach almost 2 years ago

Greatly improved translation coverage of existing web interface content.

(Redmine issue: #3734)

Revision 51388bd1
Added by Jan Mach almost 2 years ago

Fix: Prevent redirection to item detail page after item deletion.

(Redmine issue: #3734)

Revision ce046dab
Added by Jan Mach almost 2 years ago

Item form hardening by adding more validators.

(Redmine issue: #3734)

Revision c6833b7a
Added by Jan Mach almost 2 years ago

Removed the 'abuses’ attribute from network record model.

Target emails will be stored in settings_reporting object. (Redmine issue: #3734)

Revision 7190df49
Added by Jan Mach almost 2 years ago

Fix: Fixed the syntax error.

(Redmine issue: #3734)

Revision a588a93b
Added by Jan Mach almost 2 years ago

Added developer role protected printout of raw item data to creatupdate views.

The printout is implemented using Jinja macro for reusability. (Redmine issue: #3734)

Revision 69b7496f
Added by Jan Mach almost 2 years ago

Added information about page generation time into page footer.

(Redmine issue: #3734)

Revision a8119dd3
Added by Jan Mach almost 2 years ago

Moved global jinja variable to application core instead of bae view class.

(Redmine issue: #3734)

Revision 04524fdb
Added by Jan Mach almost 2 years ago

Only admin may now change the group attribute for group networks.

(Redmine issue: #3734)

Revision 06b30777
Added by Jan Mach almost 2 years ago

Included template context into dobefore() and doafter() method arguments.

(Redmine issue: #3734)

Revision 911a14e4
Added by Jan Mach almost 2 years ago

Redesigned callback hooks in CRUDL base views.

  • Renamed dobefore() hook method to do_before_action().
  • Renamed doafter() hook method to do_after_action().
  • Added the do_before_render() hook method to enable custom modifications before template rendering.

(Redmine issue: #3734)

Revision 029809c2
Added by Jan Mach almost 2 years ago

Enhanced implementation of filter management module.

The filter management plugable module is now fully implemented. (Redmine issue: #3734)

Revision 6aebab5e
Added by Jan Mach almost 2 years ago

Fix: Fixed name errors after renaming custom Jinja functions.

(Redmine issue: #3734)

Revision 658ce2d7
Added by Jan Mach almost 2 years ago

Small tweaks in a design of devtools module view.

(Redmine issue: #3734)

Revision 783a3ace
Added by Jan Mach almost 2 years ago

Simplifications and improvements in mentat.stats.rrd statistical library.

  • Simplified definition of list of generated charts.
  • Unified code for generating chart and sparkchart specifications.
  • Added lookup() method for looking up list of charts that should be available.
  • Added 'last 2 years’ chart (main reason, we need a list of charts for user interface).

(Redmine issue: #3734)

Revision f2faaad1
Added by Jan Mach almost 2 years ago

Changed default size of sparkchart images (made it smaller).

(Redmine issue: #3734)

Revision 3a0c6d17
Added by Jan Mach almost 2 years ago

Added more data into chart metadata structure.

The reason was to enable better rendering of performance statistics view in web interface. (Redmine issue: #3734)

Revision 79bb62f8
Added by Jan Mach almost 2 years ago

Implemented Jinja template helper for detecting existence of arbitrary file.

(Redmine issue: #3734)

Revision 77b1f37a
Added by Jan Mach almost 2 years ago

Implemented base view for serving arbitrary files from arbitrary directories.

(Redmine issue: #3734)

Revision 84f9403b
Added by Jan Mach almost 2 years ago

Implemented prototype of performance statistics module for Hawat.

This module is capable of displaying system performace statistic charts based on RRD library. Charts are already pregenerated by the mentat-statistician.py module, this module only renders the result page. Currently the location of chart directory is hardcoded and not configurable. This should be fixed in a similar way to for example geoip service configuration, so that there is only single configuration for chart location. (Redmine issue: #3734)

Revision 49b68e53
Added by Jan Mach almost 2 years ago

Fix: Fixed incorrect endpoint name for sparkchart image links.

(Redmine issue: #3734)

Revision a2ae9ef6
Added by Jan Mach over 1 year ago

Implemented data model and migration script for event statistics.

The model represents PostgreSQL representation of MongoDB based event statistics. The current migration script was enhanced to migrate event statistics objects. (Redmine issue: #3734)

Revision fece2712
Added by Jan Mach over 1 year ago

Implemented prototype module for viewing overall event statistics.

The Dashboards module in Hawat user interface now contains prototype implementation for calculating and viewing overall event processing statistics. The module design is based on original Perl-based interface with many enhancements both on the surface (interface) and in the backend code. The work is not yet finished, some of the tasks that remain to be done are:

  • Links from data tables to other modules of the interface.
  • Missing sorting arrows in data table headers.
  • Interface tweaks.
  • Chart descriptions and/or help texts.
  • Search form tweaks or changes (current implementation is maybe too powerfull for general user).

(Redmine issue: #3734)

Revision bbb60696
Added by Jan Mach over 1 year ago

Implemented internal and SQL model for representing event reports.

This commit includes updates in MongoDB2SQL migration script. (Redmine issue: #3734)

Revision 04c7473e
Added by Jan Mach over 1 year ago

Implemented working prototype of report access module for Hawat.

This implementation still needs lots of polishing and there are following known major issues:

  • invalid encoding of report message text coming from migrated MongoDB records
  • some parts of the interface are not working yet (report remailing, report deletion)
  • unauthorized access to reports is not working
  • report data browser is not yet implemented
  • report search result statistics are not yet implemented

These issues will be resolved in separate commits. (Redmine issue: #3734)

Revision f82b6aed
Added by Jan Mach over 1 year ago

Improved interface translations.

(Redmine issue: #3734)

Revision f604639e
Added by Jan Mach over 1 year ago

Implemented conversion library for converting IDEA messages into dict structures containing only primitive data types.

These structures will contain only most primitive data types (int, str, bool). These will be then stored into PostgreSQL`s native data type jsonb. These conversions are basically reverse operation to what mentat.idea.internal library does.

(Redmine issue: #3734)

Revision 1ad0d896
Added by Jan Mach over 1 year ago

Changed datatype of _CESNET.StorageTime attribute of IDEA message from integer to timestamp.

The numeric representation was kept in the schema for legacy purposes, but now it will be unified with other datetime attributes. As a result, the mentat.idea.internal or other compatible library is a required base class for handling IDEA messages in Mentat framework (because the idea.lite and idea.base do not properly define the custom _CESNET.StorageTime attribute).

(Redmine issue: #3734)

Revision 7e2682d0
Added by Jan Mach over 1 year ago

Implemented SQL model for representing IDEA events.

This commit contains following work:

  • SQL data model for representing IDEA events in PostgreSQL.
  • SQL schema for IDEA event table and indices.
  • Event storage service - database abstraction layer for working with IDEA events.
  • Migration script for converting events from MongoDB collection into PostgreSQL table.

This code is based on initial prototype and work done by Radko Krkoš <> and he deserves credit for a lot of research and pilot work.

(Redmine issue: #3734)

Revision 2503020f
Added by Jan Mach over 1 year ago

Fix: Fixed unit tests after changing mentat.idea.mongodb data model.

The _CESNET.StorageTime attribute is now a timestamp instead of integer. (Redmine issue: #3734)

Revision ad5bff46
Added by Jan Mach over 1 year ago

Implemented working prototype of event access module for Hawat.

This implementation still needs lots of polishing and there are following known major issues:

  • event search form is not yet fully implemented
  • event search result page is only prototype
  • event detail page is only prototype

These issues will be resolved in separate commits. (Redmine issue: #3734)

Revision 5eb88ce7
Added by Jan Mach over 1 year ago

Big improvements in IDEA event module for Hawat.

Following features were implemented or improved:
  • Event search form is almost complete. The only thing missing is implementation of logical operation 'or’ and perhaps ability to show and hide unnecessary form elements.
  • Event search result list was improved, but interlinking back to search form is in prototype state. There are many ways how this feature can be implemented and this will be polished after discussion and more testing.
  • Event detail page design was improved, but interlinking back to search form is missing and waiting for resolution from previous paragraph to be consistent.
  • Ability to search according to protocols was added to event search form.
  • Created translations for the whole module.
Features to be implemented:
  • Event delete action
  • Event download action

This commit introduces a lot of new work and not everything is yet in final state. However the live demonstration and hands-on experience is needed to move things forward.

(Redmine issue: #3734)

Revision 33582b2e
Added by Jan Mach over 1 year ago

Implemented event download feature for saving IDEA events into JSON files.

(Redmine issue: #3734)

Revision ca509d7c
Added by Jan Mach over 1 year ago

Implemented working prototype of Mentat system status module for Hawat.

This module is based on the same library as mentat-controller.py and can provide access to following information (non exhaustive list):

  • status of real-time modules (runnning, not running, ...)
  • status of cronjob modules (enabled, disabled)
  • PID file analysis
  • log file analysis
  • ... and many more

The idea is to provide administrator access to information about the state of the whole system. Current implementation is only a skeleton and fraction of possible features, they will be implemented in future versions.

(Redmine issue: #3734)

Revision b7483aa2
Added by Jan Mach over 1 year ago

Next iteration in event search form design.

Following experimental changes have been made in event search form:

  • highlighting of matched searches in result table
  • removed row index column and event delete action
  • information about current page and range of displayed entries was moved to the pager
  • created different styles of dropdown search action buttons within result table to evaluate best approach
  • displaying last SQL query for debugging purposes
  • redesigned event search form (hiding of unnecessary fields, explicit search according to storage time, removed logop between source_addrs and target_addrs)

This is an experimental work to allow testing the interface on development server, some things may yet change. (Redmine issue: #3734)

Revision 043c19a8
Added by Jan Mach over 1 year ago

Improved web interface translations.

(Redmine issue: #3734)

Revision b0a3c1c3
Added by Jan Mach over 1 year ago

Fix: Corrected the name of itemset 'detector’ cache file.

(Redmine issue: #3734)

Revision 7fab7572
Added by Jan Mach over 1 year ago

Implemented working prototype of Mentat database status module for Hawat.

The idea is to provide administrator access to information about the state of the whole database. Current implementation is only a proof of concept and fraction of possible features, they will be implemented in future versions.

(Redmine issue: #3734)

Revision 9119200d
Added by Jan Mach over 1 year ago

Minor improvements in web interface translations into Czech.

(Redmine issue: #3734)

Revision 27504166
Added by Jan Mach over 1 year ago

Minor improvements in Hawat menus.

  • Made certain texts in the menu hidden on smaller displays, so that the menu does not take two rows.
  • Menu reorganization - moved status overviews from MORE section to ADMIN section of menu.
  • Minor menu translation tweaks.

(Redmine issue: #3734)

Revision ec747a09
Added by Jan Mach over 1 year ago

Fix: Fixed the call for Jinja2 default() filter.

(Redmine issue: #3734)

Revision d0b0a2f2
Added by Jan Mach over 1 year ago

Next iteration in IDEA event search form design.

  • Added the ability to search according to the 'source|target|detector types’ and 'description’.
  • Added the ability to search according to the host address|ports|types. Host is an alias for Source OR Target.
  • Minor design improvements in main menu and other page widgets.
  • Updated mentat-precache.py according to changes mentioned above.
  • Added new 'allow_empty’ feature to mentat-precache.py.

(Redmine issue: #3752,#3734)

Revision 73bd2f51
Added by Jan Mach over 1 year ago

Next iteration in IDEA event search form, result page and detail page design.

This commit contains following design changes and fixes:

  • Fixed the issue with duplicated HTML identifiers for datetime selects.
  • Written macros for rendering search dropdown buttons for source addresses, categories and abuse groups.
  • Modified accessors in internal IDEA representation.
  • Implemented implicit sorting by storage time in case it was chosen in the search form.
  • Additional minor design fixes and tweaks.

(Redmine issue: #3734)

Revision 0d6a4633
Added by Jan Mach over 1 year ago

Improvements in 'dbstatus’ Hawat pluggable module.

Improved implementation, design, output and coding style of 'dbstatus’ pluggable module. The view page now display basic characteristics of event table (number of rows, oldest and newest record, table size, ...) as well as the values of most important PostgreSQL configurations. (Redmine issue: #3734)

Revision 3d381e66
Added by Jan Mach over 1 year ago

Improved web interface translations.

(Redmine issue: #3734)

Revision 2f3b2179
Added by Jan Mach over 1 year ago

Fix: Fixed minor typo in translation.

(Redmine issue: #3734)

Revision f2bb61d6
Added by Jan Mach over 1 year ago

Fix: Fixed bugs in database status view template.

(Redmine issue: #3734)

Revision d117c936
Added by Jan Mach over 1 year ago

Improvements in 'status’ Hawat pluggable module.

Improved coding style of the blueprint. Improved output of the status view page. The page now provides more detailed information about module and cronjob status as well as detailed information about discovered log files, pid files and processes. This should still be considered as a draft, there are many more improvements yet to be done, especialy the regarding the design of the page. (Redmine issue: #3734)

Revision 084f7bbc
Added by Jan Mach over 1 year ago

Improved czech web interface translations.

(Redmine issue: #3734)

Revision 552acc91
Added by Jan Mach over 1 year ago

Implemented actual resource authorization in Hawat web interface.

So far the access to various resources in Hawat web interface was protected only by authentication. This was to simplify the development process. As the first stable release approaches, it was time to harden the security. (Redmine issue: #3734)

Revision d01d7bc7
Added by Jan Mach over 1 year ago

Implemented unauthenticated access to reports.

All reports generated by mentat-reporter.py are now accessible via unauthenticated URL protected by security token. This feature enables easy sharing of security information even with people without full access to web interface. (Redmine issue: #3734)

Revision 221c52dd
Added by Jan Mach over 1 year ago

Improved czech web interface translations.

(Redmine issue: #3734)

Revision 0de1e5e4
Added by Jan Mach over 1 year ago

Fix: Exception should not be raised in user_loader callback in Flask-Login.

(Redmine issue: #3734)

Revision 79614baa
Added by Jan Mach over 1 year ago

Upgraded FontAwesome from major version 4 to major version 5.

(Redmine issue: #3734)

Revision 08ea1935
Added by Jan Mach over 1 year ago

Implemented the pager feature into BaseSearchView, implemented the pager feature into report searching Hawat module.

(Redmine issue: #3734)

Revision b8accc98
Added by Jan Mach over 1 year ago

Fix: Fixed the bug in searching for port numbers using event search form.

The PostgreSQL database complained, that it is unable to serach for integers using list of strings. Added explicit conversion from list of strings (received by user form) to list of integers. (Redmine issue: #3752,#3734)

Revision 88e5e0e1
Added by Jan Mach over 1 year ago

Upgraded third-party web interface libraries.

(Redmine issue: #3734,#3387)

Revision 94c365c3
Added by Jan Mach over 1 year ago

Improved web interface translations.

(Redmine issue: #3734)

Revision b764b8df
Added by Jan Mach over 1 year ago

Implemented quick measurement of request processing time in Hawat.

Request processing time is measured from the correct request beginning point in Flask (using beforerequest decorator) up to the moment of rendering Jinja2 template for particular page. There is a bit of error, because some tasks performed after page rendering might not get measured, but this feature is intended to measure mainly length of database queries. For most use cases this solution should be enough, there is always debug toolbar for developers. (Redmine issue: #3734)

Revision 9ee218fa
Added by Jan Mach over 1 year ago

Added list of reporting filters to group detail view.

(Redmine issue: #3734)

Revision 8481b458
Added by Jan Mach over 1 year ago

Added constants for role names to Hawat`s constant module.

(Redmine issue: #3734)

Revision 50346ffa
Added by Jan Mach over 1 year ago

Improvements related to account registration process using auth_env pluggable module for Hawat.

The account registration process needed more work to get it into stable and usefull state:
  • Changed validator for user login from email to string containing restricted set of characters.
  • Renamed field 'Login (eppn)' to just 'Login’. It made no sense to be used that was for example with Basic or certificate based authentication service.
  • Implemented sendind of confirmation emails after successfull registration to system admins and to the user that performed the registration. The emails are generated using Jinja2 templates.
  • Implemented sending of confirmation emails after successfull account activation to the user and as BCC to system admins.
  • Added missing rollback() calls to base View implementations to resolve SQLAlchemy errors after unsuccessfull database operation.

(Redmine issue: #3734)

Revision 62bbed91
Added by Jan Mach over 1 year ago

Improved web interface translations to Czech language.

(Redmine issue: #3734)

Revision b57c5e78
Added by Jan Mach over 1 year ago

Translated informational emails, that are sent to the user and admins after account registration and activation.

  • Registration email sent to the user is localized into the locale selected by the user during registration.
  • Registration email sent to the admins is localized into the default application locale.
  • Activation email sent to the user is localized into the user`s locale.

(Redmine issue: #3734)

Revision 6c3c2960
Added by Jan Mach over 1 year ago

Implemented new base view CreateForView.

This new base view anables implementing view for maintainer with restricted permissions. Using these views particular user can be given permission to create child items for particular parent item. For example so far only the administrator was able to create new networks of filters. (Redmine issue: #3734)

Revision dd4f372e
Added by Jan Mach over 1 year ago

Implemented new views for creating new networks and filters for particular group.

Based on previous commit new feature was implemented, that will enable group maintainers to add new filters and networks to that particular group (so far administration privileges were necessary). (Redmine issue: #3734)

Revision 3929dffc
Added by Jan Mach over 1 year ago

Improved coding style and content of flash messages of Hawat object management modules.

(Redmine issue: #3734,#3443)

Revision e81272f3
Added by Jan Mach over 1 year ago

Template design improvements for better presentation of user/group/settings detail page.

(Redmine issue: #3734)

Revision e1a6ac88
Added by Jan Mach over 1 year ago

Updated Czech translations of Hawat web interface.

(Redmine issue: #3734)

Revision 97edfedd
Added by Jan Mach over 1 year ago

Huge overhaul of Hawat`s Flask-based framework and all it`s modules.

This commit introduces huge amount of work that went into revision of existing Flask-based framework, on top of which the web interface and all of it`s modules are implemented. The process of registering modules and views into the application was greatly simplified (this will be used for further improvements of application menu). Code quality was improved using Pylint and Pyflakes tools. Documentation was improved. Many code features were refactored for performance, optimization and code style reasons. (Redmine issue: #3734,#3443)

Revision ec97c654
Added by Jan Mach over 1 year ago

Implemented automated hiding of menu entries based on permissions of current user.

(Redmine issue: #3734)

Revision 62bf1927
Added by Jan Mach over 1 year ago

Small tweaks in main menu entry organization.

(Redmine issue: #3734)

Revision d84b91b0
Added by Jan Mach over 1 year ago

Implemented support for item action menu into application menu code.

The application menu in hawat.menu module was improved to also support item action menus. These menus are currently on item detail page and contain various contextual item actions like editing or deletion. (Redmine issue: #3734)

Revision c9815bf7
Added by Jan Mach over 1 year ago

Implemented views for enabling/disabling reporting filters.

(Redmine issue: #3734)

Revision 74b8ccb2
Added by Jan Mach over 1 year ago

Fix: Fixed switched templates for item enabling and disabling actions.

(Redmine issue: #3734)

Revision ffddbaae
Added by Jan Mach over 1 year ago

Localized HTTP error pages.

(Redmine issue: #3734)

Revision ea8c3233
Added by Jan Mach over 1 year ago

Fix: The actual SQL query displayed after event searching is now being displayed only to developers.

(Redmine issue: #3734)

Revision 02f537ad
Added by Jan Mach over 1 year ago

Improved Czech web interface translations.

(Redmine issue: #3734)

Revision fba27171
Added by Jan Mach over 1 year ago

Implemented prototype of reporting statistics dashboard.

Missing feature that needs to be implemented is filtering based on abuse groups. Currently any user can view dashboards for any abuse group reports. Aside from administrator user should be able to view dashboards only for groups he/she is member of. (Redmine issue: #3734)

Revision 1c76234f
Added by Jan Mach over 1 year ago

Fix: Fixed two minor bugs - invalid icon name and incorrect Jinja macro that failed in certain conditions.

(Redmine issue: #3734)

Revision 5aa8547f
Added by Jan Mach over 1 year ago

Improved variable dumping macros for Jinja2 templates.

(Redmine issue: #3734)

Revision c92afff2
Added by Jan Mach over 1 year ago

Fix: Fixed invalid icon name.

(Redmine issue: #3734)

Revision 2862ac98
Added by Jan Mach over 1 year ago

Fix: Used common Jinja macro for rendering alert messages wherever possible.

(Redmine issue: #3734)

Revision 10820032
Added by Jan Mach over 1 year ago

Disabled broken and unfinished Javascript feature.

(Redmine issue: #3734)

Revision 96acd6ad
Added by Jan Mach over 1 year ago

Enhanced event search form to accept comma separated list of addresses for source, target and host form fields.

Now it is possible to enter multiple values to each of the mentioned form fields and all values will be automatically ORed. (Redmine issue: #3734)

Revision 6c013278
Added by Jan Mach about 1 year ago

Added mechanism for propagating information from build system to web interface.

For now we want to be able to display information about current build version in the footer of the web page. (Redmine issue: #3734)

Revision 9dd18d64
Added by Jan Mach about 1 year ago

Minor improvements in web interface footer layout.

(Redmine issue: #3734)

Revision 22f63dbc
Added by Jan Mach about 1 year ago

Minor tweaks in Hawat`s top menu bar.

Removed unnecesary title. Added tooltip to locale switcher. (Redmine issue: #3734)

Revision b6c899a9
Added by Jan Mach about 1 year ago

Fix: Fixed invalid HTTP error code in error page.

(Redmine issue: #3734)

Revision 6e3b85db
Added by Jan Mach about 1 year ago

Fix: The IP4R PostgreSQL extension does not recognize certain network addressess.

User was presented with HTTP 500 error page without knowing what is actually wrong. Because the error is encountered at database level and the value passess form validation, it was necessary to analyze the exception thrown by database and present user with meaningfull message. For example 195.113.134.0/14 is not a valid network address from the point of view of IP4R extension, because there are more 1s than necessary and the extension does no adjusting on its own. This pathc solves only the error presentation to the user, actual solution stil has to be implemented. (Redmine issue: #3752,#3734)

Revision d9450b94
Added by Jan Mach about 1 year ago

Implemented access restrictions to reports and reporting dashboards.

Only administrator can access any reports and view reporting dashboards for all groups. Otherwise user may view reporting data only for groups he/she is member of. (Redmine issue: #3734)

Revision 210fd8ea
Added by Jan Mach about 1 year ago

Moved do_before_action hook to more appropriate place.

This hook should execute after each successfull form submission (except cancel action), otherwise it is not much usefull. (Redmine issue: #3734)

Revision d2a48f83
Added by Jan Mach about 1 year ago

Improved filter management module.

  • more information is displayed in filter list view
  • fixed invalid permissions for filter create view
  • fixed bug in filter creation and update: when basic filter was created, filtering rule was not generated from form data

(Redmine issue: #3734)

Revision a43a3a73
Added by Jan Mach about 1 year ago

Improved network management module.

  • more information is displayed in network list view
  • fixed invalid permissions for network create view

(Redmine issue: #3734)

Revision 8f010016
Added by Jan Mach about 1 year ago

Unified display of development data on various views.

Variable dumps for development pyurposes were unified among multiple views. (Redmine issue: #3734)

Revision 2f2d2965
Added by Jan Mach about 1 year ago

Implemented percentage displays in statistical tables.

(Redmine issue: #3734)

Revision e7c49335
Added by Jan Mach about 1 year ago

Huge improvements in reporter module and report views in Hawat.

  • Improved report model to enable parent reports, so we can link extra reports with parent summary reports.
  • Fixed/improved statistical calculations for reports.
  • Greatly improved report view visuals.
  • Greatly improved report search form.

This commit contains backwards compatibility breaking features including necesity do alter database schema. (Redmine issue: #3734)

Revision 9998a9fa
Added by Jan Mach about 1 year ago

Improved report show and search view visuals.

Added new data displays, widgets and tooltips. (Redmine issue: #3734)

Revision 913aaa78
Added by Jan Mach about 1 year ago

Minor fixes and improvements in report show and search view visuals.

(Redmine issue: #3734)

Revision 77520fc1
Added by Jan Mach about 1 year ago

More web interface UI and UX improvements.

This patch focuses more on improving look and feel when working with event search form and event detail view. The interlinking with other parts of the system was improved when working from event detail page. Event action search dropdown functionality was also improved. (Redmine issue: #3734)

Revision d03273a2
Added by Jan Mach about 1 year ago

More improvements in event search view.

Enriched event search form with option to search according to inspection errors. Added special search options for administrators. Additional minor tweaks and improvements. (Redmine issue: #3734)

Revision 5e60f140
Added by Jan Mach about 1 year ago

Tweaked event severity and classification display in event search and show views.

(Redmine issue: #3734)

Revision f9fbe943
Added by Jan Mach about 1 year ago

Activated Traget column for large monitor sizes in event search result table.

(Redmine issue: #3734)

Revision 637fc5a9
Added by Jan Mach about 1 year ago

Improved czech web interface translations.

(Redmine issue: #3734)

Revision 06705925
Added by Jan Mach about 1 year ago

Unified form submission buttons into button-toolbar widget.

(Redmine issue: #3734)

Revision 7d87b3c2
Added by Jan Mach about 1 year ago

Improvements in the dbstatus pluggable module for Hawat.

The module is now capable of displaying information about following tables within the events database: events, thresholds, events_thresholded. (Redmine issue: #3734)

Revision 98b116f4
Added by Jan Mach about 1 year ago

Improved result visualisation in the dbstatus pluggable module for Hawat.

Added charts and other graphical items to improve result visualisation. (Redmine issue: #3734)

Revision 4d74efa5
Added by Jan Mach about 1 year ago

Improved czech web interface translations.

(Redmine issue: #3734)

Revision 35be9d1b
Added by Jan Mach about 1 year ago

Moved everall event dashboard view from separate blueprint to events blueprint.

This way it will be similar to implementation of reports dashboard, which is also in reports blueprint. (Redmine issue: #3734)

Revision 375e719a
Added by Jan Mach about 1 year ago

Fix: Fixed link to incorrect dashboard view.

Overall event dashboards can not be calculated for particular group, link was meant to pount to reporting dashboard. (Redmine issue: #3734)

Revision a1c73786
Added by Jan Mach about 1 year ago

Removed unnecessary default value and unused reporting dashboard form.

The default value for searching reports is not necessary, because there is a result limit and also result is ordered in descending order, so the latest reports will always come on top. (Redmine issue: #3734)

Revision c3f00118
Added by Jan Mach about 1 year ago

Improved/fixed czech web interface and report translations.

(Redmine issue: #3734)

Revision c146013b
Added by Jan Mach about 1 year ago

Replaced 'unassigned’ and 'undisclosed’ string it event search result table in severity and group columns with icon (to make the table more compact).

(Redmine issue: #3734)

Revision e1f8b66b
Added by Jan Mach about 1 year ago

Huge improvements in Hawat web interface.

This patch introduces huge amount of work related to Hawat web interface. I was just clicking through the interface and trying to improve everything that popped out at the moment. There are following improvements:

  • Enabled DataTable localication plugins.
  • Implemented common layout for object management views (list views).
  • Redesigned application menu implementation.
  • Enhanced application menu to support contextual item action menus in listing tables. All administration list views now use it to generate contextual action toolbars.
  • Menu object is now used even to generate breadcrumb menus.
  • Simplified working with view icons and their propagation into action menus.
  • Implemented AJAX spinner and used AJAX to fetch raw report data to web interface. There is stil a lot of room for improvements, the idea is for the user to be able to inteligently browse through the loaded data in some way.
  • Redesigned object administration views (users, groups, filters, networks).
  • Cleaned up list of built-in icons.
  • Improved web interface translations after the changes in this commit.
  • Fixed the bug in serving report data files.

(Redmine issue: #3734,#3443)

Revision 30ddfeba
Added by Jan Mach about 1 year ago

Permission system redesign.

The maintainer permission is now used as slightly less powerfull administrator. Added permission for any role. It is now enforced wherever possible. Added prototype dump of all registered view permissions, however permission enforced by authorize_item_action callback are missing in the list. (Redmine issue: #3734)

Revision 23876ead
Added by Jan Mach about 1 year ago

Added new Jinja2 function for getting reference to view class.

This method can be usefull for accessing view metadata stored as class properties like icons, titles, etc. (Redmine issue: #3734)

Revision 1fad9d52
Added by Jan Mach about 1 year ago

Added ability for base view to check that current user can access given endpoint.

This can be used to dynamically hide inaccessible parts of web interface. This method also return false in case the endpoint does not exist within the application, so you may use it to hide links to another pluggable modules. (Redmine issue: #3734)

Revision 1156c848
Added by Jan Mach about 1 year ago

Fix: Improved _layout_list.html base template to be more resistant to nonexistence of data for breadcrumbs and action menu widgets.

(Redmine issue: #3734)

Revision 0e916daf
Added by Jan Mach about 1 year ago

Implemented new Hawat pluggable module: changelogs.

Hawat can now track item changes and display them through new changelogs pluggable module. This feature can be used for monitoring changes done on various items like user and group account settings. This patch contains complete work including enhancements of database model. (Redmine issue: #3734)

Revision e735fbdd
Added by Jan Mach about 1 year ago

Fix: Minor fixes and improvements.

(Redmine issue: #3734)

Revision f0ecff99
Added by Jan Mach about 1 year ago

Improved czech web interface translations.

(Redmine issue: #3734)

Revision 21851685
Added by Jan Mach about 1 year ago

Added logging of directly served files.

(Redmine issue: #3734)

Revision 7f4e035f
Added by Jan Mach about 1 year ago

Enhanced events search form to support search parameter negations.

Certain search parameters now support negations, so it is possible to for example negate category selection. (Redmine issue: #3734)

Revision 6850cd98
Added by Jan Mach about 1 year ago

Added new action button for searching related events from report detail view.

(Redmine issue: #3734)

Revision bf25879c
Added by Jan Mach about 1 year ago

Improved czech web interface translations.

(Redmine issue: #3734)

Revision 769ee7f7
Added by Jan Mach about 1 year ago

Fix: Fixed broken event dashboard view.

(Redmine issue: #3734)

History

#1 Updated by Jan Mach almost 2 years ago

  • Parent task set to #3374

#2 Updated by Jan Mach almost 2 years ago

  • % Done changed from 0 to 10

#3 Updated by Jan Mach over 1 year ago

  • Related to Task #3362: Migrate Mentat system to new hardware added

#4 Updated by Jan Mach about 1 year ago

  • Status changed from In Progress to Closed
  • Priority changed from Low to Normal

On tuesday 24.7. 2018 we have successfully migrated our production instance of Mentat system to version of Mentat system that is completely written in Python and uses PostgreSQL as database backend. Therefore this task can be considered complete.

Also available in: Atom PDF