Task #3374

Migrate all core modules from legacy Mentat

Added by Jan Mach almost 2 years ago. Updated 7 months ago.

Status:ClosedStart date:03/21/2017
Priority:HighDue date:
Assignee:Jan Mach% Done:

100%

Category:Development - Core
Target version:2.0

Description

Following is the list of core modules, that need to be migrated, redesigned and implemented into the next generation system.

Real time processing modules:
  • mentat-enricher.py - Message enrichment module
  • mentat-inspector.py - Message inspection module
  • mentat-sampler.py - Message sampling module
  • mentat-storage.py - Message persistent storage module
Post processing modules:
  • mentat-briefer.py - Periodical system performance briefing module
  • mentat-reporter-ng.py - Periodical problem reporting module
  • mentat-statistician.py - Periodical performance statistics module
Scripts and tools:
  • mentat-backup.py - System backup tool
  • mentat-cleanup.py - System cleanup tool
  • mentat-controller.py - Module control tool
  • mentat-dbmngr.py - Database management tool
  • mentat-netmngr.py - Internal whois network record management tool
  • mentat-ideagen.py - Message generation tool
  • mentat-precache.py - Data precaching tool
User interface:
  • mentat-hawat.py - Web user interface

Subtasks

Feature #3364: Implement mentat-storage.py moduleClosedJan Mach

Feature #3365: Implement mentat-enricher.py moduleClosedJan Mach

Feature #3375: Implement mentat-sampler.py moduleClosedJan Mach

Task #3376: Polish mentat-inspector.py moduleClosedJan Mach

Feature #3373: mentat-inspector.py - Provide default values for action a...ClosedJan Mach

Feature #3372: mentat-inspector.py - Implement 'overwrite' argument for ...ClosedPavel Kácha

Task #3393: Revision of pynspect libraryClosedJan Mach

Feature #3396: mentat-inspector.py - Implement 'unique' argument for 'ta...ClosedPavel Kácha

Feature #3377: Implement mentat-informant.py moduleClosedJan Mach

Feature #3378: Implement mentat-reporter.py moduleClosedJan Mach

Feature #3379: Implement mentat-statistician.py moduleClosedJan Mach

Task #3380: Polish mentat-backup.py moduleClosedJan Mach

Task #3381: Polish mentat-cleanup.py moduleClosedJan Mach

Task #3382: Polish mentat-controller.py moduleClosedJan Mach

Task #3383: Polish mentat-dbmngr.py moduleClosedJan Mach

Task #3384: Polish mentat-ideagen.py moduleClosedJan Mach

Feature #3385: Implement mentat-netmngr.py moduleClosedJan Mach

Feature #3386: Implement mentat-precache.py moduleClosedJan Mach

Task #3734: Migrate Hawat web user inteface from Perl-base to Python-...ClosedJan Mach


Related issues

Related to Mentat - Task #3362: Migrate Mentat system to new hardware Closed 03/21/2017

Associated revisions

Revision 97980d3f
Added by Jan Mach over 1 year ago

Bugfix: Filer daemon component logs unsuccessfull file operations.

If the filesystem operation was not succesfull, the exception was catched to enable recovery, but it was not recorded into logfile or anywhere else. (Redmine issues: #3374 and #1017)

Revision 8c6542f4
Added by Jan Mach over 1 year ago

Migration, refactoring and code style unification to support latest PyZenKit and Pynspect libraries.

This huge update represents a lot of work on many parts of the system. After major upgrade in PyZenKit and Pynspect libraries it was necessary to bring this project up to date. There were many changes in the code, lot of refactoring and while all this was done, code quality was improved, documentation written and unit tests modified. Many of these changes simply make sure, that the code works with latest versions of both libraries. This commit is supposed to be a single major feature upgrade, after this only smaller commits will follow to polish this huge amount of work.

(Redmine issues: #3374, #1017, #3361, #3387, #3443)

Revision d9d19017
Added by Jan Mach over 1 year ago

Unified implementation of Mentat module execution.

Code for each Mentat module now resides in valid Python module in mentat.module and all apropriate launcher scripts are very minmalistic. This is because names of executable files are not valid Python names for Python modules and this new approach will improve code testing and documentation processes. (Redmine issue: #3374, #3361, #3443)

Revision 2e8991cf
Added by Jan Mach over 1 year ago

Code style and documentation improvements in mentat.daemon.piper module.

Improved code style using pylint tool and adjusted related documentation for module mentat.daemon.piper, which is a base module for all real-time message processing daemons. (Redmine issue: #3374, #3361)

Revision d19b06b4
Added by Jan Mach over 1 year ago

Code style and documentation improvements in mentat.script.fetcher module.

Improved code style using pylint tool and adjusted related documentation for module mentat.script.fetcher, which is a base module for all message post-processing scripts. (Redmine issue: #3374, #3361)

Revision a1cd776b
Added by Jan Mach over 1 year ago

Added helper script for fetching CESNET`s Negistry whois database.

This patch also includes related cron script. (Redmine issue: #3374)

History

#1 Updated by Jan Mach almost 2 years ago

Current status.

Real time processing modules:

mentat-enricher.py TODO
mentat-inspector.py DONE (polishing needed)
mentat-sampler.py TODO
mentat-storage.py TODO

Post processing modules:

mentat-briefer.py TODO
mentat-reporter-ng.py TODO
mentat-statistician.py TODO

Scripts and tools:

mentat-backup.py DONE (polishing needed)
mentat-cleanup.py DONE (polishing needed)
mentat-controller.py DONE (polishing needed)
mentat-dbmngr.py DONE (polishing needed)
mentat-ideagen.py DONE (polishing needed)
mentat-netmngr.py TODO
mentat-precache.py TODO

#2 Updated by Jan Mach almost 2 years ago

  • Status changed from New to In Progress

#3 Updated by Jan Mach over 1 year ago

  • Description updated (diff)

#4 Updated by Jan Mach over 1 year ago

  • Description updated (diff)

Current status summary.

Real time processing modules:

mentat-enricher.py DONE (polishing needed)
mentat-inspector.py DONE (polishing needed)
mentat-sampler.py DONE (polishing needed)
mentat-storage.py DONE (polishing needed)

Post processing modules:

mentat-briefer.py TODO
mentat-reporter-ng.py TODO
mentat-statistician.py DONE (polishing needed)

Scripts and tools:

mentat-backup.py DONE (polishing needed)
mentat-cleanup.py DONE (polishing needed)
mentat-controller.py DONE (polishing needed)
mentat-dbmngr.py DONE (polishing needed)
mentat-netmngr.py DONE (polishing needed)
mentat-ideagen.py DONE (polishing needed)
mentat-precache.py DONE (polishing needed)

User interface:

mentat-hawat.py TODO

#5 Updated by Jan Mach 9 months ago

  • Related to Task #3362: Migrate Mentat system to new hardware added

#6 Updated by Jan Mach 7 months ago

  • Status changed from In Progress to Closed

This task can be considered complete. We are releasing version 2.0 of Mentat system soon and all modules are now reimplemented in Python.

Also available in: Atom PDF