Task #3361

Documentation

Added by Jan Mach about 2 years ago. Updated 10 months ago.

Status:In ProgressStart date:03/21/2017
Priority:LowDue date:
Assignee:Jan Mach% Done:

0%

Category:Documentation
Target version:Long running

Description

Long running task to record work progress

Continually develop and improve project documentation.

Associated revisions

Revision 9252cd11
Added by Jan Mach about 2 years ago

Improved documentation and code readability in jpath.py library.

(Redmine issues: #3392 and #3361)

Revision 6803d78a
Added by Jan Mach about 2 years ago

Ensured allowed character set for node names in JPath.

Explicit list of allowed characters instead of "\w”, including unit tests and documentation. (Redmine issues: #3392, #3361 and #1017)

Revision a8355b5a
Added by Jan Mach about 2 years ago

Documentation improvements.

Tried to make the section about node delimiters and indices more clear. (Redmine issues: #3392 and #3361)

Revision 6ef84064
Added by Jan Mach about 2 years ago

Documentation improvements for mentat.filtering.jpath module.

Iproved section about node delimiters and default dict handling behavior. Made sure the documentation displays nicelly in pydoc3. Improved formating for Sphinx-doc documentation builder. (Redmine issues: #3392 and #3361)

Revision bf0598d3
Added by Jan Mach about 2 years ago

Implemented support for conditional value overwriting and uniqueness.

The jpath_set() method now optionally enables user to specify, whether already existing value should be overwritten or not, and whether value should be unique or not. The uniqueness option works only for lists at the end of JPath. (Redmine issues: #3392, #3361 and #1017, enables #3372 and #3396)

Revision ce01f6d7
Added by Jan Mach about 2 years ago

Huge revission of mentat.filtering library.

This commit represents huge revision of mentat.filtering library. Codebase has been checked, unit tests have been improved and documentation has been written to make order in current state of things. Based on this filtering language improvements will be designed. (Redmine issues: #3393, #1017 and #3361)

Revision acb3fbc7
Added by Jan Mach about 2 years ago

Implemented automatic documentation generation using Sphinx-doc.

Project documentation will now be (partly) autogenerated out of the module source code. The environment was prepared and first modules were added into documentation. Guide to generate the documentation locally was added to Redmine project wiki page (Development cheat sheet), so that the documentation is available to developers before it will be published on-line. (Redmine issue: #3361)

Revision 4164e95a
Added by Jan Mach about 2 years ago

Improvements in mentat.idea.internal library.

Greatly improved code reusability by employing typedef generator approach like in the underlying typedcol and idea.lite library. Additionally the documentation and unit tests were both improved, documentation can now be generated using Sphinx-doc tool. (Redmine issues: #3364, #3361 and #1017)

Revision daafd215
Added by Jan Mach about 2 years ago

Implemented module for converting IDEA messages to and from MongoDB.

There are classess in mentat.idea.mongodb module for easy conversion of IDEA messages to and from MongoDB representation. Appropriate documentation and unit tests were also created. (Redmine issue: #3364, #3361 and #1017)

NOTE: There is probably bug in current Perl-based library for message conversion, because all unit test conversions related to timestamps from database messages stored by legacy code fail. There is always 1 or 2 hour time difference. This issue is not yet fixed.

Revision a48fda96
Added by Jan Mach about 2 years ago

Updated root document for library documentation.

(Redmine issue: #3361)

Revision 70144b0c
Added by Jan Mach about 2 years ago

Updated license files and license headers at the beginning of each file.

(Redmine issue: #3361)

Revision 25b51380
Added by Jan Mach about 2 years ago

Finished prototype of mentat-storage.py module.

This commit introduces finished working prototype of mentat-storage.py real-time message processing module including appropriate unit tests and basic documentation work. Key features are possible customization of target database and collection, usage of core database configuration file, which can be overriden with local config file, or command line options. Messages are currently stored in database one by one, however batch processing will possibly be implemented in the future.

(Redmine issues: #3364, #1017 and #3361)

Revision 06b99623
Added by Jan Mach about 2 years ago

Documentation refactoring for pynspect library.

After refactoring of mentat.filteing library to pynspect the documentation needed update as well. (Redmine issues: #3393 and #3361)

Revision 2877bc30
Added by Jan Mach about 2 years ago

Added empty placeholder files to include empty directories into git.

This fixed the issue with Sphinx complaining about missing expected directories during build process. (Redmine issues: #3361 and #3387)

Revision 221f6ee2
Added by Jan Mach about 2 years ago

Enabled printing of last modification time in autogenerated documentation.

(Redmine issue: #3361)

Revision e97ad924
Added by Jan Mach about 2 years ago

Added build metadata to generated documentation.

Generated documentation now contains information about distribution and revision, from which it was generated. This information is presented on title page a buildbot system provides correct values during build process. This was user can clearly see, which documentation version for which distribution (production, development) he/she is currently browsing. (Redmine issues: #3361 and #3387)

Revision ba27b312
Added by Jan Mach almost 2 years ago

Propagated information about package version to documentation.

Version number gets propagated into autogenerated documentation. Buildbot system is responsible for generating appropriate metadata file. (Redmine issues: #3361 and #3387)

Revision 5a783df0
Added by Jan Mach almost 2 years ago

Propagating build number to Grunt and documentation.

Build number from Buildbot system now gets propagated to Grunt and autogenerated documentation. This feature is intended to be used for development (unstable) builds to somehow autoincrement the package version number. For production distributions this will be skipped and developer still has to set the desired version number manually. (Redmine issues: #3361 and #3387)

Revision aaaaf741
Added by Jan Mach almost 2 years ago

Documentation improvements.

(Redmine issue: #3361)

Revision 5f761799
Added by Jan Mach almost 2 years ago

Overhaul of IDEA statistic calculation library and RRD manipulation library.

Improved code, documentation and unit tests in IDEA and RRD libraries related to mentat-statistician.py. (Redmine issue: #3379, #1017, #3361)

Revision d15d3164
Added by Jan Mach almost 2 years ago

Refactored some documentation files and included documentation for submodules.

Some documentation source files had to be renamed to enable better integration with automated buildbot system. Documentation for submodules was included into overall project documentation. (Redmine issue: #3361)

Revision 8c6542f4
Added by Jan Mach almost 2 years ago

Migration, refactoring and code style unification to support latest PyZenKit and Pynspect libraries.

This huge update represents a lot of work on many parts of the system. After major upgrade in PyZenKit and Pynspect libraries it was necessary to bring this project up to date. There were many changes in the code, lot of refactoring and while all this was done, code quality was improved, documentation written and unit tests modified. Many of these changes simply make sure, that the code works with latest versions of both libraries. This commit is supposed to be a single major feature upgrade, after this only smaller commits will follow to polish this huge amount of work.

(Redmine issues: #3374, #1017, #3361, #3387, #3443)

Revision 03d9cc1a
Added by Jan Mach almost 2 years ago

Unified documentation header of all module executables.

(Redmine issue: #3361)

Revision d9d19017
Added by Jan Mach almost 2 years ago

Unified implementation of Mentat module execution.

Code for each Mentat module now resides in valid Python module in mentat.module and all apropriate launcher scripts are very minmalistic. This is because names of executable files are not valid Python names for Python modules and this new approach will improve code testing and documentation processes. (Redmine issue: #3374, #3361, #3443)

Revision 28e09f9d
Added by Jan Mach almost 2 years ago

Improved documentation of Mentat system modules (scripts and daemons).

(Redmine issue: #3361)

Revision 2e8991cf
Added by Jan Mach almost 2 years ago

Code style and documentation improvements in mentat.daemon.piper module.

Improved code style using pylint tool and adjusted related documentation for module mentat.daemon.piper, which is a base module for all real-time message processing daemons. (Redmine issue: #3374, #3361)

Revision 141781af
Added by Jan Mach almost 2 years ago

Added missing documentation pages for mentat.daemon.component... modules.

(Redmine issue: #3361)

Revision 778b5217
Added by Jan Mach almost 2 years ago

Added missing documentation files and unified contents of documentation files for mentat.plugin.enricher... modules.

(Redmine issue: #3361)

Revision 8a770462
Added by Jan Mach almost 2 years ago

Unified documentation source files for mentat.datatype... and mentat.idea... modules.

(Redmine issue: #3361)

Revision 75b38c4b
Added by Jan Mach almost 2 years ago

Added missing documentation source files for toplevel mentat utilities.

(Redmine issue: #3361)

Revision d19b06b4
Added by Jan Mach almost 2 years ago

Code style and documentation improvements in mentat.script.fetcher module.

Improved code style using pylint tool and adjusted related documentation for module mentat.script.fetcher, which is a base module for all message post-processing scripts. (Redmine issue: #3374, #3361)

Revision 2fd92909
Added by Jan Mach almost 2 years ago

Added all existing documentation pages to master documentation.

(Redmine issue: #3361)

Revision a38a94d7
Added by Jan Mach almost 2 years ago

Fixed the condition for displaying warning about documentation being for development code.

(Redmine issue: #3361)

Revision 249514d3
Added by Jan Mach almost 2 years ago

Changed documentation section headings for API documentation and list of system modules.

(Redmine issue: #3361)

Revision 5ae20567
Added by Jan Mach almost 2 years ago

Added documentation page about installation.

(Redmine issue: #3361)

Revision 48081ff3
Added by Jan Mach almost 2 years ago

Updated documentation for mentat-statistician.py module.

(Redmine issue: #3379,#3361)

Revision b1395452
Added by Jan Mach almost 2 years ago

Moved toplist calculation of message statistics into internal helper method.

(Redmine issue: #3379,#3361)

Revision b31f8e8b
Added by Jan Mach almost 2 years ago

Added toplist restriction to ASNs and Countries in calculated message statistics.

(Redmine issue: #3379,#3361)

Revision 08da1e14
Added by Jan Mach almost 2 years ago

Added missing documentation files for fetcher scripts.

(Redmine issue: #3361)

Revision 3350c701
Added by Jan Mach almost 2 years ago

Added documentation page about overall system architecture.

(Redmine issue: #3361)

Revision 57e4e1bd
Added by Jan Mach almost 2 years ago

Checked, updated and unified coding style of mentat.daemon.component... Python modules.

Used pylint tool to check for violations of coding style. Minor documentation improvements. (Redmine issue: #3361,#3443)

Revision b7b1578e
Added by Jan Mach over 1 year ago

Configuration for Sphinx-doc will now fetch current version directly from Mentat package.

This means that there is one less place to manually increment the version number when deploying new package. There still remains the issue with making the same thing for Grunt configuration. (Redmine issue: #3361,#3387)

Revision 0a724ce9
Added by Jan Mach over 1 year ago

Feature: Prototype implementation of next generation web user interface for Mentat system.

This commit represents huge amount of work and introduces new Python-based version of Hawat web user interface for Mentat system. It is built on top of the great Flask framework. Currently it is still work in progress and many features are implemented only as dummy stubs, however the core functionality is almost finished. Following is a list of current key features:

  • Focus on modular architecture using Flask`s blueprint feature.
  • Multi-layered configurability (defaults, multiple file overrides).
  • Plugin system.
  • Internationalization support.
  • Support for multiple authentication systems.

This commit provides working albeit hollow starting point for future web interface development. IT IS NOT READY FOR PRODUCTION USAGE!!!

(Redmine issue: #3734,#3361)

Revision d75c1c78
Added by Jan Mach over 1 year ago

Improved the mentat.storage library.

  • Rewritten the StorageManager, so it now expects full Mentat core configuration tree.
  • Added couple of shortcut and helper methods.
  • Added database_info methods.
  • Increased documentation coverage.

(Redmine issue: #3443,#3361)

Revision 6931b441
Added by Jan Mach over 1 year ago

Improved geoip, whois and storage services to fully use Mentat core configurations.

The mentat.geoip, mentat.whois and mentat.storage services were enhanced to fully understand and use Mentat core configurations. The main idea is that the configuration of these services should be centralized for the whole Mentat ecosystem and any module or component, that wishes to use these services should be able to easily obtain correctly configured instance and without the knowledge of the configuration directory schema and syntax. This patch introduces following new concepts:

  • Each service has corresponding service manager that is capable of understanding Mentat core configuration tree schema and syntax and quickly provide fully configured service instance.
  • Each service has ability to store reference to fully configured service instance at module level (like a singleton pattern) and any code can get a hold of this reference.

This patch also contains improvements in documentation of all above mentioned modules and also many improvements in unit testing framework, which was used to verify correct functionality. (Redmine issue: #1017,#3361)

Revision c6d94a1c
Added by Jan Mach over 1 year ago

Minor documentation improvements.

(Redmine issue: #3361)

Revision b7ffa45a
Added by Jan Mach over 1 year ago

Improved documentation of hawat.config module.

(Redmine issue: #3734,#3361)

Revision c65c3126
Added by Jan Mach over 1 year ago

Revision of the item CRUDL framework in Hawat.

The item manipulation framework (create,read,update,delete,list) was revised, simplified and better documented. All changes were implemented and tested on existing user management pluggable module. (Redmine issue: #3734,#3361)

Revision a25373b6
Added by Jan Mach over 1 year ago

Documentation improvements.

(Redmine issue: #3361,#3734)

Revision e8d8be42
Added by Jan Mach over 1 year ago

Fixed documentation after whois and geoip services name refactoring.

(Redmine issue: #3361)

Revision a10b92e8
Added by Jan Mach over 1 year ago

Improved documentation of mentat.daemon.piper module.

(Redmine issue: #3361)

Revision 4cbaffbb
Added by Jan Mach over 1 year ago

Improved documentation regarding Mentat installation an quick start tutorial.

(Redmine issue: #3361)

Revision da5e5f65
Added by Jan Mach over 1 year ago

Documentation improvements.

Enhanced section about installation, added new section about migration (based on experience from installation on new server). (Redmine issue: #3361)

Revision 355d6a88
Added by Jan Mach over 1 year ago

Added missing documentation pages for Hawat pluggable modules.

(Redmine issue: #3361)

Revision df8b6802
Added by Jan Mach about 1 year ago

Improved event reporting library.

  • Updated documentation of the library classes and its methods.
  • Fixed minor issues reported by pylint utility.
  • Improved unit test coverage of the whole library.

(Redmine issue: #3361,#1017,#3443)

Revision 5bbd7c70
Added by Jan Mach about 1 year ago

Added missing API documentation pages.

(Redmine issue: #3361)

Revision 8fa5c2d5
Added by Jan Mach 12 months ago

Improved documentation regarding event reporting.

(Redmine issue: #3361)

Revision b1d367dd
Added by Jan Mach 12 months ago

Improved documentation of mentat-reporter.py module.

(Redmine issue: #3378,#3361)

Revision 870d557f
Added by Jan Mach 12 months ago

Implemented example Apache configuration file for Basic authentication.

There are two Apache example configurations now, first is an example for Shibboleth SSO authentication, second is an example for Basic authentication. Using these two examples (especially the latter) it is possible to very quickly set up demonstration installation. The installation page was updated accordingly to reflect new installation options. (Redmine issue: #3387,#3361)

Revision 35425890
Added by Jan Mach 11 months ago

Added database schema and tuning documentation to Sphinx based documentation.

(Redmine issue: #3361)

Revision 5274bb67
Added by Jan Mach 11 months ago

Further documentation style improvents in database documentation page.

(Redmine issue: #3361)

Revision 6b0bee1d
Added by Jan Mach 11 months ago

Improved documentation and coding style of mentat-controller.py module.

(Redmine issue: #3361,#3443)

Revision 5d12a1d2
Added by Jan Mach 11 months ago

Improved Mentat installation documentation page.

There is a lots of hidden work here in this commit. To make the installation process even more simple, we have prepared three Ansible roles available via official Ansible Galaxy site, that can very quickly install and configure Mentat system. (Redmine issue: #3361,#3387)

Revision eabe7add
Added by Jan Mach 10 months ago

Updated migration documentation page.

(Redmine issue: #3361)

Revision 1357fe96
Added by Jan Mach 10 months ago

More polishing of migration documentation page.

(Redmine issue: #3361)

Revision 9334d8b0
Added by Jan Mach 10 months ago

Improved quickstart documentation page.

Added introduction section about user interface including some images. (Redmine issue: #3361)

Revision 000f743a
Added by Jan Mach 10 months ago

Improved reporting documentation page.

(Redmine issue: #3378,#3361)

Revision aba628b4
Added by Jan Mach 10 months ago

Improved migration documentation page.

(Redmine issue: #3361)

Revision 5cffdf66
Added by Radko Krkoš 10 months ago

Updated database schema and tuning documentation.

(Redmine issue: #3361)

Revision 551ef727
Added by Jan Mach 10 months ago

Fix: Fixed documentation generation after updating the PyZenKit library.

(Redmine issue: #3361)

Revision 0082758e
Added by Jan Mach 10 months ago

Further improvements in migration documentation page.

(Redmine issue: #3361)

Revision de96a634
Added by Jan Mach 10 months ago

Fix: Replaced example cronjob definition with correct value.

(Redmine issue: #3361)

Revision e1e9a07f
Added by Jan Mach 10 months ago

Improved documentation related to installation, migration and quickstart overview based on feedback from users installing the system.

(Redmine issue: #3361)

Revision a95cb66a
Added by Jan Mach 10 months ago

Added section about cleanup into migration documentation page plus few additional minor improvements.

(Redmine issue: #3361)

Revision d7dd51c4
Added by Jan Mach 10 months ago

Improved documentation based on the feedback from CTI project migrating their Mentat instance.

(Redmine issue: #3361)

Revision cd544538
Added by Jan Mach 10 months ago

Updated installation documentation using Ansible to match new role names.

(Redmine issue: #3361,#3387)

Revision efecaaec
Added by Jan Mach 9 months ago

Improved migration documentation page, section about post-migration cleanup.

Added information about how to get rid of old Perl-based Mentat system and other now unnecessary files and directories. (Redmine issue: #3361)

Revision 0b069aa0
Added by Jan Mach 9 months ago

Big documenation improvements.

Improved documentation with focus on possible developers, so the Overview and Development section were greatly improved. (Redmine issue: #3361)

Revision 13501d7e
Added by Jan Mach 9 months ago

Replaced old CESNET logo with current version.

(Redmine issue: #3361)

Revision bd653d69
Added by Jan Mach 9 months ago

Created presentation - Mentat: Introduction for developers.

This presentation can be used to introduce new developers into the Mentat system development. (Redmine issue: #3361)

Revision 38b8634d
Added by Jan Mach 9 months ago

Fixed SVG and PDF versions of Mentat logo.

Removed unnecessary whitespace around the logo. (Redmine issue: #3361)

Revision 8c4727c6
Added by Jan Mach 8 months ago

Minor improvements in architecture documentation page.

(Redmine issue: #3361)

Revision f7d5114b
Added by Jan Mach 8 months ago

Improvements in coding style, documentation and unit tests of mentat-enricher.py module.

(Redmine issue: #4213,#1017,#3361,#3443)

Revision 9e970a1c
Added by Jan Mach 8 months ago

Added documentation section about system administration.

Currently focus of this section is Mentat system health monitoring. (Redmine issue: #4222,#3361)

Revision 1414acf2
Added by Jan Mach 8 months ago

Improved documentation index page.

  • Added section for useful links and references.
  • Created substitution to create direct link to Git revision page in our project support system.
  • Created substitution for easy generation of documentation links.

(Redmine issue: #3361)

Revision 25a83071
Added by Jan Mach 8 months ago

Fix: Fixed minor bugs in documentation syntax.

The Sphinx-doc reported various minor bugs when generating the documentation. These should be fixed now. (Redmine issue: #3361)

Revision 1225777a
Added by Jan Mach 8 months ago

Improved documentation page for developers.

Moved some content from website to autogenerated documentation to prevent from repeating the same content on multiple places. Added information about versioning and tagging. (Redmine issue: #3361)

Revision 1893ca2c
Added by Jan Mach 8 months ago

Revised presentation - Mentat developer intro.

  • Added more links to external resources.
  • Minor content improvements.

(Redmine issue: #3361)

Revision ffb814da
Added by Jan Mach 8 months ago

Created presentation - Mentat Intro.

This presentation contains introductory information for Mentat system, latest news and description of current state. (Redmine issue: #3361)

Revision babb6756
Added by Jan Mach 7 months ago

Fix: Fixed invalid link to Ansible role on Ansible Galaxy service.

(Redmine issue: #3361)

Revision 546a325d
Added by Jan Mach 7 months ago

Added information about changing the default SECRET_KEY configuration for Hawat.

(Redmine issue: #3361)

Revision a9623536
Added by Jan Mach 7 months ago

Fix: Fixed the Hawat base documentation.

(Redmine issue: #3361)

Revision 00039548
Added by Jan Mach 6 months ago

Added section header to upgrade documentation page.

(Redmine issue: #3361)

Revision ebcbabcc
Added by Jan Mach 5 months ago

Improved documentation and coding style of mentat.stats.idea module.

(Redmine issue: #3361,#3443)

Revision 89be0e95
Added by Jan Mach 5 months ago

Changed PostgreSQL version number to 11 on database documentation page.

(Redmine issue: #3361)

Revision 5db7e577
Added by Jan Mach 4 months ago

Added section about using RAM filesystem for message exchange queues.

(Redmine issue: #3361,#4570)

Revision 883adabd
Added by Jan Mach 4 months ago

Minor documentation improvements.

Added few remarks pointed out by Radko Krkoš to the documentation section about RAM based filesystem. (Redmine issue: #3361)

Revision 17e371bb
Added by Jan Mach 4 months ago

Very minor documentation improvement (phrasing change).

(Redmine issue: #3361)

Revision 4a7aa0f0
Added by Jan Mach 4 months ago

Updated documentation page about upgrading Mentat.

The recipe for upgrading Mentat system for updated during the current Mentat system upgrade. (Redmine issue: #3361)

Revision d1672c44
Added by Jan Mach 3 months ago

Make the whole project more developer friendly.

Following big improvements were made to make the development more easier:

  • Created the 'setup.py’ script to support native Python packaging with setuptools and pip.
  • There is a new target 'make develop’, that will bootstrap the Python virtual environment, install all production and development requirements and install the whole project in editable mode.
  • Revised the dependency installation targets.
  • Big facelift of makefile help target.
  • Added target for build environment cleanup.
  • Added target for initialization of new translations of Mentat and Hawat.
  • Added target for building locally Python packages.
  • Added target for building locally Debian packages.
  • Updated documentation page for developers.

(Redmine issue: #4216,#3387,#3361)

Revision 8ee65f54
Added by Jan Mach 3 months ago

Fixes and improvements in master makefile.

  • Merged pylint and pyflakes tasks for simplicity,
  • Fixed broken clean task.
  • Updated development documentation to include Grunt as prerequisite.
  • Few additional minor tweaks.

(Redmine issue: #4216,#3387,#3361)

Revision 22d814dd
Added by Jan Mach 3 months ago

Removed manually created API documentation files and used sphinx-apidoc instead.

Previously all API documentation pages were manually created, which resulted is errors. (Redmine issue: #4216,#3361,#3387)

Revision 8ea53be9
Added by Jan Mach 3 months ago

Huge revision of master makefile and development documentation page.

Documentation, build procedures...almost everything was revised and rethought. (Redmine issue: #4216,#3387,#3361)

Revision 30ec34cf
Added by Jan Mach 3 months ago

Minor code documentation improvements.

(Redmine issue: #3361)

Revision bcebd8fb
Added by Jan Mach 3 months ago

Updated PyZenKit to latest version 0.54.

This version comes with improved application command line parameter documentation, some changes were necessary to make everything work again. (Redmine issue: #3387,#3361)

Revision 4bb6ba95
Added by Jan Mach 25 days ago

Updated development documentation page.

Removed unnecessary duplicit section about installation. (Redmine issue: #3361)

Revision e61f36f5
Added by Jan Mach 14 days ago

Fix: Added the missing event migrations to Debian package.

(Redmine issue: #3361,#3387,#4216)

Revision 363471bb
Added by Jan Mach 6 days ago

Added curl usage examples for API access to Hawat.

(Redmine issue: #3361)

History

#1 Updated by Jan Mach about 2 years ago

  • Subject changed from Dokumentace to Documentation
  • Description updated (diff)
  • Status changed from New to In Progress

#2 Updated by Jan Mach about 2 years ago

Before tomorrows video-conference with Protective team members, I have created wiki page about installation, that could help them see thing more clearly. It is not accessible from menu or other pages, so it is currently hidden from random users. Additionally, I have updated the demo description wiki page according to the email regarding the meeting agenda.

#3 Updated by Jan Mach 10 months ago

  • Priority changed from Normal to Low

Also available in: Atom PDF