|Status:||In Progress||Start date:||05/31/2013|
|Assignee:||Jan Mach||% Done:|
|Category:||Development - Core|
|Target version:||Long running|
Long running task to record work progress
Continually develop and improve test code to ensure quality and stability of the project. Ideally every class and module should be covered with unit tests or other type of test code.
Bugfix: JPath library was not working with IDEA messages.
The JPath library did not work with IDEA messages, there are no dicts and lists. Isinstance checks had to be updated accordingly. Added assertions for working with IDEA messages into unit test case to prevent this from happening in the future. (Redmine issues: #3392 and #1017)
Implemented support for conditional value overwriting and uniqueness.
The jpath_set() method now optionally enables user to specify, whether already existing value should be overwritten or not, and whether value should be unique or not. The uniqueness option works only for lists at the end of JPath. (Redmine issues: #3392, #3361 and #1017, enables #3372 and #3396)
Implemented support for default argument values in inspection actions.
When not explicitly set and appropriate, default value will be used as and inspection action agument. Updated log line styles to match. Updated demo and default configuration files to provide clues of new options. Updated unit tests to verify new functionality (visually, currently without assertions. (Redmine issues: #3373 and #1017)
Huge revission of mentat.filtering library.
This commit represents huge revision of mentat.filtering library. Codebase has been checked, unit tests have been improved and documentation has been written to make order in current state of things. Based on this filtering language improvements will be designed. (Redmine issues: #3393, #1017 and #3361)
Implemented internal IDEA message representation class.
The default idea.lite library for representing IDEA messages does not take into account custom subkeys, that can appear in messages handled by Mentat. This patch introduces prototype of new library, which is based on idea.lite and takes custom keys into account. (Redmine issues: #3364 and #1017)
Improvements in mentat.idea.internal library.
Greatly improved code reusability by employing typedef generator approach like in the underlying typedcol and idea.lite library. Additionally the documentation and unit tests were both improved, documentation can now be generated using Sphinx-doc tool. (Redmine issues: #3364, #3361 and #1017)
Implemented module for converting IDEA messages to and from MongoDB.
There are classess in mentat.idea.mongodb module for easy conversion of IDEA messages to and from MongoDB representation. Appropriate documentation and unit tests were also created. (Redmine issue: #3364, #3361 and #1017)
NOTE: There is probably bug in current Perl-based library for message conversion, because all unit test conversions related to timestamps from database messages stored by legacy code fail. There is always 1 or 2 hour time difference. This issue is not yet fixed.
Performance optimization of mentat.filtering.jpath.jpath_parse() function.
Implemented jpath_parse_c() as a caching variant of jpath_parse() function. Benchmarking confirmed huge performance improvement, see mentat.filtering.benchmark.bench_jpath.py for details. Another small improvement was archived by moving compilation of chunk regular expression to global module variable (14% performance increase). All module functions now use this caching variant internally. Also implemented cache_size() and cache_clear() functions for cache management. Beware, that jpath_parse_c() function does not make deep copy of returned value from cache (for performance reasons). Treat those values as read only, or suffer the consequences. (Redmine issues: #3392 and #1017)
Fixed the problem with bson.BSON.encode being unable to encode typedcol.TypedList objects.
In commit 9865c900af39a98c2c5254b06ed4f132dded6929 the mentat.idea.mongodb was unable to store IDEA messages into MongoDB. The issue was with bson.BSON encoder, which was hardcoded in a way that handled any unknown object as dict. We were not able to convince the encoder to treat TypedList objects as lists, so we had to use different approach and supply appropriate data structure. The mentat.idea.mongodb.IdeaIn convertor now produces data structure composed of simple dicts and lists instead of TypedDicts and TypedLists.
Current implementation should however be considered as prototype and proof of concept, because it probably will be possible to write it in more elegant way. The current problem is, that the idea.base.idea_typedef contains hardcoded calls for typedcol.typed_list(), which are not customizable from outside of the module via flavour mechanism. The addon feature was used to monkeypatch these definitions. This is of course not optimal solution, because any changes in underlying library must be propagated manually into mongodb library.
Additionally, IDEA messages stored in database contain some additional attributes, that are database specific and internal and should be stripped upor retrieving from database. Currently this must be done manually using truncate() function call, more optimal solution would be to incorporate this into typedcol library and strip these attributes during object instantination/conversion process.
Finished prototype of mentat-storage.py module.
This commit introduces finished working prototype of mentat-storage.py real-time message processing module including appropriate unit tests and basic documentation work. Key features are possible customization of target database and collection, usage of core database configuration file, which can be overriden with local config file, or command line options. Messages are currently stored in database one by one, however batch processing will possibly be implemented in the future.
Improved unit tests for mentat-inspector module.
Implemented test suite for better testing daemon components. (Redmine issue: #1017)
Changed default configuration for mentat-storage.py module.
The mentat-storage.py module will most likelly be the last module in processing chain, so it should delete the messages by default, othewise the whole procesing chain will hang. Additionally, there was a small fix that needed to be done in module unit test file. (Redmine issue: #3364 and #1017)
Fixed all unit tests to working state.
All currently implemented unit tests are working, even if only in most basic manner. This enables us to implement automated testing into our build system. (Redmine issues: #1017 and 3387)
Fix: Fixed the unit test for geoip service.
Test script attempts to obtain geoip databases before execution of test code, so that it is runnable anywhere and database files do not have to be part of the distribution. (Redmine issue: #1017)
Fix: Changed call to subprocess.call() method to support older versions of Python3.
(Redmine issue: #1017)
Migration, refactoring and code style unification to support latest PyZenKit and Pynspect libraries.
This huge update represents a lot of work on many parts of the system. After major upgrade in PyZenKit and Pynspect libraries it was necessary to bring this project up to date. There were many changes in the code, lot of refactoring and while all this was done, code quality was improved, documentation written and unit tests modified. Many of these changes simply make sure, that the code works with latest versions of both libraries. This commit is supposed to be a single major feature upgrade, after this only smaller commits will follow to polish this huge amount of work.
Fix: Fixed unit tests.
All unit tests now pass, even though they still need a lot of work and in some case do not do much of a testing. But at least the testing feature can be now enabled on the automated build system. As the unit tests will get improved in time, the deployment process will get more and more reliable. (Redmine issue: #1017,#3387)
Fix: Fixed broken unit test.
Unit test for mentat.script.fetcher was broken on machines, where mentat is not installed and /etc/metat/core directory does not exist. (Redmine issue: #1017)
Improved geoip, whois and storage services to fully use Mentat core configurations.
The mentat.geoip, mentat.whois and mentat.storage services were enhanced to fully understand and use Mentat core configurations. The main idea is that the configuration of these services should be centralized for the whole Mentat ecosystem and any module or component, that wishes to use these services should be able to easily obtain correctly configured instance and without the knowledge of the configuration directory schema and syntax. This patch introduces following new concepts:
- Each service has corresponding service manager that is capable of understanding Mentat core configuration tree schema and syntax and quickly provide fully configured service instance.
- Each service has ability to store reference to fully configured service instance at module level (like a singleton pattern) and any code can get a hold of this reference.
This patch also contains improvements in documentation of all above mentioned modules and also many improvements in unit testing framework, which was used to verify correct functionality. (Redmine issue: #1017,#3361)
Fix: Fixed missing PostgreSQL database initialization in fetcher unit test.
The unit test for fetcher script was missing the SQL schema creation and PostgreSQL extension registration. (Redmine issue: #1017)
Fix: Disabled build breaking tests.
Test in mentat.system module need more thinking, they were used for development and debugging, however they fail on deployment server (because those locations do not exist). Testing this module will be little bit tricky and will require to create fake filesystem structures. (Redmine issue: #1017,#3387)
Fix: Fixed unit tests broken after implementing timezone selection feature into reports.
(Redmine issue: #1017)
Fix: Fixed broken unit tests for PostgreSQL service.
On some versions of PostgreSQL the mogrify method returns differently formatted strings (extra space in after comma in lists). This needed to be fixed. (Redmine issue: #1017)
Fix: Fixed broken unit test after changing default value for max attachment size.
(Redmine issue: #1017)