Task #3752
closed
Migration from MongoDB to PostgreSQL
Added by Jan Mach over 6 years ago.
Updated over 5 years ago.
Category:
Development - Core
Description
After many months of rigorous testing we have finally decided to migrate from MongoDB to PostgreSQL. The MongoDB performance and long response times are decreasing the usability of the web interface and even of the reporting subsystem. We will now abandon the MongoDB in favor of PostgreSQL.
The migration will occur in following phases:
- Define data models for utility objects (users, groups, etc.)
- Implement new storage service module for utility object management.
- Rewrite all current code working with utility objects.
- Implement new storage service for IDEA event management (maybe it will be possible to use the same service as above, maybe there will have to be different one, highly optimized and with lower overhead).
- Rewrite all current code working with IDEA messages through persistent database storage.
Notes for future production migration:
- both db engines on the same machine
- user/group management + relevant data – can be migrated independently of event db
- then management GUI
- we will need migration script, which will allow reporter/statistician use Perl DB until also rewritten
- statistician
- simple incremental conversion script from Mongo to PQ + GUI, which already looks into PQ
- then statistician can be written in python, switched and conversion script ditched
- reporter
- simple incremental conversion script from Mongo to PQ + GUI, which already looks into PQ
- then reporter can be written in python, switched and conversion script ditched
- incoming events forked into both old (Mongo) storage and new (PQ) storage daemons (events go simultaneously into both DBs)
- then GUI can be written independently
- Related to Task #3362: Migrate Mentat system to new hardware added
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
On tuesday 24.7. 2018 we have successfully migrated our production instance of Mentat system to version of Mentat system that is completely written in Python and uses PostgreSQL as database backend. Therefore this task can be considered complete.
- Related to Bug #2808: Python and batch_size constraint added
Also available in: Atom
PDF