Project

General

Profile

Actions

Task #3752

closed

Migration from MongoDB to PostgreSQL

Added by Jan Mach almost 5 years ago. Updated about 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Development - Core
Target version:
Start date:
10/10/2017
Due date:
% Done:

100%

Estimated time:
To be discussed:

Description

After many months of rigorous testing we have finally decided to migrate from MongoDB to PostgreSQL. The MongoDB performance and long response times are decreasing the usability of the web interface and even of the reporting subsystem. We will now abandon the MongoDB in favor of PostgreSQL.

The migration will occur in following phases:
  1. Define data models for utility objects (users, groups, etc.)
  2. Implement new storage service module for utility object management.
  3. Rewrite all current code working with utility objects.
  4. Implement new storage service for IDEA event management (maybe it will be possible to use the same service as above, maybe there will have to be different one, highly optimized and with lower overhead).
  5. Rewrite all current code working with IDEA messages through persistent database storage.

Related issues

Related to Mentat - Task #3362: Migrate Mentat system to new hardwareClosedJan Mach03/21/2017

Actions
Related to Mentat - Bug #2808: Python and batch_size constraintClosedJan Mach03/01/2016

Actions
Actions #1

Updated by Pavel Kácha almost 5 years ago

Notes for future production migration:
  • both db engines on the same machine
  • user/group management + relevant data – can be migrated independently of event db
    • then management GUI
    • we will need migration script, which will allow reporter/statistician use Perl DB until also rewritten
  • statistician
    • simple incremental conversion script from Mongo to PQ + GUI, which already looks into PQ
    • then statistician can be written in python, switched and conversion script ditched
  • reporter
    • simple incremental conversion script from Mongo to PQ + GUI, which already looks into PQ
    • then reporter can be written in python, switched and conversion script ditched
  • incoming events forked into both old (Mongo) storage and new (PQ) storage daemons (events go simultaneously into both DBs)
    • then GUI can be written independently
Actions #2

Updated by Jan Mach over 4 years ago

  • Related to Task #3362: Migrate Mentat system to new hardware added
Actions #3

Updated by Jan Mach about 4 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

On tuesday 24.7. 2018 we have successfully migrated our production instance of Mentat system to version of Mentat system that is completely written in Python and uses PostgreSQL as database backend. Therefore this task can be considered complete.

Actions #4

Updated by Jan Mach about 4 years ago

  • Related to Bug #2808: Python and batch_size constraint added
Actions

Also available in: Atom PDF