Support #4628
closedConsider/implement modeling of event database with SQLAlchemy declarative base.
0%
Description
For performance reasons we now use psycopg2 driver to manage and access. It would feel right to consolidate some of the database creation and management code by modelling the database schema with SQLAlchemy declarative base. It would be then possible to create and manage the database and database migrations in uniform way.
Related issues
Updated by Jan Mach almost 6 years ago
- Related to Feature #4230: Make use of existing or implement own mechanism for handling SQL schema migrations. added
Updated by Radko Krkoš about 4 years ago
- Tracker changed from Feature to Support
- Category changed from Development - Core to Research and analysis
- Assignee changed from Jan Mach to Radko Krkoš
Updated by Radko Krkoš about 4 years ago
Based on the videocall discussion, the motivation behind this issue is to get rid of Alembic
. Alembic
is a database schema migration tool created by the same author as SQLAlchemy
[1], mainly to support the database migrations for SQLAlchemy
-based projects. Of course, Alembic
can be used without SQLAlchemy
. As it happens, both mentat_main
, which is SQLAlchemy
managed, and mentat_events
, which is managed using SQL
DDL
descriptions, use Alembic
for migrations, the scripts are source:/lib/hawat/migrations/versions@devel and source:/migrations-events/versions@devel respectively. Alembic does have limited support for generating migrations scripts from SQLAlchemy
schema changes, those must always be reviewed, fixed and tested manually. As for the mentat_events
database, the migration scripts there surpass the abilities of automatic script generation greatly, so a schema definition using SQLAlchemy
would not have helped much.
To sum it up, discarding Alembic
does not seem realistic (without replacing it with something roughly identical) and Alembic
does not depend on SQLAlchemy
and can be used separately.
Based on discussion in issue #4230, which this issue stemmed from, the motivation for this issue was to unify "database creation, management and cleanup tasks". It would be fairly trivial to replace the direct SQL
DDL
description in source:/lib/mentat/services/eventstorage.py@devel, namely the database_create()
and index_create()
methods, using SQLAlchemy
definitions and DDL
calls ([2] is a simple tutorial containing almost all the required information). Then, for the cleanup and management tasks, a significant portion of EventStorage
would have to be reimplemented. Of course the actual data access (and even some management tasks, such as the enum
tables manipulation) is too complex to implement using pure SQLAlchemy
, but that is outside of the scope of this issue anyways.
[1] https://pypi.org/project/alembic/
[2] https://www.tutorialspoint.com/sqlalchemy/sqlalchemy_core_creating_table.htm
Updated by Radko Krkoš about 4 years ago
CTEs (common table expressions) are in fact possible with SQLAlchemy
[1], so enum
table manipulation queries can be translated. However the syntax is quite lengthy and subjectively non-obvious, as illustrated in the example code in the manual.
Also, it is possible to define GIN
indices in the model. [2]
[1] https://docs.sqlalchemy.org/en/14/orm/query.html#sqlalchemy.orm.Query.cte
[2] https://docs.sqlalchemy.org/en/14/dialects/postgresql.html#postgresql-indexes
Updated by Pavel Kácha about 4 years ago
- Status changed from New to Rejected
- To be discussed deleted (
Yes)
- Migrations have been yet pretty much always so complex that they cannot be generated by Alembic from SQL Alchemy, but handcrafted.
- Main and Events logic is very separate, we would not reach reasonable unification even with SQL Alchemy declaration.
- User/Group/Network/Filter relations make sense to be modelled and enforced by means of SQL Alchemy, but we don't have any of those in Events logic (except just one relation to raw data table).
- Table description is quite unreadable in SQL Alchemy, same for coping with cleanup and management tasks.
Ergo - quite a hunk of work not balanced by benefits.
Mek, you've raised it up originally, so if you disagree or we do neglect something, please speak up/reopen.
Updated by Pavel Kácha almost 4 years ago
- Target version changed from Backlog to 2.7