Config #6251
openRethink the indices and planner cost optimizations in light of the current usage patterns
0%
Description
When the Mentat was migrated to PotgreSQL, the DB was version 9.6. The current version is v12, with v13 nearing. Tremendous amount of work was done upstream in the database, many of the premises valid then are no longer true. Also, the capabilities of Mentat grew in ways not foreseen or at least not originally intended (thanks to CTI).
We need to go back to drawing board, analyze the analytics processing needs of current Mentat, the processing capabilities of current PostgreSQL, and come up with a new database configuration and indexing schema, addressing the new requirements.
Related issues
Updated by Radko Krkoš about 4 years ago
- Related to Bug #6211: Missing indices on aggregated columns added
Updated by Radko Krkoš over 3 years ago
- To be discussed changed from No to Yes
It seems that most of the current performance problems have proposed solutions that are being implemented. Are there any remaining data access methods with insufficient performance? Perhaps the CTI API was not analyzed thoroughly in this regard?
Updated by Radko Krkoš over 3 years ago
- Related to Feature #6413: Autarkic DB queries for timeline added
Updated by Radko Krkoš over 3 years ago
- Status changed from New to Deferred
- Assignee changed from Radko Krkoš to Pavel Kácha
Based on discussion, currently the performance is adequate. NERD is happy with the improvement for IP search. CTI's opinion is to be queried on some future meeting.
Updated by Radko Krkoš almost 3 years ago
- To be discussed changed from No to Yes
Radko Krkoš wrote in #note-5:
CTI's opinion is to be queried on some future meeting.
Any progress on this?
Updated by Pavel Kácha almost 3 years ago
From CTI Apache log:
1961x /mentat/api/events/dashboard
92x /mentat/api/events/search
43x /mentat/api/events/metadata
6x /mentat/api/events/ID/show
Updated by Radko Krkoš almost 3 years ago
- Status changed from Deferred to In Progress
- Assignee changed from Pavel Kácha to Radko Krkoš
Based on discussion, there are some problems with dashboard performance for CTI's use. Dashboard is based primarily on statistician's output, so the flow of operation should be analyzed. Also, the possibility of replacing statistician by timeline mechanism should be considered.
Updated by Radko Krkoš almost 3 years ago
- To be discussed changed from Yes to No
Updated by Radko Krkoš over 2 years ago
- dashboard (intensive)
- search (rare)
- event display
Out of which, there seem to be some performance issues with the dashboard. As this component is based on statistician and we are evaluating the possibility of its replacement with the timelines, the use cases should be incorporated in the plan. In the meanwhile a mild analysis & optimization pass can be made on the slow queries of the dashboard.
Updated by Radko Krkoš almost 2 years ago
- Status changed from In Progress to Deferred
This should be postponed until after the statistician situation is dealt with.