Project

General

Profile

Actions

Config #6251

open

Rethink the indices and planner cost optimizations in light of the current usage patterns

Added by Radko Krkoš over 1 year ago. Updated about 2 months ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
Design
Target version:
Start date:
03/05/2020
Due date:
% Done:

0%

Estimated time:
To be discussed:
No

Description

When the Mentat was migrated to PotgreSQL, the DB was version 9.6. The current version is v12, with v13 nearing. Tremendous amount of work was done upstream in the database, many of the premises valid then are no longer true. Also, the capabilities of Mentat grew in ways not foreseen or at least not originally intended (thanks to CTI).
We need to go back to drawing board, analyze the analytics processing needs of current Mentat, the processing capabilities of current PostgreSQL, and come up with a new database configuration and indexing schema, addressing the new requirements.


Related issues

Related to Mentat - Bug #6211: Missing indices on aggregated columnsClosedPavel Kácha02/07/2020

Actions
Related to Mentat - Feature #6413: Autarkic DB queries for timelineDeferred07/07/2020

Actions
Actions #1

Updated by Radko Krkoš over 1 year ago

  • Related to Bug #6211: Missing indices on aggregated columns added
Actions #2

Updated by Radko Krkoš 9 months ago

  • To be discussed changed from No to Yes

It seems that most of the current performance problems have proposed solutions that are being implemented. Are there any remaining data access methods with insufficient performance? Perhaps the CTI API was not analyzed thoroughly in this regard?

Actions #3

Updated by Radko Krkoš 9 months ago

  • Related to Feature #6413: Autarkic DB queries for timeline added
Actions #4

Updated by Radko Krkoš 9 months ago

  • To be discussed changed from Yes to No
Actions #5

Updated by Radko Krkoš 8 months ago

  • Status changed from New to Deferred
  • Assignee changed from Radko Krkoš to Pavel Kácha

Based on discussion, currently the performance is adequate. NERD is happy with the improvement for IP search. CTI's opinion is to be queried on some future meeting.

Actions #6

Updated by Radko Krkoš about 2 months ago

  • To be discussed changed from No to Yes

Radko Krkoš wrote in #note-5:

CTI's opinion is to be queried on some future meeting.

Any progress on this?

Actions #7

Updated by Pavel Kácha about 2 months ago

From CTI Apache log:

   1961x /mentat/api/events/dashboard
     92x /mentat/api/events/search
     43x /mentat/api/events/metadata
      6x /mentat/api/events/ID/show

Actions #8

Updated by Radko Krkoš about 2 months ago

  • Status changed from Deferred to In Progress
  • Assignee changed from Pavel Kácha to Radko Krkoš

Based on discussion, there are some problems with dashboard performance for CTI's use. Dashboard is based primarily on statistician's output, so the flow of operation should be analyzed. Also, the possibility of replacing statistician by timeline mechanism should be considered.

Actions #9

Updated by Radko Krkoš about 2 months ago

  • To be discussed changed from Yes to No
Actions

Also available in: Atom PDF