Project

General

Profile

Actions

Feature #4609

closed

Arbitrary grouping and sorting in Events

Added by Pavel Kácha almost 6 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
High
Assignee:
Category:
Development - GUI
Target version:
Start date:
01/30/2019
Due date:
% Done:

30%

Estimated time:
To be discussed:

Description

Comes from the CTI project - we will need to support arbitrary GROUP BY and SORT. We know this can be unexpected performance hog, so let's start with implementing it in a trivial way - add new subsection (along with Event, Detector, Admin, ...) like "Postprocess", and add there possibility to choose grouping columns and sorting column, which will translate to straightforward GROUP BY and ORDER BY. We'll then explore the limits (and maybe enforce some). Also, big fat warning should be added there that query time can be unpredictable.


Related issues

Related to Mentat - Bug #4384: Possibility of DoS by repeating long queryClosedJan Mach10/19/2018

Actions
Related to Mentat - Bug #6252: Timeline graphing is causing mayhem on productionClosedJan Mach03/05/2020

Actions
Related to Mentat - Feature #6256: Review possibilities of support of timeline calculation on dbFeedbackPavel Kácha07/07/2020

Actions
Related to Mentat - Feature #6257: Review calculated statistics in timelineClosedJakub Maloštik03/09/2020

Actions
Actions #1

Updated by Pavel Kácha almost 6 years ago

  • Related to Bug #4384: Possibility of DoS by repeating long query added
Actions #2

Updated by Radko Krkoš almost 6 years ago

I would advise against altering the Event search view and instead for introducing a new "Analytics" view capable of such arbitrary queries. From experience the users (such as UPJŠ Košice with timelines) are more accepting of a new feature that is not optimized yet, than breaking existing stuff. Also the proposed output grouping is in my opinion not really useful for event search.

Actions #3

Updated by Pavel Kácha almost 6 years ago

Radko Krkoš wrote:

I would advise against altering the Event search view and instead for introducing a new "Analytics" view capable of such arbitrary queries. From experience the users (such as UPJŠ Košice with timelines) are more accepting of a new feature that is not optimized yet, than breaking existing stuff. Also the proposed output grouping is in my opinion not really useful for event search.

My thought behind that was similar in that we can hide that "Postprocess" or "Analytics" button for admins, such as "Admins" button, until we're ready to publish it.

Pros/cons:

  • additional button in Alerts means we would have to make result table display more flexible (because columns from grouping can be very different, and also we can have different sort). So I see what you mean by possibility of breaking existing stuff.
  • Different view means duplication of pretty much all of Alerts query view plus adding some more.

Not sure which is better, maybe you're right for the beginning and we'll see if it can be integrated better somewhere.

Thinking about that, maybe it overlaps in funcionality partially with Timeline view - Timeline view is in fact very simple query grouped by various attributes (Counts, #abuses, #analysers, ...), sorted by resulting counts and shown on both timeline and piechart.

I've asked Martin for usecase CTI document to have more tangible image of what the customer actually wants, I'll post relevant excerpt here as soon as I have it.

Actions #4

Updated by Pavel Kácha almost 6 years ago

Excerpt from CTI customer requirements (doc in czech):

Analýza

Kategorie:Proaktivita

Případ užití: CTI bude podporovat analýzu událostí uchovávaných v systému pomocí dotazování se na uchovávané události. Dotazy v sobě budou typicky zahrnovat výběr časového okna dotazu, filtr, agregaci, řazení (top-n) dle relevantních polí události.

Příklad uživatelského scénáře: GovCERT.CZ zaznamená zvýšený počet událostí typu hádání hesel. Analytik se dotazuje systému, aby odhalil, zda toto zvýšení koreluje s výskytem například nového malware.

Actions #5

Updated by Pavel Kácha almost 6 years ago

From 2019-02-07 meeting: We should be able to do "analytics" from the same set of conditions as searching for Alerts. If we go for separate top level "tab", we will lose the possibility to easily run the same query for both alerts search and analytics - or we'll have to allow some user-conscious propagation of the query form between alerts search and analytics.

Not resolved on the meeting, will need some more discussion.

Actions #6

Updated by Pavel Kácha over 5 years ago

Forgotten update from live meeting:

  • let's start with widening timeline form by using form from Alerts search template
  • generate the same timeline info from obtained data in the same way
  • add navigation links Alerts -> Timeline and Timeline -> Alerts to allow obtaining the same data in both full and aggregated way
Actions #7

Updated by Pavel Kácha almost 5 years ago

  • To be discussed changed from No to Yes
Actions #8

Updated by Pavel Kácha almost 5 years ago

  • Target version changed from Backlog to 2.7
Actions #9

Updated by Pavel Kácha almost 5 years ago

  • To be discussed deleted (Yes)
Actions #10

Updated by Jan Mach almost 5 years ago

  • Priority changed from Normal to High
Actions #11

Updated by Jan Mach over 4 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 30
  • To be discussed set to Yes
Actions #12

Updated by Pavel Kácha over 4 years ago

  • Related to Bug #6252: Timeline graphing is causing mayhem on production added
Actions #13

Updated by Pavel Kácha over 4 years ago

  • Related to Feature #6256: Review possibilities of support of timeline calculation on db added
Actions #14

Updated by Pavel Kácha over 4 years ago

  • Related to Feature #6257: Review calculated statistics in timeline added
Actions #15

Updated by Pavel Kácha over 4 years ago

  • Status changed from In Progress to Closed

Basic functionality implemented, problems and potentional rehaul/changes are tracked in their own tickets.

Actions #16

Updated by Pavel Kácha over 4 years ago

  • To be discussed deleted (Yes)
Actions

Also available in: Atom PDF