Feature #4609
closedArbitrary grouping and sorting in Events
30%
Description
Comes from the CTI project - we will need to support arbitrary GROUP BY and SORT. We know this can be unexpected performance hog, so let's start with implementing it in a trivial way - add new subsection (along with Event, Detector, Admin, ...) like "Postprocess", and add there possibility to choose grouping columns and sorting column, which will translate to straightforward GROUP BY and ORDER BY. We'll then explore the limits (and maybe enforce some). Also, big fat warning should be added there that query time can be unpredictable.
Related issues
Updated by Pavel Kácha almost 6 years ago
- Related to Bug #4384: Possibility of DoS by repeating long query added
Updated by Radko Krkoš almost 6 years ago
I would advise against altering the Event search view and instead for introducing a new "Analytics" view capable of such arbitrary queries. From experience the users (such as UPJŠ Košice with timelines) are more accepting of a new feature that is not optimized yet, than breaking existing stuff. Also the proposed output grouping is in my opinion not really useful for event search.
Updated by Pavel Kácha almost 6 years ago
Radko Krkoš wrote:
I would advise against altering the Event search view and instead for introducing a new "Analytics" view capable of such arbitrary queries. From experience the users (such as UPJŠ Košice with timelines) are more accepting of a new feature that is not optimized yet, than breaking existing stuff. Also the proposed output grouping is in my opinion not really useful for event search.
My thought behind that was similar in that we can hide that "Postprocess" or "Analytics" button for admins, such as "Admins" button, until we're ready to publish it.
Pros/cons:
- additional button in Alerts means we would have to make result table display more flexible (because columns from grouping can be very different, and also we can have different sort). So I see what you mean by possibility of breaking existing stuff.
- Different view means duplication of pretty much all of Alerts query view plus adding some more.
Not sure which is better, maybe you're right for the beginning and we'll see if it can be integrated better somewhere.
Thinking about that, maybe it overlaps in funcionality partially with Timeline view - Timeline view is in fact very simple query grouped by various attributes (Counts, #abuses, #analysers, ...), sorted by resulting counts and shown on both timeline and piechart.
I've asked Martin for usecase CTI document to have more tangible image of what the customer actually wants, I'll post relevant excerpt here as soon as I have it.
Updated by Pavel Kácha almost 6 years ago
Excerpt from CTI customer requirements (doc in czech):
Analýza¶
Kategorie:Proaktivita
Případ užití: CTI bude podporovat analýzu událostí uchovávaných v systému pomocí dotazování se na uchovávané události. Dotazy v sobě budou typicky zahrnovat výběr časového okna dotazu, filtr, agregaci, řazení (top-n) dle relevantních polí události.
Příklad uživatelského scénáře: GovCERT.CZ zaznamená zvýšený počet událostí typu hádání hesel. Analytik se dotazuje systému, aby odhalil, zda toto zvýšení koreluje s výskytem například nového malware.
Updated by Pavel Kácha almost 6 years ago
From 2019-02-07 meeting: We should be able to do "analytics" from the same set of conditions as searching for Alerts. If we go for separate top level "tab", we will lose the possibility to easily run the same query for both alerts search and analytics - or we'll have to allow some user-conscious propagation of the query form between alerts search and analytics.
Not resolved on the meeting, will need some more discussion.
Updated by Pavel Kácha over 5 years ago
Forgotten update from live meeting:
- let's start with widening timeline form by using form from Alerts search template
- generate the same timeline info from obtained data in the same way
- add navigation links Alerts -> Timeline and Timeline -> Alerts to allow obtaining the same data in both full and aggregated way
Updated by Pavel Kácha almost 5 years ago
- To be discussed changed from No to Yes
Updated by Pavel Kácha almost 5 years ago
- Target version changed from Backlog to 2.7
Updated by Jan Mach almost 5 years ago
- Status changed from New to In Progress
- % Done changed from 0 to 30
- To be discussed set to Yes
Updated by Pavel Kácha almost 5 years ago
- Related to Bug #6252: Timeline graphing is causing mayhem on production added
Updated by Pavel Kácha almost 5 years ago
- Related to Feature #6256: Review possibilities of support of timeline calculation on db added
Updated by Pavel Kácha almost 5 years ago
- Related to Feature #6257: Review calculated statistics in timeline added
Updated by Pavel Kácha almost 5 years ago
- Status changed from In Progress to Closed
Basic functionality implemented, problems and potentional rehaul/changes are tracked in their own tickets.