Project

General

Profile

Feature #4609

Arbitrary grouping and sorting in Events

Added by Pavel Kácha about 1 year ago. Updated 20 days ago.

Status:
Closed
Priority:
High
Assignee:
Category:
Development - GUI
Target version:
Start date:
01/30/2019
Due date:
% Done:

30%

Estimated time:
To be discussed:

Description

Comes from the CTI project - we will need to support arbitrary GROUP BY and SORT. We know this can be unexpected performance hog, so let’s start with implementing it in a trivial way - add new subsection (along with Event, Detector, Admin, ...) like “Postprocess”, and add there possibility to choose grouping columns and sorting column, which will translate to straightforward GROUP BY and ORDER BY. We’ll then explore the limits (and maybe enforce some). Also, big fat warning should be added there that query time can be unpredictable.


Related issues

Related to Mentat - Bug #4384: Possibility of DoS by repeating long queryClosed10/19/2018

Related to Mentat - Bug #6252: Timeline graphing is causing mayhem on productionNew03/05/2020

Related to Mentat - Feature #6256: Review possibilities of support of timeline calculation on dbNew03/09/2020

Related to Mentat - Feature #6257: Review calculated statistics in timelineNew03/09/2020

Associated revisions

Revision 021eedad (diff)
Added by Jan Mach about 2 months ago

This patch brings in timeline search improvements.

  • Support for almost full event search form (except page, limit and searchby parameres, which do not make sense).
  • Redirection from Timeline search result page to Event search page and vice versa.
    (Redmine issue: #4609)

Revision ee01ef40 (diff)
Added by Jan Mach about 2 months ago

Fix: Fixed issue with invalid time windows when linking between event search and timeline views.

As it turned out the timepager component was modifiing request parameters to generate the links and subsequent components were unable to access original values. (Redmine issue: #4609)

Revision 6ecf0eb1 (diff)
Added by Jan Mach about 2 months ago

Added default 'detect time from’ time boundary in case there are no time intervals defined.

  • If none of the following search parameters (dt_from, dt_to, st_from, st_to) is present in the URL, provide default dt_from (last 7 days) to reduce amount of processed events.
  • User still has the ability to perform unbound search by submitting empty form.

(Redmine issue: #4609)

Revision ece4af50 (diff)
Added by Jan Mach about 1 month ago

Added more insistent default time value for 'dt_from’ in certain views.

Endpoints events.search, timeline.search and reports.search now have more insistent default value for dt_from parameter to make default queries less demanding on resources. (Redmine issue: #4609,#6252)

History

#1 Updated by Pavel Kácha about 1 year ago

  • Related to Bug #4384: Possibility of DoS by repeating long query added

#2 Updated by Radko Krkoš about 1 year ago

I would advise against altering the Event search view and instead for introducing a new “Analytics” view capable of such arbitrary queries. From experience the users (such as UPJŠ Košice with timelines) are more accepting of a new feature that is not optimized yet, than breaking existing stuff. Also the proposed output grouping is in my opinion not really useful for event search.

#3 Updated by Pavel Kácha about 1 year ago

Radko Krkoš wrote:

I would advise against altering the Event search view and instead for introducing a new “Analytics” view capable of such arbitrary queries. From experience the users (such as UPJŠ Košice with timelines) are more accepting of a new feature that is not optimized yet, than breaking existing stuff. Also the proposed output grouping is in my opinion not really useful for event search.

My thought behind that was similar in that we can hide that “Postprocess” or “Analytics” button for admins, such as “Admins” button, until we’re ready to publish it.

Pros/cons:

  • additional button in Alerts means we would have to make result table display more flexible (because columns from grouping can be very different, and also we can have different sort). So I see what you mean by possibility of breaking existing stuff.
  • Different view means duplication of pretty much all of Alerts query view plus adding some more.

Not sure which is better, maybe you’re right for the beginning and we’ll see if it can be integrated better somewhere.

Thinking about that, maybe it overlaps in funcionality partially with Timeline view - Timeline view is in fact very simple query grouped by various attributes (Counts, #abuses, #analysers, ...), sorted by resulting counts and shown on both timeline and piechart.

I’ve asked Martin for usecase CTI document to have more tangible image of what the customer actually wants, I’ll post relevant excerpt here as soon as I have it.

#4 Updated by Pavel Kácha about 1 year ago

Excerpt from CTI customer requirements (doc in czech):

Analýza

Kategorie:Proaktivita

Případ užití: CTI bude podporovat analýzu událostí uchovávaných v systému pomocí dotazování se na uchovávané události. Dotazy v sobě budou typicky zahrnovat výběr časového okna dotazu, filtr, agregaci, řazení (top-n) dle relevantních polí události.

Příklad uživatelského scénáře: GovCERT.CZ zaznamená zvýšený počet událostí typu hádání hesel. Analytik se dotazuje systému, aby odhalil, zda toto zvýšení koreluje s výskytem například nového malware.

#5 Updated by Pavel Kácha about 1 year ago

From 2019-02-07 meeting: We should be able to do “analytics” from the same set of conditions as searching for Alerts. If we go for separate top level “tab”, we will lose the possibility to easily run the same query for both alerts search and analytics - or we’ll have to allow some user-conscious propagation of the query form between alerts search and analytics.

Not resolved on the meeting, will need some more discussion.

#6 Updated by Pavel Kácha 11 months ago

Forgotten update from live meeting:

  • let’s start with widening timeline form by using form from Alerts search template
  • generate the same timeline info from obtained data in the same way
  • add navigation links Alerts → Timeline and Timeline → Alerts to allow obtaining the same data in both full and aggregated way

#7 Updated by Pavel Kácha 2 months ago

  • To be discussed changed from No to Yes

#8 Updated by Pavel Kácha about 2 months ago

  • Target version changed from Future to 2.7

#9 Updated by Pavel Kácha about 2 months ago

  • To be discussed deleted (Yes)

#10 Updated by Jan Mach about 2 months ago

  • Priority changed from Normal to High

#11 Updated by Jan Mach about 1 month ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 30
  • To be discussed set to Yes

#12 Updated by Pavel Kácha about 1 month ago

  • Related to Bug #6252: Timeline graphing is causing mayhem on production added

#13 Updated by Pavel Kácha about 1 month ago

  • Related to Feature #6256: Review possibilities of support of timeline calculation on db added

#14 Updated by Pavel Kácha about 1 month ago

  • Related to Feature #6257: Review calculated statistics in timeline added

#15 Updated by Pavel Kácha 20 days ago

  • Status changed from In Progress to Closed

Basic functionality implemented, problems and potentional rehaul/changes are tracked in their own tickets.

#16 Updated by Pavel Kácha 20 days ago

  • To be discussed deleted (Yes)

Also available in: Atom PDF