https://homeproj.cesnet.cz/https://homeproj.cesnet.cz/httpauth-login/favicon.ico?16194486082020-03-09T12:54:58ZHomeproj: Redmine for CESNETMentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=264622020-03-09T12:54:58ZPavel Káchaph@cesnet.cz
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-5 priority-5 priority-high3 closed" href="/issues/4609">Feature #4609</a>: Arbitrary grouping and sorting in Events</i> added</li></ul> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=265692020-03-30T08:22:18ZPavel Káchaph@cesnet.cz
<ul></ul><p>My thoughts, only proposition, open for discussion.</p>
<ul>
<li>Totals
<ul>
<li>One of key measures.</li>
<li>For timeline can be calculated relatively quickly (count(*) ... group<br /> by).</li>
<li>For statistician data it's small simple value.</li>
<li><strong>Keep.</strong></li>
</ul></li>
</ul>
<ul>
<li>Recurrences
<ul>
<li>I'm not sure if its useful in its current form. Pie chart is almost<br /> always 100%. Bars may show that in some specific time there was a peak<br /> in recurrent events, alas there's no possibility of finding out which<br /> ones (apart from trying to correlate with columns on other graphs with<br /> possibly different scales and the drilling into Alerts searches).</li>
<li>As i understand it, it needs some consultation with relapse<br /> cache during calculation, which might be costly <span class="wiking smiley smiley-question" title="(?)"></span>.</li>
<li>If we accept classes as first class citizen (pun not intended), partial<br /> information can be visible from graph (sans source)</li>
<li>I incline to <strong>ditching</strong> it.</li>
</ul></li>
</ul>
<ul>
<li>Abuses
<ul>
<li>Now makes sense only for constituency admins (like us).</li>
<li>Usable for abuse team admins only after we have hierarchical networks.</li>
<li><strong>Keep</strong>, we'll need it in the future.</li>
</ul></li>
</ul>
<ul>
<li>Analyzers
<ul>
<li>Names of analyzers are completely arbitrary, often cryptic.</li>
<li>Useful only for constituency admins (like us).</li>
<li><strong>Ditch.</strong></li>
</ul></li>
</ul>
<ul>
<li>ASNs
<ul>
<li>I believe this is used very seldomly.</li>
<li>Not sure if useful for security/forensics team work.</li>
<li>However nice in graphs...</li>
<li>... if accompanied with company name.</li>
<li><strong>Not sure.</strong></li>
</ul></li>
</ul>
<ul>
<li>Categories
<ul>
<li>Straightforward (one category).</li>
<li>Problem with ambigous statistics (sum is more than number of events).</li>
<li>Personaly I've never used it, always used category sets.</li>
<li>Calculable from Category sets statistics, if needed.</li>
<li>For my part, <strong>ditch.</strong></li>
</ul></li>
</ul>
<ul>
<li>Category sets
<ul>
<li>One of key measures.</li>
<li>Statistics sums up.</li>
<li>Useful would be possibility to merge some category sets, like 'omit<br /> Test, group the rest'.</li>
<li><strong>Keep.</strong></li>
</ul></li>
</ul>
<ul>
<li>Countries
<ul>
<li>Same as ASNs.</li>
<li>Isn't this derivable from ASNs? (For stats precalculation.)</li>
<li><strong>Not sure.</strong></li>
</ul></li>
</ul>
<ul>
<li>Detectors
<ul>
<li>One of key measures.</li>
<li><strong>Keep.</strong></li>
</ul></li>
</ul>
<ul>
<li>Detector software
<ul>
<li>Similar to analyzers.</li>
<li>Moreover, most of the clients have only one SW behind (except Filers, which we don't care dearly for.)</li>
<li><strong>Ditch.</strong></li>
</ul></li>
</ul>
<ul>
<li>IPs
<ul>
<li>One of key measures.</li>
<li>Rename to Sources?</li>
<li><strong>Keep.</strong></li>
</ul></li>
</ul>
<ul>
<li>Classes
<ul>
<li>Mmmkay.</li>
<li>Not convinced strongly, but let's <strong>keep.</strong></li>
</ul></li>
</ul>
<ul>
<li>Severities
<ul>
<li>Also not sure, if useful in current form. Bars may show that in some<br /> specific time there was a peak in recurrent events, alas there's no<br /> possibility of finding out which ones (apart from trying to correlate<br /> with columns on other graphs with possibly different scales and<br /> drilling into Alerts searches).</li>
<li>Have anybody ever used this stats?</li>
<li><strong>Ditch?</strong></li>
</ul></li>
</ul>
<p>We don't have:</p>
<ul>
<li>Ports
<ul>
<li>Would be nice for spotting rise of attacks to specific ports.</li>
<li><strong>Add?</strong></li>
</ul></li>
</ul>
<ul>
<li>Services
<ul>
<li>(Un)fortunately it's partially orthogonal to ports.</li>
<li>Would be nice for spotting rise of attacks to specific services.</li>
<li><strong>Add?</strong></li>
</ul></li>
</ul>
<ul>
<li>Targets
<ul>
<li>Doesn't it make sense for network admins to see their hotspots?</li>
<li><strong>Add?</strong></li>
</ul></li>
</ul> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=265722020-03-30T13:12:28ZPavel Káchaph@cesnet.cz
<ul><li><strong>To be discussed</strong> changed from <i>No</i> to <i>Yes</i></li></ul> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=265852020-04-01T13:38:47ZJan Machjan.mach@cesnet.cz
<ul><li><strong>File</strong> <a href="/attachments/3526">screencapture-mentat-alt-cesnet-cz-mentat-timeline-search-2020-04-01-15_27_13.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/3526/screencapture-mentat-alt-cesnet-cz-mentat-timeline-search-2020-04-01-15_27_13.png">screencapture-mentat-alt-cesnet-cz-mentat-timeline-search-2020-04-01-15_27_13.png</a> added</li><li><strong>File</strong> <a href="/attachments/3527">screencapture-mentat-alt-cesnet-cz-mentat-timeline-search-2020-04-01-15_28_36_profiler.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/3527/screencapture-mentat-alt-cesnet-cz-mentat-timeline-search-2020-04-01-15_28_36_profiler.png">screencapture-mentat-alt-cesnet-cz-mentat-timeline-search-2020-04-01-15_28_36_profiler.png</a> added</li><li><strong>% Done</strong> changed from <i>0</i> to <i>10</i></li></ul><p>According to the Pavel`s recommendation I have done example query including time measurement and profiling. Results are in attached files.</p>
<pre>
Number of events: 1,498,994
Time interval: Mar 31, 2020, 11:00:00 PM - Apr 1, 2020, 4:00:00 PM
Time interval delta: 17 hours
Timeline steps: 170 x 0:06:00
Query:
SELECT * FROM events AS "_mentatq(88_gkbyww)_" WHERE "detecttime" >= '2020-03-31T21:00:00+00:00'::timestamptz
Time marks :
search:preprocess_begin
search:preprocess_end 0:00:00.000624 s 0%
search:search_begin 0:00:00.000426 s 0%
search:search_end 0:16:19.091261 s 62%
search:postprocess_begin 0:00:00.030789 s 0%
search:postprocess_end 0:10:06.621327 s 38%
Total duration: 0:26:25.744427
Duration to rendering time: 0:26:27.138112
</pre> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=266152020-04-03T12:07:27ZPavel Káchaph@cesnet.cz
<ul><li><strong>Assignee</strong> changed from <i>Pavel Kácha</i> to <i>Radko Krkoš</i></li></ul><p>From today's video meeting:</p>
<ul>
<li>From timemarks is seems like DB query is long, calculations are shorter. However, from 'search' part comprises also creating lite Idea objects (typedcols, ipranges) and some jpath accesses, which seems to take most of the data fetching part. So the possibilities:
<ul>
<li>Somehow get rid of the object creation and abstraction layers, however even though it might shortcut most of the processing, it would also sidestep most of the logic and go against the Hawat architecture.</li>
</ul>
<ul>
<li>Somehow coerce PostgreSQL to do the calculations itself and hand back just final data.</li>
</ul></li>
</ul>
<p>So, Radko will try to come up with some queries, calculating data for graph, and roughly estimating performance, then we'll evaluate next steps.</p> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=266162020-04-03T12:15:51ZPavel Káchaph@cesnet.cz
<ul></ul><p>Also, discussion of grouped categories appreciated.</p>
<p>Considering recurrence - combined aggregation of IP + class might show interesting toplist of recurrent events.</p> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=266642020-04-08T11:59:49ZRadko Krkoškrkos@cesnet.cz
<ul><li><strong>File</strong> <a href="/attachments/3528">mentat_timeline_bug.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/3528/mentat_timeline_bug.png">mentat_timeline_bug.png</a> added</li><li><strong>% Done</strong> changed from <i>10</i> to <i>20</i></li></ul><p>The analytic queries have been developed according to the currently presented results. The performance was assessed.<br />As a baseline, let's select a plain query for the then current day, 00:00:00 - 06:00:00. The current result is available here:<br /><a class="external" href="https://mentat-alt.cesnet.cz/mentat/timeline/search?dt_from=2020-04-08+00%3A00%3A00&dt_to=2020-04-08+06%3A00%3A00&source_addrs=&source_ports=&submit=Search">https://mentat-alt.cesnet.cz/mentat/timeline/search?dt_from=2020-04-08+00%3A00%3A00&dt_to=2020-04-08+06%3A00%3A00&source_addrs=&source_ports=&submit=Search</a></p>
Hawat decided for 3 minute intervals and the calculation took 2 m 50 s (170s).<br />To obtain the same results, we need several queries, the primary:<br /><pre>
SELECT '2020-04-08 00:00:00'::timestamp + '3 mins'::interval * (-1 + width_bucket(detecttime, (SELECT array_agg(buckets) FROM generate_series('2020-04-08 00:00:00'::timestamp, '2020-04-08 06:00:00', '3 mins'::interval) AS buckets))) AS bucket, COUNT(*) FROM events WHERE detecttime > '2020-04-08 00:00:00' AND detecttime < '2020-04-08 06:00:00' GROUP BY bucket ORDER BY bucket;
</pre>
<ul>
<li>took ~150ms.</li>
</ul>
For abuses, this can be trivially extended:<br /><pre>
SELECT '2020-04-08 00:00:00'::timestamp + '3 mins'::interval * (-1 + width_bucket(detecttime, (SELECT array_agg(buckets) FROM generate_series('2020-04-08 00:00:00'::timestamp, '2020-04-08 06:00:00', '3 mins'::interval) AS buckets))) AS bucket, events.cesnet_resolvedabuses AS abuses, COUNT(*) FROM events WHERE detecttime > '2020-04-08 00:00:00' AND detecttime < '2020-04-08 06:00:00' GROUP BY bucket, abuses ORDER BY bucket;
</pre>
<ul>
<li>took ~800ms.</li>
</ul>
For multivalue arrays, <code>unnest()</code> can be used like this:<br /><pre>
SELECT '2020-04-08 00:00:00'::timestamp + '3 mins'::interval * (-1 + width_bucket(detecttime, (SELECT array_agg(buckets) FROM generate_series('2020-04-08 00:00:00'::timestamp, '2020-04-08 06:00:00', '3 mins'::interval) AS buckets))) AS bucket, unnest(events.category) AS cat, COUNT(*) FROM events WHERE detecttime > '2020-04-08 00:00:00' AND detecttime < '2020-04-08 06:00:00' GROUP BY bucket, cat ORDER BY bucket;
</pre>
<ul>
<li>took ~800ms.</li>
</ul>
<strong>Now for 1 month, 8 hour intervals</strong> (as a scaling test, this would not be practical with current implementation):<br /><pre>
SELECT '2020-03-01 00:00:00'::timestamp + '8 hours'::interval * (-1 + width_bucket(detecttime, (SELECT array_agg(buckets) FROM generate_series('2020-03-01 00:00:00'::timestamp, '2020-04-01 00:00:00', '8 hours'::interval) AS buckets))) AS bucket, COUNT(*) FROM events WHERE detecttime > '2020-03-01 00:00:00' AND detecttime < '2020-04-01 00:00:00' GROUP BY bucket ORDER BY bucket;
</pre>
<ul>
<li>took ~6s.</li>
</ul>
Another column:<br /><pre>
SELECT '2020-03-01 00:00:00'::timestamp + '8 hours'::interval * (-1 + width_bucket(detecttime, (SELECT array_agg(buckets) FROM generate_series('2020-03-01 00:00:00'::timestamp, '2020-04-01 00:00:00', '8 hours'::interval) AS buckets))) AS bucket, cesnet_resolvedabuses AS abuses, COUNT(*) FROM events WHERE detecttime > '2020-03-01 00:00:00' AND detecttime < '2020-04-01 00:00:00' GROUP BY bucket, abuses ORDER BY bucket;
</pre>
<ul>
<li>took ~15s.</li>
</ul>
Multivalue arrays:<br /><pre>
SELECT '2020-03-01 00:00:00'::timestamp + '8 hours'::interval * (-1 + width_bucket(detecttime, (SELECT array_agg(buckets) FROM generate_series('2020-03-01 00:00:00'::timestamp, '2020-04-01 00:00:00', '8 hours'::interval) AS buckets))) AS bucket, unnest(category) as cat, COUNT(*) FROM events WHERE detecttime > '2020-03-01 00:00:00' AND detecttime < '2020-04-01 00:00:00' GROUP BY bucket, cat ORDER BY bucket;
</pre>
<ul>
<li>took ~17s.</li>
<li>Remark: The run times also depend on the number of distinct array values (the aggregate dimensions).</li>
</ul>
<p><strong>The queries are parametrized by 4 values:</strong> start and end time (which are user input), the interval time, which is presumably already determined by some magic, and the optional additional column. Based on the runtimes, it should be practical to get the results per screen on user request.</p>
<p>If just an interval index since the beginning is required (based on how the graphing is implemented), the query can be simplified as follows:<br /><pre>
SELECT width_bucket(detecttime, (SELECT array_agg(buckets) FROM generate_series('2020-04-08 00:00:00'::timestamp, '2020-04-08 06:00:00', '6 minutes'::interval) AS buckets)) AS bucket, COUNT(*) FROM events WHERE detecttime > '2020-04-08 00:00:00' AND detecttime < '2020-04-08 06:00:00' GROUP BY bucket ORDER BY bucket;
</pre><br />This version is somewhat faster due to simpler sorting.</p>
<p>Also, calculation inside DB would help getting rid of yet another incarnation of the timezone bug. At least for me, the graph looks like in the attachment - the bins are shifted 2 hours to the future relative to data obtained.</p> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=266672020-04-09T11:16:21ZPavel Káchaph@cesnet.cz
<ul></ul><p>Just a note - is ORDER BY clause really necessary? I believe that the result for frontend will be dict of dicts and order of data processing doesn't matter.</p> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=266682020-04-09T11:26:09ZRadko Krkoškrkos@cesnet.cz
<ul></ul><p><a class="user active" href="https://homeproj.cesnet.cz/users/17">Pavel Kácha</a> wrote:</p>
<blockquote>
<p>Just a note - is ORDER BY clause really necessary? I believe that the result for frontend will be dict of dicts and order of data processing doesn't matter.</p>
</blockquote>
<p>Yes, I wanted to discuss that, but have forgotten.<br />My thinking is also that it should be left out. Also, the <code>bucket</code> column is now calculated as a timestamp. If just a bucket index would be enough, the calculation could be left out, shaving off some time.</p> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=266802020-04-09T15:39:04ZPavel Káchaph@cesnet.cz
<ul></ul><p><a class="user active" href="https://homeproj.cesnet.cz/users/391">Radko Krkoš</a> wrote:</p>
<blockquote>
<p><a class="user active" href="https://homeproj.cesnet.cz/users/17">Pavel Kácha</a> wrote:</p>
<blockquote>
<p>Just a note - is ORDER BY clause really necessary? I believe that the result for frontend will be dict of dicts and order of data processing doesn't matter.</p>
</blockquote>
<p>Yes, I wanted to discuss that, but have forgotten.<br />My thinking is also that it should be left out. Also, the <code>bucket</code> column is now calculated as a timestamp. If just a bucket index would be enough, the calculation could be left out, shaving off some time.</p>
</blockquote>
<p>Okay, I'll leave the decisions on your and Mek's discussion during implementation. <span class="wiking smiley smiley-smiley" title=":)"></span></p> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=266812020-04-09T15:42:11ZPavel Káchaph@cesnet.cz
<ul></ul><p>Also, somewhere on the course of implementing <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: Reimplement timeline calculations on database (Closed)" href="https://homeproj.cesnet.cz/issues/6308">#6308</a> (before or after) we should incorporate changes/removes/additions from <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: Review calculated statistics in timeline (Closed)" href="https://homeproj.cesnet.cz/issues/6257#note-2">#6257#note-2</a>.</p> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=266912020-04-14T08:03:02ZRadko Krkoškrkos@cesnet.cz
<ul><li><strong>Assignee</strong> changed from <i>Radko Krkoš</i> to <i>Pavel Kácha</i></li></ul><p>After discussion, now that I understand it more:</p>
<p><a class="user active" href="https://homeproj.cesnet.cz/users/17">Pavel Kácha</a> wrote:</p>
<blockquote>
<ul>
<li>Totals
<ul>
<li>One of key measures.</li>
<li>For timeline can be calculated relatively quickly (count(*) ... group<br />by).</li>
<li>For statistician data it's small simple value.</li>
<li><strong>Keep.</strong></li>
</ul></li>
</ul>
</blockquote>
<p>Definitely keep, this is the first level of any statistics.<br />As for the pie (doughnut) chart, I would either leave it out (it is always 100%), or graph individual time slots in it - displays relative distribution of events over the time period, must be done for fewer slots, hundred is too much.</p>
<p><a class="user active" href="https://homeproj.cesnet.cz/users/17">Pavel Kácha</a> wrote:</p>
<blockquote>
<ul>
<li>Recurrences
<ul>
<li>I'm not sure if its useful in its current form. Pie chart is almost<br />always 100%. Bars may show that in some specific time there was a peak<br />in recurrent events, alas there's no possibility of finding out which<br />ones (apart from trying to correlate with columns on other graphs with<br />possibly different scales and the drilling into Alerts searches).</li>
<li>As i understand it, it needs some consultation with relapse<br />cache during calculation, which might be costly <span class="wiking smiley smiley-question" title="(?)"></span>.</li>
<li>If we accept classes as first class citizen (pun not intended), partial<br />information can be visible from graph (sans source)</li>
<li>I incline to <strong>ditching</strong> it.</li>
</ul></li>
</ul>
</blockquote>
<p>I am not sure how to calculate that, not even algorithmically, surely not in the DB, so I would be for ditching it also.</p>
<p><a class="user active" href="https://homeproj.cesnet.cz/users/17">Pavel Kácha</a> wrote:</p>
<blockquote>
<ul>
<li>Abuses
<ul>
<li>Now makes sense only for constituency admins (like us).</li>
<li>Usable for abuse team admins only after we have hierarchical networks.</li>
<li><strong>Keep</strong>, we'll need it in the future.</li>
</ul></li>
</ul>
<ul>
<li>Analyzers
<ul>
<li>Names of analyzers are completely arbitrary, often cryptic.</li>
<li>Useful only for constituency admins (like us).</li>
<li><strong>Ditch.</strong></li>
</ul></li>
</ul>
</blockquote>
<p>I incline on keep, this might be useful for some Warden related outputs, Warden client cleanup work and such.</p>
<p><a class="user active" href="https://homeproj.cesnet.cz/users/17">Pavel Kácha</a> wrote:</p>
<blockquote>
<ul>
<li>ASNs
<ul>
<li>I believe this is used very seldomly.</li>
<li>Not sure if useful for security/forensics team work.</li>
<li>However nice in graphs...</li>
<li>... if accompanied with company name.</li>
<li><strong>Not sure.</strong></li>
</ul></li>
</ul>
<ul>
<li>Categories
<ul>
<li>Straightforward (one category).</li>
<li>Problem with ambigous statistics (sum is more than number of events).</li>
<li>Personaly I've never used it, always used category sets.</li>
<li>Calculable from Category sets statistics, if needed.</li>
<li>For my part, <strong>ditch.</strong></li>
</ul></li>
</ul>
</blockquote>
<p>We discussed the "inverse Aristotelian dissonance" <span class="wiking smiley smiley-smiley" title=":)"></span> and the calculation should be changed to relative portion of the whole, therefore not inflating the total count. Then the result is something that was requested in SABU. I would keep this, with the changes.</p>
<p><a class="user active" href="https://homeproj.cesnet.cz/users/17">Pavel Kácha</a> wrote:</p>
<blockquote>
<ul>
<li>Category sets
<ul>
<li>One of key measures.</li>
<li>Statistics sums up.</li>
<li>Useful would be possibility to merge some category sets, like 'omit<br />Test, group the rest'.</li>
<li><strong>Keep.</strong></li>
</ul></li>
</ul>
</blockquote>
<p>While adding conditions like that is trivial, I am not sure how to represent that in the UI. The <code>Test</code> category is special, maybe have another view, sans all <code>Test</code> events? That could be useful for decisions around migrating Warden clients from test or how much effort to allocate to them.</p>
<p><a class="user active" href="https://homeproj.cesnet.cz/users/17">Pavel Kácha</a> wrote:</p>
<blockquote>
<ul>
<li>Countries
<ul>
<li>Same as ASNs.</li>
<li>Isn't this derivable from ASNs? (For stats precalculation.)</li>
<li><strong>Not sure.</strong></li>
</ul></li>
</ul>
</blockquote>
<p>This looks nice in presentations. Then, there were cases of geographic filtering on repackaging Warden sources in the past, so it might be (heavily) skewed, so the utility is questionable.</p>
<p><a class="user active" href="https://homeproj.cesnet.cz/users/17">Pavel Kácha</a> wrote:</p>
<blockquote>
<ul>
<li>Detectors
<ul>
<li>One of key measures.</li>
<li><strong>Keep.</strong></li>
</ul></li>
</ul>
<ul>
<li>Detector software
<ul>
<li>Similar to analyzers.</li>
<li>Moreover, most of the clients have only one SW behind (except Filers, which we don't care dearly for.)</li>
<li><strong>Ditch.</strong></li>
</ul></li>
</ul>
<ul>
<li>IPs
<ul>
<li>One of key measures.</li>
<li>Rename to Sources?</li>
<li><strong>Keep.</strong></li>
</ul></li>
</ul>
</blockquote>
<p>I agree with the rename, especially if the Target IP statistics would be added (and I agree with that).</p>
<p><a class="user active" href="https://homeproj.cesnet.cz/users/17">Pavel Kácha</a> wrote:</p>
<blockquote>
<ul>
<li>Classes
<ul>
<li>Mmmkay.</li>
<li>Not convinced strongly, but let's <strong>keep.</strong></li>
</ul></li>
</ul>
<ul>
<li>Severities
<ul>
<li>Also not sure, if useful in current form. Bars may show that in some<br />specific time there was a peak in recurrent events, alas there's no<br />possibility of finding out which ones (apart from trying to correlate<br />with columns on other graphs with possibly different scales and<br />drilling into Alerts searches).</li>
<li>Have anybody ever used this stats?</li>
<li><strong>Ditch?</strong></li>
</ul></li>
</ul>
</blockquote>
<p>This could be useful as a high level information in a manager graph somewhere.</p>
<p><a class="user active" href="https://homeproj.cesnet.cz/users/17">Pavel Kácha</a> wrote:</p>
<blockquote>
<p>We don't have:</p>
<ul>
<li>Ports
<ul>
<li>Would be nice for spotting rise of attacks to specific ports.</li>
<li><strong>Add?</strong></li>
</ul></li>
</ul>
</blockquote>
<p>I am for adding this.</p>
<p><a class="user active" href="https://homeproj.cesnet.cz/users/17">Pavel Kácha</a> wrote:</p>
<blockquote>
<ul>
<li>Services
<ul>
<li>(Un)fortunately it's partially orthogonal to ports.</li>
<li>Would be nice for spotting rise of attacks to specific services.</li>
<li><strong>Add?</strong></li>
</ul></li>
</ul>
</blockquote>
<p>True as the orthogonality argument goes, but might be useful also. And it could help with analysis if any discrepancies between ports and services exist, leading to improving the Warden clients and therefore the system as a whole.</p>
<p><a class="user active" href="https://homeproj.cesnet.cz/users/17">Pavel Kácha</a> wrote:</p>
<blockquote>
<ul>
<li>Targets
<ul>
<li>Doesn't it make sense for network admins to see their hotspots?</li>
<li><strong>Add?</strong></li>
</ul></li>
</ul>
</blockquote>
<p>Definitely.</p>
<p>As a whole, I see the statistics and timelines as a useful tool. Only changes I would ask for are to present relative counts to the whole, instead of inflating the total count, as was discussed. The DB based way of calculation is quite efficient, even for attributes with extreme value ranges (source/target ip arrays), there is a lot of room for expansion of this feature.</p> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=268202020-04-30T11:10:16ZPavel Káchaph@cesnet.cz
<ul><li><strong>Blocked by</strong> <i><a class="issue tracker-2 status-5 priority-4 priority-default closed" href="/issues/6308">Feature #6308</a>: Reimplement timeline calculations on database</i> added</li></ul> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=268212020-04-30T11:11:39ZPavel Káchaph@cesnet.cz
<ul></ul><p>Lets postpone discussion after finishing <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: Reimplement timeline calculations on database (Closed)" href="https://homeproj.cesnet.cz/issues/6308">#6308</a>, as it involves adding some columns to flat data table.</p> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=268472020-05-07T08:00:29ZPavel Káchaph@cesnet.cz
<ul><li><strong>To be discussed</strong> deleted (<del><i>Yes</i></del>)</li></ul> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=273052020-07-07T12:35:32ZRadko Krkoškrkos@cesnet.cz
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-5 priority-4 priority-default closed child" href="/issues/6413">Feature #6413</a>: Autarkic DB queries for timeline</i> added</li></ul> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=283212020-10-05T11:29:26ZRajmund Hruška
<ul></ul><p>Is there still any work to do? I see that the unnecessary statistics are removed and new statistics are added.</p> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=283362020-10-06T12:11:44ZPavel Káchaph@cesnet.cz
<ul></ul><p>If I did not miss anything, we dont have yet:</p>
<p>ASNs, Countries<br /> (Not sure.) Nice in presentations, however probably heavily skewed (because of sources of Warden data).</p>
<p>Categories<br /> Should be converted to Category sets, bigger task in itself.</p>
<p>We don't have yet: Services, Targets<br />(With it should go also renaming of IPs to Sources.)</p>
<p>All these additions would also need adding the column into database, ergo some planning and outage.</p>
<p>I'd either put this into Future, or split aforementioned into their own tickets.</p> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=363792022-08-16T08:32:44ZRadko Krkoškrkos@cesnet.cz
<ul><li><strong>Assignee</strong> changed from <i>Pavel Kácha</i> to <i>Jakub Maloštik</i></li><li><strong>Target version</strong> changed from <i>Backlog</i> to <i>2.11</i></li></ul><p>As per the <a href="#note-18">#note-18</a>, there are some parts left to implement. At least the Services and Targets should not be too difficult.</p> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=363952022-09-22T14:21:33ZPavel Káchaph@cesnet.cz
<ul></ul><p>Result from local discussion with Radko:</p>
<ul>
<li>Want Targets we <strong>do</strong>. (As master Yoda succinctly articulates). (And Targets in "metadata" events table already we have.)</li>
<li>Want ASN and Country we <strong>don't</strong> . (Yet.) They occupy space, are available only when Enricher is configured in the pipeline (and has correct data), are misleading (it's usually just jumphost, real attack does not in fact go from there), and if we really wanted, we might also consider loading Maxmind data into db table and do calculation on unnested/joined data. Let's drop those for now and leave them for if there is real demand or usecase.</li>
</ul> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=364122022-11-03T13:18:17ZRajmund Hruška
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Review</i></li></ul><p>Merged <a class="changeset" title="Renamed 'ips' statistics to 'sources' (Redmine issue: #6257)" href="https://homeproj.cesnet.cz/projects/mentat/repository/mentat-ng/revisions/730adabbd498583d6be2bdd4a0c189b3bd961c35">730adabb</a> and <a class="changeset" title="Feature: Add 'targets' section into timeline (Redmine issue: #6257)" href="https://homeproj.cesnet.cz/projects/mentat/repository/mentat-ng/revisions/ffdb1f6089da569a544a420349d492d6a2ebd3bd">ffdb1f60</a> into <strong>devel</strong>. The new version is deployed on mentat-alt.</p> Mentat - Feature #6257: Review calculated statistics in timelinehttps://homeproj.cesnet.cz/issues/6257?journal_id=365542023-04-06T11:26:59ZRajmund Hruška
<ul><li><strong>Status</strong> changed from <i>In Review</i> to <i>Closed</i></li></ul>