Project

General

Profile

Actions
Copy link

Feature #7577

closed

Implement the notion of detector credibility and use it for reporting

Added by Pavel Kácha about 2 years ago. Updated 10 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Rajmund Hruška
Category:
Development - Core
Target version:
2.11
Start date:
03/23/2022
Due date:
% Done:

90%

Estimated time:
To be discussed:

Description

Some detectors are a bit unreliable and sometimes report false positives. We might consider reporting events from them only if more reliable detector hits, or more of less reliable detectors hit the same IP.
Note that detectors, which produce more types of events, might exhibit different levels of reliability for different types.

Possible solutions:

  • mark the reliability in the inspector
    • pro: allows for high granularity (not only detector)
    • con: not in line with Warden notion of detectors
  • implement db of detectors within Mentat
    • con: needs some (semi)manual means of keeping in line with Warden
  • implement full Warden client management
    • pro: one source of truth
    • con: needs to implement the API on the Warden side
Questions:
  • where and how to store the credibility information (or rules) - Warden or Mentat side? One value or more granular rules?
  • how to combine the values for reporting - two level, linear/nonlinear combination..?
Actions
Copy link
 #1

Updated by Rajmund Hruška almost 2 years ago

  • Assignee set to Rajmund Hruška
Actions
Copy link
 #2

Updated by Pavel Kácha over 1 year ago

From discussion 26. 7. 2022:

  • own client db (plus relatively independent import script, similar to network info from Negistry)
  • maybe in the future - plugin for Warden clietnt management
  • From Warden we need
    • widening of client info – let's start simply - big float
    • we need API for getting client info
    • later maybe for management
    • clients need new permission bit for client info management
Actions
Copy link
 #3

Updated by Pavel Kácha over 1 year ago

From discussion 2. 8. 2022:

  • Let's start lightly
    • New db for client data - id, name, desc, credibility
    • (Generated database is not a good point for reusing, as it server different purpose – set of client names from history for search forms is quite different from present set of clients.)
    • New view for clients, similar to Users, Groups, Networks, etc.
Actions
Copy link
 #4

Updated by Rajmund Hruška over 1 year ago

  • Status changed from New to Feedback
  • Target version changed from Backlog to 2.11
  • % Done changed from 0 to 10
  • To be discussed changed from No to Yes

Pavel Kácha wrote in #note-3:

From discussion 2. 8. 2022:

  • Let's start lightly
    • New db for client data - id, name, desc, credibility
    • (Generated database is not a good point for reusing, as it server different purpose – set of client names from history for search forms is quite different from present set of clients.)
    • New view for clients, similar to Users, Groups, Networks, etc.

I finished this part of the assignment. I also put source attribute to the detectors table and implemented a view for updating detectors. This got me thinking that updating detectors in Warden can be done using this view. When the information about the detectors will be fetched, the new detectors will be created with source Warden. Then, either after updating a detector or periodically, when import script is executed, the updated information will be sent to Warden.

Oh, and just to clarify - are client and detector the same thing? Or is it like cz.cesnet.nemea.filer and cz.cesnet.nemea.bruteforce are two detectors of the client cz.cesnet.nemea?

Actions
Copy link
 #6

Updated by Rajmund Hruška over 1 year ago

So, I checked the events detected in the last 24 hours.

What happens frequently is an event being detected by cz.cesnet.nemea and cz.casablanca.nemea

Quite a common is an event being detected by both cz.cesnet.tmc.log_analyzer and cz.cesnet.nemea.hoststats
https://mentat-alt.cesnet.cz/mentat/events/6efdb99e-cb4f-485f-adb4-eb9d01b7862e/show
https://mentat-alt.cesnet.cz/mentat/events/3455da72-d9f3-4bb0-bb91-4136cb104469/show

I saw one event being detected by cz.cesnet.nemea.hoststats and cz.cesnet.gc15.
https://mentat-alt.cesnet.cz/mentat/events/d5c8c4d5-93d5-4487-83b5-466de3f2951b/show
https://mentat-alt.cesnet.cz/mentat/events/1663564515_gc15.cesnet.cz_1212_src_ip_13.67.114.156_32/show

Actions
Copy link
 #7

Updated by Rajmund Hruška over 1 year ago

Rajmund Hruska wrote in #note-6:

So, I checked the events detected in the last 24 hours.

What happens frequently is an event being detected by cz.cesnet.nemea and cz.casablanca.nemea

Quite a common is an event being detected by both cz.cesnet.tmc.log_analyzer and cz.cesnet.nemea.hoststats
https://mentat-alt.cesnet.cz/mentat/events/6efdb99e-cb4f-485f-adb4-eb9d01b7862e/show
https://mentat-alt.cesnet.cz/mentat/events/3455da72-d9f3-4bb0-bb91-4136cb104469/show

I saw one event being detected by cz.cesnet.nemea.hoststats and cz.cesnet.gc15.
https://mentat-alt.cesnet.cz/mentat/events/d5c8c4d5-93d5-4487-83b5-466de3f2951b/show
https://mentat-alt.cesnet.cz/mentat/events/1663564515_gc15.cesnet.cz_1212_src_ip_13.67.114.156_32/show

And it's pretty much the same for 72 hours window.

Actions
Copy link
 #8

Updated by Pavel Kácha over 1 year ago

Thanks, Rajmund. So I guess we may have a tad of overengineering.

Mainly it means we don't have to hurry with the second phase of the plan – probabilities and calculated decision making in reporter as naive algorithm will be enough for some time.

Actions
Copy link
 #9

Updated by Rajmund Hruška over 1 year ago

  • % Done changed from 10 to 20

I am currently working on importing detectors data from Warden. Right now, I am sending everything from the database, as it can be seen here: https://hruska.vm.cesnet.cz/warden3/getClients?secret=test01. I am not sure if Mentat should be able to get everything or only a subset of client information.

Actions
Copy link
 #10

Updated by Rajmund Hruška over 1 year ago

  • % Done changed from 20 to 50

I merged the current work to devel and deployed on mentat-alt. The detectors can be accessed at https://mentat-alt.cesnet.cz/mentat/detectors/list.

Actions
Copy link
 #11

Updated by Rajmund Hruška over 1 year ago

  • Status changed from Feedback to In Progress
  • To be discussed deleted (Yes)
So based on today's meeting:
  • add hits (similar to filters) and the date when was the client created to detectors management
  • when reporting events, discard events from detectors with credibility lower than 0.5
  • inform about the discarded events in the reports
Actions
Copy link
 #12

Updated by Rajmund Hruška over 1 year ago

  • % Done changed from 50 to 90

Rajmund Hruška wrote in #note-11:

So based on today's meeting:
  • add hits (similar to filters) and the date when was the client created to detectors management
  • when reporting events, discard events from detectors with credibility lower than 0.5
  • inform about the discarded events in the reports

I added the hits column and I wrote the code which handles discarding events from reports.

Currently, the reports have this type of information:

Event counts: 54 reported (458 matched, 413 new events, 0 filtered out, 404 thresholded, 45 relapsed)

To add the information about low credibility, like this:

Event counts: 44 reported (458 matched, 413 new events, 0 filtered out, 10 uncredible, 404 thresholded, 45 relapsed)

a database migration of reports is required. It's about 200000 records for each mentat-hub and mentat-alt, and 2 integer columns will be added. I think it should be alright.

Actions
Copy link
 #13

Updated by Rajmund Hruška over 1 year ago

Rajmund Hruška wrote in #note-12:

Rajmund Hruška wrote in #note-11:

So based on today's meeting:
  • add hits (similar to filters) and the date when was the client created to detectors management
  • when reporting events, discard events from detectors with credibility lower than 0.5
  • inform about the discarded events in the reports

I added the hits column and I wrote the code which handles discarding events from reports.

Currently, the reports have this type of information:
[...]

To add the information about low credibility, like this:
[...]

a database migration of reports is required. It's about 200000 records for each mentat-hub and mentat-alt, and 2 integer columns will be added. I think it should be alright.

This was done in 78bc2d3b.

I would like to add some more tests. It seems to be working well on my local machine though.

Actions
Copy link
 #14

Updated by Rajmund Hruška 10 months ago

  • Status changed from In Progress to Closed
Actions
Copy link

Also available in: Atom PDF