Feature #7577
closedImplement the notion of detector credibility and use it for reporting
Added by Pavel Kácha over 2 years ago. Updated over 1 year ago.
90%
Description
Some detectors are a bit unreliable and sometimes report false positives. We might consider reporting events from them only if more reliable detector hits, or more of less reliable detectors hit the same IP.
Note that detectors, which produce more types of events, might exhibit different levels of reliability for different types.
Possible solutions:
- mark the reliability in the inspector
- pro: allows for high granularity (not only detector)
- con: not in line with Warden notion of detectors
- implement db of detectors within Mentat
- con: needs some (semi)manual means of keeping in line with Warden
- implement full Warden client management
- pro: one source of truth
- con: needs to implement the API on the Warden side
- where and how to store the credibility information (or rules) - Warden or Mentat side? One value or more granular rules?
- how to combine the values for reporting - two level, linear/nonlinear combination..?
Updated by Pavel Kácha over 2 years ago
From discussion 26. 7. 2022:
- own client db (plus relatively independent import script, similar to network info from Negistry)
- maybe in the future - plugin for Warden clietnt management
- From Warden we need
- widening of client info – let's start simply - big float
- we need API for getting client info
- later maybe for management
- clients need new permission bit for client info management
Updated by Pavel Kácha over 2 years ago
From discussion 2. 8. 2022:
- Let's start lightly
- New db for client data - id, name, desc, credibility
- (Generated database is not a good point for reusing, as it server different purpose – set of client names from history for search forms is quite different from present set of clients.)
- New view for clients, similar to Users, Groups, Networks, etc.
Updated by Rajmund Hruška about 2 years ago
- Status changed from New to Feedback
- Target version changed from Backlog to 2.11
- % Done changed from 0 to 10
- To be discussed changed from No to Yes
Pavel Kácha wrote in #note-3:
From discussion 2. 8. 2022:
- Let's start lightly
- New db for client data - id, name, desc, credibility
- (Generated database is not a good point for reusing, as it server different purpose – set of client names from history for search forms is quite different from present set of clients.)
- New view for clients, similar to Users, Groups, Networks, etc.
I finished this part of the assignment. I also put source
attribute to the detectors table and implemented a view for updating detectors. This got me thinking that updating detectors in Warden can be done using this view. When the information about the detectors will be fetched, the new detectors will be created with source Warden. Then, either after updating a detector or periodically, when import script is executed, the updated information will be sent to Warden.
Oh, and just to clarify - are client and detector the same thing? Or is it like cz.cesnet.nemea.filer
and cz.cesnet.nemea.bruteforce
are two detectors of the client cz.cesnet.nemea
?
Updated by Rajmund Hruška about 2 years ago
I checked reports to see which source IP addresses were detected by more than one detector.
In the last ~300 reports, it has happened a few times.
https://mentat-hub.cesnet.cz/mentat/reports/196276/show
https://mentat-hub.cesnet.cz/mentat/reports/196252/show
https://mentat-hub.cesnet.cz/mentat/reports/196231/show
https://mentat-hub.cesnet.cz/mentat/reports/196172/show
https://mentat-hub.cesnet.cz/mentat/reports/196159/show
https://mentat-hub.cesnet.cz/mentat/reports/196132/show
https://mentat-hub.cesnet.cz/mentat/reports/196094/show
https://mentat-hub.cesnet.cz/mentat/reports/196099/show
https://mentat-hub.cesnet.cz/mentat/reports/196062/show
https://mentat-hub.cesnet.cz/mentat/reports/196063/show
https://mentat-hub.cesnet.cz/mentat/reports/196012/show
https://mentat-hub.cesnet.cz/mentat/reports/196013/show
https://mentat-hub.cesnet.cz/mentat/reports/196006/show
Updated by Rajmund Hruška about 2 years ago
So, I checked the events detected in the last 24 hours.
What happens frequently is an event being detected by cz.cesnet.nemea
and cz.casablanca.nemea
Quite a common is an event being detected by both cz.cesnet.tmc.log_analyzer
and cz.cesnet.nemea.hoststats
https://mentat-alt.cesnet.cz/mentat/events/6efdb99e-cb4f-485f-adb4-eb9d01b7862e/show
https://mentat-alt.cesnet.cz/mentat/events/3455da72-d9f3-4bb0-bb91-4136cb104469/show
I saw one event being detected by cz.cesnet.nemea.hoststats
and cz.cesnet.gc15
.
https://mentat-alt.cesnet.cz/mentat/events/d5c8c4d5-93d5-4487-83b5-466de3f2951b/show
https://mentat-alt.cesnet.cz/mentat/events/1663564515_gc15.cesnet.cz_1212_src_ip_13.67.114.156_32/show
Updated by Rajmund Hruška about 2 years ago
Rajmund Hruska wrote in #note-6:
So, I checked the events detected in the last 24 hours.
What happens frequently is an event being detected by
cz.cesnet.nemea
andcz.casablanca.nemea
Quite a common is an event being detected by both
cz.cesnet.tmc.log_analyzer
andcz.cesnet.nemea.hoststats
https://mentat-alt.cesnet.cz/mentat/events/6efdb99e-cb4f-485f-adb4-eb9d01b7862e/show
https://mentat-alt.cesnet.cz/mentat/events/3455da72-d9f3-4bb0-bb91-4136cb104469/showI saw one event being detected by
cz.cesnet.nemea.hoststats
andcz.cesnet.gc15
.
https://mentat-alt.cesnet.cz/mentat/events/d5c8c4d5-93d5-4487-83b5-466de3f2951b/show
https://mentat-alt.cesnet.cz/mentat/events/1663564515_gc15.cesnet.cz_1212_src_ip_13.67.114.156_32/show
And it's pretty much the same for 72 hours window.
Updated by Pavel Kácha about 2 years ago
Thanks, Rajmund. So I guess we may have a tad of overengineering.
Mainly it means we don't have to hurry with the second phase of the plan – probabilities and calculated decision making in reporter as naive algorithm will be enough for some time.
Updated by Rajmund Hruška about 2 years ago
- % Done changed from 10 to 20
I am currently working on importing detectors data from Warden. Right now, I am sending everything from the database, as it can be seen here: https://hruska.vm.cesnet.cz/warden3/getClients?secret=test01. I am not sure if Mentat should be able to get everything or only a subset of client information.
Updated by Rajmund Hruška about 2 years ago
- % Done changed from 20 to 50
I merged the current work to devel and deployed on mentat-alt. The detectors can be accessed at https://mentat-alt.cesnet.cz/mentat/detectors/list.
Updated by Rajmund Hruška almost 2 years ago
- Status changed from Feedback to In Progress
- To be discussed deleted (
Yes)
- add hits (similar to filters) and the date when was the client created to detectors management
- when reporting events, discard events from detectors with credibility lower than 0.5
- inform about the discarded events in the reports
Updated by Rajmund Hruška almost 2 years ago
- % Done changed from 50 to 90
Rajmund Hruška wrote in #note-11:
So based on today's meeting:
- add hits (similar to filters) and the date when was the client created to detectors management
- when reporting events, discard events from detectors with credibility lower than 0.5
- inform about the discarded events in the reports
I added the hits column and I wrote the code which handles discarding events from reports.
Currently, the reports have this type of information:
Event counts: 54 reported (458 matched, 413 new events, 0 filtered out, 404 thresholded, 45 relapsed)
To add the information about low credibility, like this:
Event counts: 44 reported (458 matched, 413 new events, 0 filtered out, 10 uncredible, 404 thresholded, 45 relapsed)
a database migration of reports is required. It's about 200000 records for each mentat-hub and mentat-alt, and 2 integer columns will be added. I think it should be alright.
Updated by Rajmund Hruška almost 2 years ago
Rajmund Hruška wrote in #note-12:
Rajmund Hruška wrote in #note-11:
So based on today's meeting:
- add hits (similar to filters) and the date when was the client created to detectors management
- when reporting events, discard events from detectors with credibility lower than 0.5
- inform about the discarded events in the reports
I added the hits column and I wrote the code which handles discarding events from reports.
Currently, the reports have this type of information:
[...]To add the information about low credibility, like this:
[...]a database migration of reports is required. It's about 200000 records for each mentat-hub and mentat-alt, and 2 integer columns will be added. I think it should be alright.
This was done in 78bc2d3b.
I would like to add some more tests. It seems to be working well on my local machine though.
Updated by Rajmund Hruška over 1 year ago
- Status changed from In Progress to Closed