Feature #4214
closedNested rule chains in Inspector
Added by Pavel Kácha over 6 years ago. Updated almost 4 years ago.
0%
Description
Extend inspector to support nested rule chains. Needs thinking out and clarifying, however this would help to simplify the daemon queue for some setups (one inspector instance instead of couple).
Files
| inspector_profile.txt (2.59 KB) inspector_profile.txt | One inspector with combined rules for 1000 events | Rajmund Hruška, 01/04/2021 02:03 PM | |
| inspector_profile_2.txt (5.14 KB) inspector_profile_2.txt | Two inspectors with different rules for 1000 events | Rajmund Hruška, 01/04/2021 02:03 PM | |
| inspector_profile_10000.txt (2.6 KB) inspector_profile_10000.txt | One inspector with combined rules for 10000 events | Rajmund Hruška, 01/04/2021 02:34 PM | |
| inspector_profile_2_10000.txt (5.09 KB) inspector_profile_2_10000.txt | Two inspectors with different rules for 10000 events | Rajmund Hruška, 01/04/2021 02:34 PM | |
| inspector_profile_2_tottime.txt (5.42 KB) inspector_profile_2_tottime.txt | Two inspectors, 1000 events | Rajmund Hruška, 01/12/2021 12:24 PM | |
| inspector_profile_tottime.txt (2.72 KB) inspector_profile_tottime.txt | One inspector, 1000 events | Rajmund Hruška, 01/12/2021 12:24 PM |
Related issues
Updated by Rajmund Hruška about 4 years ago
- Assignee changed from Jan Mach to Rajmund Hruška
Updated by Pavel Kácha almost 4 years ago
Inspector manages chain of boolean rules and accompanying actions. However, if some rule needs decision/action by different rule, we have to chain more Inspectors. And we do - we use first one for assigning event class (https://csirt.cesnet.cz/cs/services/eventclass) and second one for checking event attributes based on already assigned class. The aim of this issue is to be able to reflect this within one inspector/configuration and get rid of additional instances. Sure, additional instances can help with performance, however we can always run more same parallel instances.
Possibilities for consideration from video meeting:- Nested rules
rule/action subrule/action rule/action subrule/action subsubrule/action subrule/action - Independent serial chains
Chain 1 rule/action rule/action Chain 2 rule/action
- Nested chains by reference (reusable chains, as in for example iptables)
Chain Initial rule/action rule/action, call ReusableChain ReusableChain rule/action, call ThirdChain rule/action ThirdChain ...
Updated by Rajmund Hruška almost 4 years ago
- Status changed from New to Feedback
- Assignee changed from Rajmund Hruška to Pavel Kácha
If I understand correctly, currently there are two instances of mentat-inspector with diffent sets of rules. However, the rules from mentat-inspector-b are not dependent on attributes set by the first mentat-inspector. I tried to append the rules from the mentat-inspector-b to the rules of the first mentat-inspector, I removed the mentat-inspector-b from the pipe and it seems to be working. So this feature (nested rule chains) is not necessary to get rid of multiple instances of mentat-inspector.
Updated by Rajmund Hruška almost 4 years ago
- File inspector_profile.txt inspector_profile.txt added
- File inspector_profile_2.txt inspector_profile_2.txt added
- File inspector_profile_2_10000.txt inspector_profile_2_10000.txt added
- File inspector_profile_10000.txt inspector_profile_10000.txt added
Based on the phone call, I tried to run cProfile. I checked cbk_event_message_process from lib/mentat/daemon/component/inspector.py and I used 1000 and 10000 IDEA events so I am not sure if I did it right. Anyway, from what I can see, most of the time is spent in the pynspect module.
Updated by Pavel Kácha almost 4 years ago
- Assignee changed from Pavel Kácha to Rajmund Hruška
Rajmund Hruska wrote in #note-7:
Based on the phone call, I tried to run cProfile. I checked
cbk_event_message_processfrom lib/mentat/daemon/component/inspector.py and I used 1000 and 10000 IDEA events so I am not sure if I did it right. Anyway, from what I can see, most of the time is spent in thepynspectmodule.
Which is expected result. However cumtime toplist does not say much, it times call of routine including all subroutines. What we would like to know is what is the hotspot, the bottleneck. If we end up in PLY, there's nothing we can do, if we end up somewhere in our code, there might be. So more interesting would probably be tottime or percall (excluding subroutines).
Also, interesting might be the simple throughput (how many events are we able to process, or how quick is one event processing), compared to other daemons - so we don't chase for optimisation with negligible effect.
Updated by Rajmund Hruška almost 4 years ago
Pavel Kácha wrote in #note-8:
I measured the average time it takes to process one file (methodAlso, interesting might be the simple throughput (how many events are we able to process, or how quick is one event processing), compared to other daemons - so we don't chase for optimisation with negligible effect.
cbk_event_message_process) while processing 10000 events:
- inspector: 0.00205 s
- inspector-b: 0.00205 s
- enricher: 0.00683 s
- storage: 0.00519 s
- inspector (just 1 with concatenated rules): 0.00482 s
Based on that it seems that the bottleneck is enricher module.
Updated by Rajmund Hruška almost 4 years ago
- File inspector_profile_2_tottime.txt inspector_profile_2_tottime.txt added
- File inspector_profile_tottime.txt inspector_profile_tottime.txt added
Pavel Kácha wrote in #note-8:
Which is expected result. However cumtime toplist does not say much, it times call of routine including all subroutines. What we would like to know is what is the hotspot, the bottleneck. If we end up in PLY, there's nothing we can do, if we end up somewhere in our code, there might be. So more interesting would probably be tottime or percall (excluding subroutines).
I made a profile sorted by tottime of inspector with one set of rules and two inspectors for 1000 events, but to be honest, I don't know how to interpret the result. However, based on my previous reply #note-9, I think we can just use one inspector with one set of rules.
Updated by Rajmund Hruška almost 4 years ago
- Assignee changed from Rajmund Hruška to Pavel Kácha
Updated by Pavel Kácha almost 4 years ago
- Assignee changed from Pavel Kácha to Rajmund Hruška
Rajmund Hruska wrote in #note-10:
Pavel Kácha wrote in #note-8:
Which is expected result. However cumtime toplist does not say much, it times call of routine including all subroutines. What we would like to know is what is the hotspot, the bottleneck. If we end up in PLY, there's nothing we can do, if we end up somewhere in our code, there might be. So more interesting would probably be tottime or percall (excluding subroutines).
I made a profile sorted by tottime of inspector with one set of rules and two inspectors for 1000 events, but to be honest, I don't know how to interpret the result. However, based on my previous reply #note-9, I think we can just use one inspector with one set of rules.
Tottime is the time spent in the function alone, without subcalls. So in my opinion it shows best where the most of the time is spent. So it might make sense to look to the routines at the top of the trace (in the pynspect), whether there's some low hanging fruit for optimization, or some apparent culprit (too much copying? "extend" instead of "for ... append"? less isinstance calls?). If not, we still have another options.
Updated by Pavel Kácha almost 4 years ago
Rajmund Hruska wrote in #note-9:
Pavel Kácha wrote in #note-8:
I measured the average time it takes to process one file (methodAlso, interesting might be the simple throughput (how many events are we able to process, or how quick is one event processing), compared to other daemons - so we don't chase for optimisation with negligible effect.
cbk_event_message_process) while processing 10000 events:
- inspector: 0.00205 s
- inspector-b: 0.00205 s
- enricher: 0.00683 s
- storage: 0.00519 s
- inspector (just 1 with concatenated rules): 0.00482 s
Based on that it seems that the bottleneck is enricher module.
Yes. However, concatenated inspector is within 10% from storage speed, which is not much breathing space.
We will need rules for checking specific classes (if class is this, we want to check that key, that key and another key). In linear concatenated ruleset that means that these will apply to each and every event regardless the class.
We will most probably need another filtered external targets like Protective (inspector-c), which currently uses 'final' clause, which might be a problem in linear concatenated ruleset. Rules in inspector-c might be possible to rewrite in one humongous condition, however... readability counts.
But these are still only my views, open to discussion, Radko, Mek, Rajmund, any opinion on that?
Updated by Pavel Kácha almost 4 years ago
- Related to Bug #6948: Pynspect might take advantage of some optimizations added
Updated by Pavel Kácha almost 4 years ago
- Related to Config #6949: Merge inspector configs into one added
Updated by Pavel Kácha almost 4 years ago
- Status changed from Feedback to Deferred
Updated by Pavel Kácha almost 4 years ago
- Target version changed from Backlog to 2.8
Updated by Pavel Kácha almost 4 years ago
- Status changed from Deferred to Rejected