Fix filtering mechanism in reporter
Filtering mechanism in reporter has a flaw, that can cause that events that should be filtered out still make it to the final report. This is most probably due to the fact, that single IDEA event can contain multiple sources and it passes filtering in case there is any source, that should not be filtered out. This of course needs to be corrected, perhaps by adding secondary filter before rendering the appropriate report line.
This bug should be resolved promptly, because it was reported a while ago.
Updated by Pavel Kácha 8 months ago
- Assignee changed from Jan Mach to Jan Žerdík
After discussion it’s obvious secondary filter will not suffice. In fact, only part of the event (containing IP address) is matched, and we should not report it, but we should report the rest. Possible solutions:
1. Learn PySpect to actually search (along with reporting found positions in events), instead of only matching. However, JŽ correctly noted, that more complex conditions would need to return inconveniently complex data pointers or results, which would be also hard to interpret.
2. Cleanup rules, convert advanced to simple where feasible, solve for simple rules, not for advanced.
3. Consider widening simple rule possibilities, so we can convert more from advanced.
4. Match event, than match each unfiltered event for all sources separately (with respective Source.?.IP? replaced with one item array), so we know exactly for which source it matched and omit this source from report.
Seems like 4 is winner in terms of ROI. Maybe it will add some overhead, however if we don’t allow Blabla[x] rules (which don’t make much sense anyway), we will have 100% covered.
Updated by Jan Žerdík 6 months ago
- To be discussed changed from No to Yes
After more testing... The filtering does not let through sources that do match filter. Example that Mek send me was badly written when it checked equality with subnet instead of “IN” test.
On the contrary, filter does not allow event through when at least one source match because of implementation of “OP_IN” operation in pynspect BaseFilteringTreeTraverser. For example, if Source.IP4 return [192.168.0.1, 192.168.1.1] and filter test is “Source.IP4 in [192.168.0.0/24]” than event match and is filtered out.
Should we flip discussed solution and recheck sources if filter match (even if that is not really subject of this ticket)?