Project

General

Profile

Actions

Bug #4489

closed

Fix filtering mechanism in reporter

Added by Jan Mach over 5 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
High
Assignee:
Jan Žerdík
Category:
Development - Core
Target version:
Start date:
01/03/2019
Due date:
% Done:

0%

Estimated time:
To be discussed:

Description

Filtering mechanism in reporter has a flaw, that can cause that events that should be filtered out still make it to the final report. This is most probably due to the fact, that single IDEA event can contain multiple sources and it passes filtering in case there is any source, that should not be filtered out. This of course needs to be corrected, perhaps by adding secondary filter before rendering the appropriate report line.

This bug should be resolved promptly, because it was reported a while ago.

Actions #1

Updated by Pavel Kácha over 5 years ago

Note: On resolution, notify recipients of Message-Id: <>.

Actions #2

Updated by Jan Mach almost 5 years ago

  • Target version changed from 2.4 to 2.5
Actions #3

Updated by Jan Mach over 4 years ago

  • Target version changed from 2.5 to 2.6
Actions #4

Updated by Pavel Kácha over 4 years ago

  • Assignee changed from Jan Mach to Jan Žerdík

After discussion it's obvious secondary filter will not suffice. In fact, only part of the event (containing IP address) is matched, and we should not report it, but we should report the rest. Possible solutions:

1. Learn PySpect to actually search (along with reporting found positions in events), instead of only matching. However, JŽ correctly noted, that more complex conditions would need to return inconveniently complex data pointers or results, which would be also hard to interpret.
2. Cleanup rules, convert advanced to simple where feasible, solve for simple rules, not for advanced.
3. Consider widening simple rule possibilities, so we can convert more from advanced.
4. Match event, than match each unfiltered event for all sources separately (with respective Source.?.IP? replaced with one item array), so we know exactly for which source it matched and omit this source from report.

Seems like 4 is winner in terms of ROI. Maybe it will add some overhead, however if we don't allow Blabla[x] rules (which don't make much sense anyway), we will have 100% covered.

Actions #5

Updated by Jan Mach over 4 years ago

  • To be discussed changed from No to Yes
Actions #6

Updated by Pavel Kácha over 4 years ago

  • To be discussed changed from Yes to No
Actions #7

Updated by Pavel Kácha over 4 years ago

Mek, could you please look up vhor's example?

Actions #8

Updated by Jan Mach over 4 years ago

  • Target version changed from 2.6 to 2.7
Actions #9

Updated by Jan Žerdík over 4 years ago

  • To be discussed changed from No to Yes

After more testing... The filtering does not let through sources that do match filter. Example that Mek send me was badly written when it checked equality with subnet instead of "IN" test.
On the contrary, filter does not allow event through when at least one source match because of implementation of "OP_IN" operation in pynspect BaseFilteringTreeTraverser. For example, if Source.IP4 return [192.168.0.1, 192.168.1.1] and filter test is "Source.IP4 in [192.168.0.0/24]" than event match and is filtered out.
Should we flip discussed solution and recheck sources if filter match (even if that is not really subject of this ticket)?

Actions #10

Updated by Pavel Kácha over 4 years ago

Yup, we need to fix this also. From video - fixes will go to Mentat-alt for at least two weeks of testing.

Actions #11

Updated by Pavel Kácha over 4 years ago

  • To be discussed changed from Yes to No
Actions #12

Updated by Pavel Kácha about 4 years ago

  • To be discussed changed from No to Yes
Actions #13

Updated by Pavel Kácha about 4 years ago

  • To be discussed changed from Yes to No
Actions #14

Updated by Jan Žerdík about 4 years ago

  • To be discussed changed from No to Yes
Actions #15

Updated by Pavel Kácha about 4 years ago

So needs some testing on alt (artifical events).

Actions #16

Updated by Pavel Kácha about 4 years ago

  • To be discussed deleted (Yes)
Actions #17

Updated by Jan Žerdík almost 4 years ago

  • Status changed from New to Closed

After some time and testing, everything looks fine.

Actions

Also available in: Atom PDF