Mentat

If we need to send reports to more recipients (To, Cc), we have a clash in reporting preferences (which takes precedence and how), and also in working with summary reports.

Settings adaptation gets tracked in #6239.

After discussion, summary reports can be solved by a bit different splitting. We now have nonoverlapping abuse contact resolution, so let's create summary reports for the most specific networks. That may mean more summaries, however only for handful of organisations, for whom we have more specific data - and these will most probably benefit from it. Also, it is possible to "merge" summaries across boundaries in overlapping networks (GRID).

• we need new config key which will get these stray reports
• error line in report (thus in template)

• let it also go to stray report contact with error line

Needs solution in Negistry (to not allow "unsolvable" situations).

From the 2021-01-22 meeting:

• event id, ip+class (or sort+concat category), group id, timestamp
• record for each event, so we can just join in tresholded events in case of relapse

• fetch the list of related events for the reporting window, then apply reporting cycle only for affected groups (result should be still the same as before)
  • after the meeting Radko correctly noted that fetching all the events is not the good idea from the memory standpoint (Python does not return back), and we already have the abuse field parsed out in the metatable, so we might just select ... distinct abuse groups for the reporting time window and the rest of algorith stays the same)
• then make changes necessary to accomodate negistry (more granular abuse contact info in Enricher) and change the algorithm to iterate over those more specific sets of abuse contact

However - due to personal changes the #6239 haven't reached devel correctly, co we need to merge d282e3a0 (and nothing after) and work on top of that.

• Currently, my solution assigns emails to cc header after filtering the events for the most specific group, just before the email is sent. However, determining groups in cc header should be done before the filtering, so that some groups won't get events which would be otherwise filtered out.
• The database should store each group to which the report is sent. This is tracked in a separate issue #7052

Rajmund Hruska wrote in #note-18:

From the 2021-03-05 call:
My idea was to save the network hierarchy in the database. It turns out that Mentat supports the hierarchy of groups rather than the hierarchy of the networks. So, I will find a different approach to save necessary information needed for more specific reporting based on the severity.

Please, take look whether the hierarchy could be created from the netblock subset/superset relations. I have theorized about using Organisation field from RIPE, however this field will is not set for non-RIPE feeds, and also it does not say anything about parent/child block relation, as I originally thought.

Also - careful with regenerating the db. Groups do have associated a lot of settings, which would be lost. Deleting and reassigning only the networks db might be ok, though. (However, I would be careful and compare the result for possible differences or anomalies.)

I made a script which compares resolved abuses for IP addresses from negistry-multifeed-abuses-segmented-ipranges.json and from new Mentat resolving. I ran across a problem of having two almost identical networks, which only differ in the rank and the feed. I thought there was some problem in having both records in the database, but it seems that it's actually ok, so I will change importer to allow this.

Other than that, there is a broken unit test regarding reporting, but current version allows sending reports to multiple groups and takes rank into consideration. The only feature left is considering severity level in Negistry data for reporting.

Rajmund Hruska wrote in #note-23:

I ran the comparing script again and I found out that there is still quite an issue. Look at these records from negistry-multifeed-abuses-full-ipranges.json:

• feed: cuni_subnets
• rank: 500
• primary_key: 195.113.149.128 - 195.113.149.131
• resolved_abuses: {low: vhor@cuni.cz}

• feed: ripe_cesnet
• rank: 100
• primary_key: 195.113.149.128 - 195.113.149.131
• resolved_abuses: {low: abuse@cuni.cz}

The resolved abuse from cuni_subnets feed should be in 'abuse_to' header and the other one should be in 'abuse_cc' header, but in negistry-multifeed-abuses-segmented-ipranges.json it's opposite. Isn't there a mistake in that segmented file?

Is this really so? Excerpt from negistry-multifeed-abuses-segmented-ipranges.json:

{
    "ip4_start": "195.113.149.128", 
    "ip4_end": "195.113.149.131", 
    "cidrs": [
      "195.113.149.128/30" 
    ], 
    "first": 3279000960, 
    "last": 3279000963, 
    "ranges": [
      {
        "feed": "cuni_subnets", 
        "primary_key": "195.113.149.128 - 195.113.149.131" 
      }, 
      {
        "feed": "ripe_cesnet", 
        "primary_key": "195.113.149.128 - 195.113.149.131" 
      }, 
      {
        "feed": "ripe_cesnet", 
        "primary_key": "195.113.0.0 - 195.113.255.255" 
      }
    ], 
    "abuses": {
      "high": {
        "cc": [
          "abuse@cuni.cz", 
          "abuse@cesnet.cz" 
        ], 
        "to": [
          "vhor@cuni.cz" 
        ]
      }, 
      "medium": {
        "cc": [
          "abuse@cuni.cz", 
          "abuse@cesnet.cz" 
        ], 
        "to": [
          "vhor@cuni.cz" 
        ]
      }, 
      "critical": {
        "cc": [
          "abuse@cuni.cz", 
          "abuse@cesnet.cz" 
        ], 
        "to": [
          "vhor@cuni.cz" 
        ]
      }, 
      "low": {
        "cc": [
          "abuse@cuni.cz", 
          "abuse@cesnet.cz" 
        ], 
        "to": [
          "vhor@cuni.cz" 
        ]
      }
    }
  }, 

vhor is in To (from cuni_subnets), then abuse@cuni and abuse@cesnet (from ripe_cesnet).

Oh, so the file attached to this issue is probably outdated, because here is the record I was referring to.

  {
    "ip4_start": "195.113.149.128", 
    "ip4_end": "195.113.149.131", 
    "cidrs": [
      "195.113.149.128/30" 
    ], 
    "first": 3279000960, 
    "last": 3279000963, 
    "feeds": [
      "ripe_cesnet", 
      "cuni_subnets", 
      "ripe_cesnet" 
    ], 
    "ranks": [
      100, 
      500, 
      100
    ], 
    "primary_keys": [
      "195.113.149.128 - 195.113.149.131", 
      "195.113.149.128 - 195.113.149.131", 
      "195.113.0.0 - 195.113.255.255" 
    ], 
    "netnames": [
      [
        "CUNI-UJOP-TCZ" 
      ], 
      [
        "UJOP-PODEBRADY" 
      ], 
      [
        "CZ-TEN-34-970317" 
      ]
    ], 
    "client_ids": [
      "C010000", 
      null, 
      "A010000" 
    ], 
    "abuse_to": {
      "low": [
        "abuse@cuni.cz" 
      ]
    }, 
    "abuse_cc": [
      {
        "low": [
          "vhor@cuni.cz" 
        ]
      }, 
      {
        "low": [
          "abuse@cesnet.cz" 
        ]
      }
    ]
  }

Rajmund Hruska wrote in #note-23:

I made a change which allows having multiple networks with the same primary key. Because of that, I have changed the way how source in networks table is calculated. Now, if network from the input file contains feed attribute, then this attribute becomes the part of the source of that particular network (e.g. whois/grid_devices).

I realized that this change will make every network in the current database (from old Negistry) different from the networks from the new Negistry even with the same primary key, because the source attribute is different. Networks in the database don't have special indispensable attributes so I think it's OK to create them anew. I can try to think of another way of solving this though, if necessary.

I fixed the issue with some events being filtered out even though they shouldn't be - it was the same issue as in #4489. I thought I had this case covered but apparently it wasn't covered.

I have also fixed the broken unit tests.

Based on the meeting om 2021-06-24:

Rajmund Hruska wrote in #note-31:

I noticed that at one point Mentat tried sending a report with empty to and cc headers. I checked the target abuse group and found out that this group only has abuse contacts for events with severity high or critical and the event Mentat was reporting had severity medium. I think it was a test group but this type of issue should probably be handled.

Mentat now filters out the groups from resolved abuse groups list where the reported severity is lower than the expected severity.

I did not write a code for checking that any IP address from any network stored in the database can be reported for any severity. First of all, it's not related to this issue. Secondly, do we really want to enforce that any IP address belonging to the network of some abuse group can be reported for any severity?

The other issue is with the shown IP address in the reports. If I remember correctly, if the event has source IP addresses from multiple (unrelated) abuse groups, the report should only show IP addresses which belong to the particular group which owns the report. Also, the web interface in the tab Statistics -> # IPs only shows IP4 addresses and does not show IP6 addresses.

I should check how it is displayed in the current version of Mentat (mentat-alt), by using artificial Idea event with multiple source IP addresses which belong to multiple groups. This is yet to be done.

Rajmund Hruska wrote in #note-32:

The other issue is with the shown IP address in the reports. If I remember correctly, if the event has source IP addresses from multiple (unrelated) abuse groups, the report should only show IP addresses which belong to the particular group which owns the report. Also, the web interface in the tab Statistics -> # IPs only shows IP4 addresses and does not show IP6 addresses.

I should check how it is displayed in the current version of Mentat (mentat-alt), by using artificial Idea event with multiple source IP addresses which belong to multiple groups. This is yet to be done.

I created artificials events (1, 2 and 3) with multiple sources and stored it at mentat-alt. The behaviour in my branch implementing this feature is the same as the current version of Mentat (2.8). In the report message, only the relevant IP addresses are shown which is expected. But "Metadata" tab of the report counts all IP4 addresses across all reported events and doesn't count IP6 addresses. Similarly, "Statistics" tab, section "# IPs" only shows all IP4 addresses. The reports for each groups can be found here: 1, 2, 3, 4

At 2021-06-24 meeting, I mentioned that when creating a name of abuse group which consists of multiple emails I assume that those emails stay in the same severity across all networks.

Basically, I assume that situations like this don't occur:

{
...
"resolved_abuses": {
      "medium": ["aaa@bbb.cz"],
      "low": ["aaa@bbb2.cz"]
    }
}
{
...
"resolved_abuses": {
      "high": ["aaa@bbb.cz"],
      "medium": ["aaa@bbb2.cz"]
    }
}

During the meeting it was said that this assumption should be checked. I wrote a function called in netmngr.py module which checks that and logs inconsistencies. There is no inconsistency in the data from Negistry.

Rajmund Hruska wrote in #note-36:

Rajmund Hruska wrote in #note-34:

At 2021-06-24 meeting, I mentioned that when creating a name of abuse group which consists of multiple emails I assume that those emails stay in the same severity across all networks.

Basically, I assume that situations like this don't occur:
[...]This has

During the meeting it was said that this assumption should be checked. I wrote a function called in netmngr.py module which checks that and logs inconsistencies. There is no inconsistency in the data from Negistry.

With the new fallback option this is no longer true. There is a network with only one resolved_abuses item (fallback: abuse@cesnet.cz) and there is another network with only one resolved_abuses item (low: abuse@cesnet.cz). It is still just one group with one resolved email, but the email occurs in different 'severities'.

Currently, in the database there are 3 entities: groups, networks and reporting settings. Each group has exactly one reporting settings, where the resolved abuses are stored.

There are probably multiple ways how to handle fallback option. The simplest one I can think of is storing the fallback in the networks table. The disadvantage of that is that some emails would be stored in the networks table and some in the reporting settings table.

What is your opinion?

After discussion we came to three options:

• storing the fallback in the networks table (needs model change, migration, maybe UI changes, etc.)
• split the concerned group to two with different attributes, akin to abuse@cesnet.cz as now + abuse@cesnet.cz-fallback for fallback networks (makes UI cluttered for group admin, need to manage two groups)
• those base ranges where fallback attribute is used are specific for CESNET/Negistry, we could rely on global fallback (which would need to be implemented) and basically ignore fallback attribute (maybe check whether the value is expected - abuse@cesnet.cz, to nail down the case when model on the side of Negistry somehow changes)

Third seems to be the least invasive and work fine for now.

Pavel Kácha wrote in #note-37:

• those base ranges where fallback attribute is used are specific for CESNET/Negistry, we could rely on global fallback (which would need to be implemented) and basically ignore fallback attribute (maybe check whether the value is expected - abuse@cesnet.cz, to nail down the case when model on the side of Negistry somehow changes)

Third seems to be the least invasive and work fine for now.

• create global fallback, which will be used if we are unable to resolve any abuse contact (possibly more addresses)
• this creates ambiguity as to whom should get the ownership of the report, when there is no resolve. This is deeper problem, and we decided to change the behavior to assign the report to ALL related groups, but use low/medium/high only when sending mails - and use global fallback only when no mail would be sent
• maybe we could look into getting this information into reporting templates, to be able to clearly signal in the report that is's "orphaned", so the fallback recipient clearly knows what happened
• update Negistry importer script to just check the fallback attribute for abuse@cesnet.cz and quack when it's not the case (to underpin possible Negistry model change)

I chose to implement a global fallback. In the netmngr.py script which imports abuse groups and networks, if network has fallback it is never stored in the database. If I stored such networks I would successfully resolve IP address and the issue wouldn't be solved. The networks with fallback option are stored temporarily to check that this option is the same as the global fallback, which is stored in core/reporting.json.conf. If those fallbacks differ, an error is logged but the script continues without stopping.

Finally, filtering abuse groups based on severity is done after the report is created and if there is no email address to send the report to, the global fallback is used.

I haven't made any change in reporting template yet.

Rajmund Hruska wrote in #note-40:

I haven't made any change in reporting template yet.

After discussion, don't change the template(s) as this would need storing the data somewhere and in this case it is clearly visible in target emails (To or CC in mail and Metadata on web).

Rajmund Hruska wrote in #note-41:

Rajmund Hruska wrote in #note-35:

Also, at some point I tried comparing resolved abuses from Mentat (with data from Negistry) with those from negistry-multifeed-abuses-segmented-ipranges.json and for some IP addresses 'to' emails and 'cc' emails differ (Mentat says that email should be in 'cc' header and negistry-multifeed-abuses-segmented-ipranges.json says that the email should be in 'to' header). I should run this comparison again.

I tried comparing resolved emails from Mentat (with data from Negistry) and emails from negistry-multifeed-abuses-segmented-ipranges.json. One of the issues that I came across, is that the "fallback" option is used for some networks which originally didn't have fallback option defined. One of such networks is 193.84.68.2 - 193.84.68.2. There is a resolved abuse contact for medium severity but there is none for low severity so the global fallback option is used.

The other issue is that some emails that should be in 'cc' header are in 'to' header. For example network 78.128.192.0 - 78.128.199.255 has only resolved email netadm@kolej.mff.cuni.cz for critical severity. This network is a part of network 78.128.192.0 - 78.128.207.255 which has resolved email abuse@cuni.cz for low severity. So for events with critical severity, the headers should be: to = [netadm@kolej.mff.cuni.cz], cc = [abuse@cuni.cz]. But in negistry-multifeed-abuses-segmented-ipranges.json the headers are: to = [netadm@kolej.mff.cuni.cz, abuse@cuni.cz], cc = [].

Is that OK?

I tried that again with new data but those two issues are still present. The examples are the same I wrote about in the previous post.

Rajmund, could you please look into #3676#note-145 and try to describe the Mentat algorithm in very similar vein for comparison?

Pavel Kácha wrote in #note-45:

Rajmund, could you please look into #3676#note-145 and try to describe the Mentat algorithm in very similar vein for comparison?

Here is how resolving is done in Mentat:
0. Sort IP ranges by the first address as primary key and the last address as secondary key (results in sorting intersecting ranges by size)
1. Out of all networks from the database, take those which the requested IP address belongs to
2. Stable sort those networks by rank
(Networks are now sorted primarily by rank and secondarily by the size of the range)
3. Take resolved abuses (groups) from each network and uniquify them
4. Remove abuse groups which require higher severity
5. Add the emails of the first group to "To" header
6. Add unique emails of the other groups to "Cc" header
7. Use fallback emails if "To" header is empty

So, the difference was that in Negistry in "To" header were all emails from the networks with the highest rank, whereas in Mentat only emails from network with the most specific range were used. Anyway, I read the following comments in #3676 and I found out that it was changed in the Negistry.

I ran the comparison once again. Now, there is only difference - global fallback is used for addresses where resolved abuse is not defined for some severities. the networks are 14.22.7.0/24, 14.22.8.0/24, 169.254.1.1 - 169.254.1.1, 193.84.68.2 - 193.84.68.2 and 193.84.68.3 - 193.84.68.3. That's OK though, isn't it?

Rajmund Hruska wrote in #note-46:

Anyway, I read the following comments in #3676 and I found out that it was changed in the Negistry.

I ran the comparison once again. Now, there is only difference - global fallback is used for addresses where resolved abuse is not defined for some severities. the networks are 14.22.7.0/24, 14.22.8.0/24, 169.254.1.1 - 169.254.1.1, 193.84.68.2 - 193.84.68.2 and 193.84.68.3 - 193.84.68.3. That's OK though, isn't it?

First two are nonsensical ones (testing), so nevermind. Others show that feed import on Nevel was done as medium severity for CESNET/PASNET-DEVICES, which shouldn't be the case - I fixed this to low which should solve the issue?

Pavel Kácha wrote in #note-47:

Rajmund Hruska wrote in #note-46:

Anyway, I read the following comments in #3676 and I found out that it was changed in the Negistry.

I ran the comparison once again. Now, there is only difference - global fallback is used for addresses where resolved abuse is not defined for some severities. the networks are 14.22.7.0/24, 14.22.8.0/24, 169.254.1.1 - 169.254.1.1, 193.84.68.2 - 193.84.68.2 and 193.84.68.3 - 193.84.68.3. That's OK though, isn't it?

First two are nonsensical ones (testing), so nevermind. Others show that feed import on Nevel was done as medium severity for CESNET/PASNET-DEVICES, which shouldn't be the case - I fixed this to low which should solve the issue?

Yes it solved the issue. Now there is no difference.

Dev Negistry (as in Nevel, with multifeeds and such) have been deployed to production.

Data are without non-RIPE feeds as yet, so negistry-ripe-cesnet-ipranges.json should be unchanged, however negistry-multifeed-abuses-*-ipranges.json feeds are available now and ready for migration tests.

From discussion about preparation for migration.

On production (old Negistry import script) we could:

• for networks, which have "negistry" as a source, we can maybe (*) manually change to "negistry/ripe_cesnet"
• for networks, which are external, but now are in Negistry, we can maybe (*) manually change to "negistry/ripe_cesnet"
• either during or after migration we have to merge (or rename+delete) abuse@radio.cz and abuse@rozhlas.cz to "abuse@radio.cz,abuse@rozhlas.cz"

(*) Depends on whether this does not break current Negistry import script, needs to be reviewed/tested.

I noticed that on 2021-11-23 there was a new abuse group created on mentat-hub called abuse@oabrno.cz. This group is actually just renamed old group abuse@oakobrno.cz, so it's the same situation as I described in the second case of #note-44 and the old group can be renamed and the new one removed.

Rajmund Hruska wrote in #note-51:

I noticed that on 2021-11-23 there was a new abuse group created on mentat-hub called abuse@oabrno.cz. This group is actually just renamed old group abuse@oakobrno.cz, so it's the same situation as I described in the second case of #note-44 and the old group can be renamed and the new one removed.

Thx, done.

There was an issue with abuse@nbu.cz which was treated as a new group even though it already existed in the database. The reason for that is that this group has the same target email as group named GovCERT - mentat-reports+nbu@cesnet.cz - so mentat-reports+nbu@cesnet.cz was first a dictionary key for value abuse@nbu.cz and later was replaced by group GovCERT.

I think it doesn't make sense to have two different abuse groups with the same target email. I suggest adding abuse@nbu.cz as target email for group abuse@nbu.cz, alongside the mentat-reports+nbu@cesnet.cz.

On mentat-hub abuse group abuse@upol.cz has two networks with the same range - 195.113.148.148-195.113.148.151. The only difference is source (manual/negistry) and description. I think it's safe to delete the network with manual source.

Rajmund Hruska wrote in #note-53:

There was an issue with abuse@nbu.cz which was treated as a new group even though it already existed in the database. The reason for that is that this group has the same target email as group named GovCERT - mentat-reports+nbu@cesnet.cz - so mentat-reports+nbu@cesnet.cz was first a dictionary key for value abuse@nbu.cz and later was replaced by group GovCERT.

I think it doesn't make sense to have two different abuse groups with the same target email. I suggest adding abuse@nbu.cz as target email for group abuse@nbu.cz, alongside the mentat-reports+nbu@cesnet.cz.

According to phone call, I've changed target email for GovCERT.

Rajmund Hruska wrote in #note-54:

On mentat-hub abuse group abuse@upol.cz has two networks with the same range - 195.113.148.148-195.113.148.151. The only difference is source (manual/negistry) and description. I think it's safe to delete the network with manual source.

Thx. "Manual" one deleted.

Pavel Kácha wrote in #note-50:

From discussion about preparation for migration.

On production (old Negistry import script) we could:

• for networks, which have "negistry" as a source, we can maybe (*) manually change to "negistry/ripe_cesnet"
• for networks, which are external, but now are in Negistry, we can maybe (*) manually change to "negistry/ripe_cesnet"
• either during or after migration we have to merge (or rename+delete) abuse@radio.cz and abuse@rozhlas.cz to "abuse@radio.cz,abuse@rozhlas.cz"

(*) Depends on whether this does not break current Negistry import script, needs to be reviewed/tested.

So, I downloaded data form mentat-hub, changed the source of networks from negistry to negistry/ripe_cesnet.
update networks set source = 'negistry/ripe_cesnet' where source = 'negistry';

• 195.113.151.144-195.113.151.159
  • Netname: 'CESNET-WEBCON-BRNO' -> 'CESNET-HQVIDEO-BRNO'
• 195.113.209.128-195.113.209.191
  • Netname: 'CESNET-META-OSTRAVA' -> 'CESNET-META-BRNO'
• 2001:718:1:4::/64
  • Netname: 'CESNET-6TK' -> 'CESNET-6SRV2'
• 2001:718:1:2c::/63
  • Netname: 'CESNET-6JN' -> 'CESNET-6JM'
• 195.113.243.64-195.113.243.127
  • Netname: 'JIC-TCZ' -> 'JIC-2-TCZ'
• 195.113.192.32-195.113.192.47
  • Netname: 'OAKOBRNO-TCZ' -> 'OABRNO-TCZ'
• 2001:718:1e00:8800::/56
  • Netname: 'VFN-6TCZ' -> 'VFN-IC-6TCZ'

I ran the nrtmngr.py script in devel and nothing has changed, no update was required.

After switching to my negistry branch and migrating the database, I deleted abuse@rozhlas.cz, renamed abuse@radio.cz to abuse@radio.cz,abuse@rozhlas.cz and changed the target emails for low severity to abuse@radio.cz,abuse@rozhlas.cz in reporting settings.

I ran the netmngr.py script and the only thing that changed was a new rank for networks (from None to some value).

Rajmund Hruska wrote in #note-59:

The solution which I can think of now is to remove the networks which has fallback attribute (in Negistry export) from the database. The other way of solving this is to start storing fallback networks in the database with some flag and modify reporting.

I think it's not wise to drop those networks altogether, so I'd go for the last solution - flag the networks as 'base' or 'fallback'. However, if this net is reached as the sole source of contact info (thus fallback), I'd use owner group contact info here - it would allow more 'base' networks with different owner group. Global fallback would then cover only the cases where even fallback network does not exist.

Seems there is some glitch, report M20220301SL-75KS6 on Mentat-alt went to abuse@cesnet instead of abuse@cuni.

Rajmund Hruska wrote in #note-63:

Pavel Kácha wrote in #note-62:

Seems there is some glitch, report M20220301SL-75KS6 on Mentat-alt went to abuse@cesnet instead of abuse@cuni.

I found out what was the issue. There is an active filter on Mentat-alt - Category eq 'Recon.Scanning' and Source.IP4 in [78.128.160.0/19]. So abuse@cuni.cz was filtered out and the only group which was left was the fallback group abuse@cesnet.cz. The reason, why it wasn't in the log is that this information is stored with log level debug.

Thanks.

Mmmkay, that opens a bit of another can of worms - as it does not keep existing semantics. To keep it, I'd suggest to leave out reports which hit some of the rules altogether (with some hint in logs). And sometime later we will probably have to enhance the rules capability to be able to specify the "scope" (probably something akin to global/local/local-nested).

Pavel Kácha wrote in #note-64:

Rajmund Hruska wrote in #note-63:

Pavel Kácha wrote in #note-62:

Seems there is some glitch, report M20220301SL-75KS6 on Mentat-alt went to abuse@cesnet instead of abuse@cuni.

I found out what was the issue. There is an active filter on Mentat-alt - Category eq 'Recon.Scanning' and Source.IP4 in [78.128.160.0/19]. So abuse@cuni.cz was filtered out and the only group which was left was the fallback group abuse@cesnet.cz. The reason, why it wasn't in the log is that this information is stored with log level debug.

... To keep it, I'd suggest to leave out reports which hit some of the rules altogether (with some hint in logs). ...

I don't really understand what you mean. Reports can contain information that some of the events were filtered out. Those reports shouldn't be reported at all? Or maybe you meant to say that we should leave out events, which hit some rules. That doesn't sound like the best solution to me. So, could you please clarify your suggestions?

Rajmund Hruska wrote in #note-65:

Pavel Kácha wrote in #note-64:

Rajmund Hruska wrote in #note-63:

Pavel Kácha wrote in #note-62:

Seems there is some glitch, report M20220301SL-75KS6 on Mentat-alt went to abuse@cesnet instead of abuse@cuni.

I found out what was the issue. There is an active filter on Mentat-alt - Category eq 'Recon.Scanning' and Source.IP4 in [78.128.160.0/19]. So abuse@cuni.cz was filtered out and the only group which was left was the fallback group abuse@cesnet.cz. The reason, why it wasn't in the log is that this information is stored with log level debug.

... To keep it, I'd suggest to leave out reports which hit some of the rules altogether (with some hint in logs). ...

I don't really understand what you mean. Reports can contain information that some of the events were filtered out. Those reports shouldn't be reported at all? Or maybe you meant to say that we should leave out events, which hit some rules. That doesn't sound like the best solution to me. So, could you please clarify your suggestions?

Yeah, you're right, I'm mixing up terminology again. Events, of course.
I meant - if event (or part of it) is left out for at least one of the target groups, discard it for all groups in this chain.

For all of the rules currently in the db we don't want matching data reported to anybody - people at 'fallback' addresses would be probably puzzled and maybe overwhelmed with closing unnecessary tickets. This might change in future and that's what I meant by that 'scope' babble.

If still unsure or it looks too invasive change, feel free to call, we'll try to find a reasonable solution.

Pavel Kácha wrote in #note-66:

Rajmund Hruska wrote in #note-65:

Pavel Kácha wrote in #note-64:

Rajmund Hruska wrote in #note-63:

Pavel Kácha wrote in #note-62:

Seems there is some glitch, report M20220301SL-75KS6 on Mentat-alt went to abuse@cesnet instead of abuse@cuni.

I found out what was the issue. There is an active filter on Mentat-alt - Category eq 'Recon.Scanning' and Source.IP4 in [78.128.160.0/19]. So abuse@cuni.cz was filtered out and the only group which was left was the fallback group abuse@cesnet.cz. The reason, why it wasn't in the log is that this information is stored with log level debug.

... To keep it, I'd suggest to leave out reports which hit some of the rules altogether (with some hint in logs). ...

I don't really understand what you mean. Reports can contain information that some of the events were filtered out. Those reports shouldn't be reported at all? Or maybe you meant to say that we should leave out events, which hit some rules. That doesn't sound like the best solution to me. So, could you please clarify your suggestions?

Yeah, you're right, I'm mixing up terminology again. Events, of course.
I meant - if event (or part of it) is left out for at least one of the target groups, discard it for all groups in this chain.

For all of the rules currently in the db we don't want matching data reported to anybody - people at 'fallback' addresses would be probably puzzled and maybe overwhelmed with closing unnecessary tickets. This might change in future and that's what I meant by that 'scope' babble.

If still unsure or it looks too invasive change, feel free to call, we'll try to find a reasonable solution.

Thank you for the clarification, I see what you mean now. It seems to be a simple change, can I deploy this to mentat-alt right away so it can be tested ASAP?

Rajmund Hruska wrote in #note-67:

Pavel Kácha wrote in #note-66:

Rajmund Hruska wrote in #note-65:

Pavel Kácha wrote in #note-64:

Rajmund Hruska wrote in #note-63:

Pavel Kácha wrote in #note-62:

Seems there is some glitch, report M20220301SL-75KS6 on Mentat-alt went to abuse@cesnet instead of abuse@cuni.

I found out what was the issue. There is an active filter on Mentat-alt - Category eq 'Recon.Scanning' and Source.IP4 in [78.128.160.0/19]. So abuse@cuni.cz was filtered out and the only group which was left was the fallback group abuse@cesnet.cz. The reason, why it wasn't in the log is that this information is stored with log level debug.

... To keep it, I'd suggest to leave out reports which hit some of the rules altogether (with some hint in logs). ...

I don't really understand what you mean. Reports can contain information that some of the events were filtered out. Those reports shouldn't be reported at all? Or maybe you meant to say that we should leave out events, which hit some rules. That doesn't sound like the best solution to me. So, could you please clarify your suggestions?

Yeah, you're right, I'm mixing up terminology again. Events, of course.
I meant - if event (or part of it) is left out for at least one of the target groups, discard it for all groups in this chain.

For all of the rules currently in the db we don't want matching data reported to anybody - people at 'fallback' addresses would be probably puzzled and maybe overwhelmed with closing unnecessary tickets. This might change in future and that's what I meant by that 'scope' babble.

If still unsure or it looks too invasive change, feel free to call, we'll try to find a reasonable solution.

Thank you for the clarification, I see what you mean now. It seems to be a simple change, can I deploy this to mentat-alt right away so it can be tested ASAP?

That'd be awesome.

Let's assume that every event has only one source address. I resolve the networks based on the IP address from this source and split them into two lists - base (fallback) networks and the other networks (this information is useful later in the code). Then I iterate over the list of those other networks and for each network I try all of the filters of the abuse group which owns this network. If at least one of the filtering rules is matched, I disregard the event.

But if none of the filtering rules were matched, I proceed to do the filtering for base networks. My question is, should the same rule of disregarding event be applied even for these base networks? I would say it is valid to send the report to the owner of the second network even if the owner of the first network decided to filter the event.

Rajmund Hruska wrote in #note-69:

Let's assume that every event has only one source address. I resolve the networks based on the IP address from this source and split them into two lists - base (fallback) networks and the other networks (this information is useful later in the code). Then I iterate over the list of those other networks and for each network I try all of the filters of the abuse group which owns this network. If at least one of the filtering rules is matched, I disregard the event.

But if none of the filtering rules were matched, I proceed to do the filtering for base networks. My question is, should the same rule of disregarding event be applied even for these base networks? I would say it is valid to send the report to the owner of the second network even if the owner of the first network decided to filter the event.

So you mean first (other list) network does not get it, and the second (fallback list) should?

Pavel Kácha wrote in #note-70:

Rajmund Hruska wrote in #note-69:

Let's assume that every event has only one source address. I resolve the networks based on the IP address from this source and split them into two lists - base (fallback) networks and the other networks (this information is useful later in the code). Then I iterate over the list of those other networks and for each network I try all of the filters of the abuse group which owns this network. If at least one of the filtering rules is matched, I disregard the event.

But if none of the filtering rules were matched, I proceed to do the filtering for base networks. My question is, should the same rule of disregarding event be applied even for these base networks? I would say it is valid to send the report to the owner of the second network even if the owner of the first network decided to filter the event.

So you mean first (other list) network does not get it, and the second (fallback list) should?

No. If the first list filters the event then no one gets the event. So if the event passes the filters of the first rules then the report will be created for all groups from the first list. But it is possible that there are multiple fallback networks and some of them don't want to receive the event (based on the filters). Fallback networks will receive the report only if all of the groups from the first list don't want to receive the report by email.

Rajmund Hruska wrote in #note-71:

Pavel Kácha wrote in #note-70:

Rajmund Hruska wrote in #note-69:

Let's assume that every event has only one source address. I resolve the networks based on the IP address from this source and split them into two lists - base (fallback) networks and the other networks (this information is useful later in the code). Then I iterate over the list of those other networks and for each network I try all of the filters of the abuse group which owns this network. If at least one of the filtering rules is matched, I disregard the event.

But if none of the filtering rules were matched, I proceed to do the filtering for base networks. My question is, should the same rule of disregarding event be applied even for these base networks? I would say it is valid to send the report to the owner of the second network even if the owner of the first network decided to filter the event.

So you mean first (other list) network does not get it, and the second (fallback list) should?

No. If the first list filters the event then no one gets the event. So if the event passes the filters of the first rules then the report will be created for all groups from the first list. But it is possible that there are multiple fallback networks and some of them don't want to receive the event (based on the filters). Fallback networks will receive the report only if all of the groups from the first list don't want to receive the report by email.

I see (hopefully), you meant like the first network from the fallback list vs the second network from the fallback list. However, wWe should not have more fallback addresses active in one time I guess? Or have we? But if we have, I guess you're right, we could be more conservative in discarding data here.

Pavel Kácha wrote in #note-72:

Rajmund Hruska wrote in #note-71:

Pavel Kácha wrote in #note-70:

Rajmund Hruska wrote in #note-69:

Let's assume that every event has only one source address. I resolve the networks based on the IP address from this source and split them into two lists - base (fallback) networks and the other networks (this information is useful later in the code). Then I iterate over the list of those other networks and for each network I try all of the filters of the abuse group which owns this network. If at least one of the filtering rules is matched, I disregard the event.

But if none of the filtering rules were matched, I proceed to do the filtering for base networks. My question is, should the same rule of disregarding event be applied even for these base networks? I would say it is valid to send the report to the owner of the second network even if the owner of the first network decided to filter the event.

So you mean first (other list) network does not get it, and the second (fallback list) should?

No. If the first list filters the event then no one gets the event. So if the event passes the filters of the first rules then the report will be created for all groups from the first list. But it is possible that there are multiple fallback networks and some of them don't want to receive the event (based on the filters). Fallback networks will receive the report only if all of the groups from the first list don't want to receive the report by email.

I see (hopefully), you meant like the first network from the fallback list vs the second network from the fallback list. However, wWe should not have more fallback addresses active in one time I guess? Or have we? But if we have, I guess you're right, we could be more conservative in discarding data here.

Yes, that's what I meant. Isn't it in theory possible to have multiple base (fallback) networks? I remember you asked me what would happen in that case so I thought there is such a case in the data from negistry.

Rajmund Hruska wrote in #note-73:

Pavel Kácha wrote in #note-72:

Rajmund Hruska wrote in #note-71:

Pavel Kácha wrote in #note-70:

Rajmund Hruska wrote in #note-69:

Let's assume that every event has only one source address. I resolve the networks based on the IP address from this source and split them into two lists - base (fallback) networks and the other networks (this information is useful later in the code). Then I iterate over the list of those other networks and for each network I try all of the filters of the abuse group which owns this network. If at least one of the filtering rules is matched, I disregard the event.

But if none of the filtering rules were matched, I proceed to do the filtering for base networks. My question is, should the same rule of disregarding event be applied even for these base networks? I would say it is valid to send the report to the owner of the second network even if the owner of the first network decided to filter the event.

So you mean first (other list) network does not get it, and the second (fallback list) should?

No. If the first list filters the event then no one gets the event. So if the event passes the filters of the first rules then the report will be created for all groups from the first list. But it is possible that there are multiple fallback networks and some of them don't want to receive the event (based on the filters). Fallback networks will receive the report only if all of the groups from the first list don't want to receive the report by email.

I see (hopefully), you meant like the first network from the fallback list vs the second network from the fallback list. However, wWe should not have more fallback addresses active in one time I guess? Or have we? But if we have, I guess you're right, we could be more conservative in discarding data here.

Yes, that's what I meant. Isn't it in theory possible to have multiple base (fallback) networks? I remember you asked me what would happen in that case so I thought there is such a case in the data from negistry.

It shouldn't, but it is, as we're already a bit on a wrong side of real world data complexity here.

Rajmund Hruska wrote in #note-75:

Pavel Kácha wrote in #note-74:

Rajmund Hruska wrote in #note-73:

Yes, that's what I meant. Isn't it in theory possible to have multiple base (fallback) networks? I remember you asked me what would happen in that case so I thought there is such a case in the data from negistry.

It shouldn't, but it is, as we're already a bit on a wrong side of real world data complexity here.

I see. I have updated the code and deployed the new version so there will be a couple of reports before the next meeting.

Thank you!