Project

General

Profile

Actions

Bug #7200

open

Reporting events after group name has changed

Added by Rajmund Hruska over 1 year ago. Updated 4 months ago.

Status:
In Progress
Priority:
Low
Category:
-
Target version:
Start date:
04/12/2021
Due date:
% Done:

50%

Estimated time:
To be discussed:

Description

Changing the name of an abuse group results in thresholded events not being reported. Also, the events which are not yet processed by reporter module (because events are reported in time intervals) would also not be reported.

Actions #1

Updated by Rajmund Hruska 11 months ago

  • Status changed from New to Feedback
  • Priority changed from Normal to Low
  • To be discussed changed from No to Yes
There are 2 places where this needs to be solved:
  • thresholded events - Currently, the thresholding table in the database uses group name. Instead of that, a group ID could be used. The existing records in the database could be migrated by looking up an ID for a given group name.
  • not reported events - The events to be reported are searched in the database by group name. The ID could be used instead of that, but that would make _Mentat/ResolvedAbuses much more difficult to comprehend. The idea I had was to store the list of the previous names of a group in the database and then search by this list as well. The previous names of a group are already kind of stored by changelogs_items table in the database, but this seems pretty annoying to parse.
Actions #2

Updated by Rajmund Hruska 10 months ago

  • Status changed from Feedback to In Progress
  • To be discussed deleted (Yes)

Actually, I will look at the possibility of parsing the previous names from the changelogs_items table.

Actions #3

Updated by Rajmund Hruska 10 months ago

Working at this issue made me realize that the following scenario is possible:

There are 2 groups: GROUP_A and GROUP_B. GROUP_A changes its name to GROUP_A_EDIT and GROUP_B changes its name to GROUP_A. Now when reporting for GROUP_A (former group GROUP_B) the events for GROUP_A_EDIT (former GROUP_A) will be fetched. I think this is OK though, as those events won't be reported because the sources from those events don't belong to networks owned by GROUP_A (former GROUP_B).

Actions #4

Updated by Rajmund Hruska 10 months ago

  • % Done changed from 0 to 10

Rajmund Hruska wrote in #note-2:

Actually, I will look at the possibility of parsing the previous names from the changelogs_items table.

I found out that it's actually pretty easy to parse the name from the changelogs_items table.

Actions #5

Updated by Pavel Kácha 10 months ago

Rajmund Hruska wrote in #note-3:

Working at this issue made me realize that the following scenario is possible:

There are 2 groups: GROUP_A and GROUP_B. GROUP_A changes its name to GROUP_A_EDIT and GROUP_B changes its name to GROUP_A. Now when reporting for GROUP_A (former group GROUP_B) the events for GROUP_A_EDIT (former GROUP_A) will be fetched. I think this is OK though, as those events won't be reported because the sources from those events don't belong to networks owned by GROUP_A (former GROUP_B).

All of this is pretty much best effort, unless we implement more structured logs or put IPs into Idea events.

Just a thought - to keep both exactness AND readability we could put into ResolvedAbuses both, something akin to "ResolvedAbuses": ["[37]"]. It's not nice, but get's the work done, however I'm not sure in how many places we actually do some parsing/extracting. Or, we couls add ResolvedAbusesIDs.

However, are the names inside JSON actually a problem? Couldn't it be solved by switching to IDs in events.resolved_abuses (the price is a need for translation for searching I guess).

Actions #6

Updated by Rajmund Hruska 10 months ago

  • Target version changed from Backlog to 2.10
  • % Done changed from 10 to 50

Rajmund Hruska wrote in #note-2:

Actually, I will look at the possibility of parsing the previous names from the changelogs_items table.

I managed to write a function for searching the not reported events with old names of the groups.

Actions #7

Updated by Rajmund Hruska 10 months ago

  • To be discussed set to Yes
Actions #8

Updated by Rajmund Hruska 10 months ago

  • To be discussed deleted (Yes)
Actions #9

Updated by Rajmund Hruska 9 months ago

  • Status changed from In Progress to Feedback
  • To be discussed set to Yes

So, what needs to be done next is to migrate events_thresholded table of mentat_events so it will use groupid instead of groupname. However, groups are stored in the mentat_main database, so cross-database query should be executed. I don't have any experience in cross-database querying but I found on the internet that postgres_fdw is the way to go. Radko Krkoš would you also recommend this approach or another one?

Actions #10

Updated by Rajmund Hruska 9 months ago

  • To be discussed deleted (Yes)
Actions #11

Updated by Rajmund Hruska 9 months ago

  • Status changed from Feedback to In Progress
Actions #12

Updated by Pavel Kácha 4 months ago

  • Target version changed from 2.10 to 2.11
Actions

Also available in: Atom PDF