Project

General

Profile

Feature #6800

Rename _CESNET key to _Mentat (or make it configurable)

Added by Pavel Kácha 7 months ago. Updated 3 days ago.

Status:
Resolved
Priority:
Normal
Category:
-
Target version:
Start date:
11/19/2020
Due date:
% Done:

100%

Estimated time:
To be discussed:

Description

Historically we save Mentat specific keys (abuse contacts, classes, enrichments, ...) within _CESNET namespace in Idea. That does not make sense for instances running in different organisations. We should thus rename the key (potentially to _Mentat), or even make the name configurable.


Related issues

Related to Mentat - Feature #4231: Implement and make use of cleanup mechanism for "_" namespaces from IDEA messages.ClosedRajmund Hruska07/27/2018

Actions
Blocks Mentat - Config #6949: Merge inspector configs into oneClosedRajmund Hruska01/22/2021

Actions
#2

Updated by Pavel Kácha 7 months ago

  • Related to Feature #4231: Implement and make use of cleanup mechanism for "_" namespaces from IDEA messages. added
#3

Updated by Rajmund Hruska 7 months ago

  • Status changed from New to In Progress
  • To be discussed changed from No to Yes
#4

Updated by Pavel Kácha 7 months ago

Will need (quite long) transition period, as we have Idea events, which are quite old (over year for CESNET networks).
Correct way would be to allow for "envelope" for Idea events, passing through the system, with metadata fields, however that would need quite a redesign. So we'll stay with current model, "injecting" our own fields into suitable place within Idea. Let's start with simply renaming to _Mentat, we might move some fields into more "standardized" positions later.

However it would be nice to also rename database fields, let's drop the "cesnet_" prefix in column names altogether to prevent another possible renames in the future.

As for transition/compatibility, the aim is to start adding fields into new (_Mentat prefixed) names in Enricher, Inspector and Storage, however consider both namespaces (_CESNET and _Mentat) when reading. That could be implemented in various ways (everywhere on use, translation in database layer, replacement by generated fields from metadata table, some combination thereof).

#5

Updated by Rajmund Hruska 6 months ago

  • Status changed from In Progress to Feedback
  • Assignee changed from Rajmund Hruska to Pavel Kácha

I noticed that in pynspect, there are also _CESNET keys in the test files. Should those keys be renamed to _Mentat?

#6

Updated by Rajmund Hruska 6 months ago

  • % Done changed from 0 to 80
#7

Updated by Rajmund Hruska 6 months ago

Also, in lib/mentat/stats/idea.py there is some work done with _CESNET (lines 99-103). In my previous commit, I just renamed the key but I am not sure whether it should be also compatible with the _CESNET or not.

#8

Updated by Rajmund Hruska 6 months ago

  • % Done changed from 80 to 50
#9

Updated by Rajmund Hruska 6 months ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Pavel Kácha to Rajmund Hruska
  • To be discussed deleted (Yes)
#10

Updated by Pavel Kácha 6 months ago

Rajmund Hruska wrote in #note-7:

Also, in lib/mentat/stats/idea.py there is some work done with _CESNET (lines 99-103). In my previous commit, I just renamed the key but I am not sure whether it should be also compatible with the _CESNET or not.

From live video: I'm afraid it will have to be more clever - just adding new key would add two types of stats. Solutions (from the top of my head) to be considered:

  • If LIST_AGGREGATIONS are considered linearly everywhere, solution might be just to add second line, like:
    [ST_SKEY_ABUSES,     '_CESNET.ResolvedAbuses',        KEY_UNKNOWN],
    [ST_SKEY_ABUSES,     '_Mentat.ResolvedAbuses',        KEY_UNKNOWN],
  • Or make middle term multiple and modify statistical logic:
    [ST_SKEY_ABUSES,     ('_CESNET.ResolvedAbuses', '_Mentat.ResolvedAbuses'),        KEY_UNKNOWN],
  • Or - of course - hardwire somehow.
#11

Updated by Pavel Kácha 6 months ago

Rajmund Hruska wrote in #note-5:

I noticed that in pynspect, there are also _CESNET keys in the test files. Should those keys be renamed to _Mentat?

From live video: They are concerning only tests, only example Idea, and not probably really used anywhere. Would make sense to rename them or get rid of the altogether, but I'd do that after we're ready on our side.

#12

Updated by Pavel Kácha 6 months ago

From live video: Also, still to do is
  • database schema change
  • Alembic migration
#13

Updated by Rajmund Hruska 6 months ago

Pavel Kácha wrote in #note-10:

Rajmund Hruska wrote in #note-7:

Also, in lib/mentat/stats/idea.py there is some work done with _CESNET (lines 99-103). In my previous commit, I just renamed the key but I am not sure whether it should be also compatible with the _CESNET or not.

From live video: I'm afraid it will have to be more clever - just adding new key would add two types of stats. Solutions (from the top of my head) to be considered:

  • If LIST_AGGREGATIONS are considered linearly everywhere, solution might be just to add second line, like:

[...]

  • Or make middle term multiple and modify statistical logic:

[...]

  • Or - of course - hardwire somehow.

I decided to make tuples - 38233d03

#14

Updated by Rajmund Hruska 6 months ago

  • Status changed from In Progress to Resolved
  • Assignee changed from Rajmund Hruska to Pavel Kácha
  • % Done changed from 50 to 100

Pavel Kácha wrote in #note-12:

From live video: Also, still to do is

I think all is done now.

  • database schema change

78149a32

  • Alembic migration

cb23deef

#15

Updated by Pavel Kácha 5 months ago

  • Blocks Config #6949: Merge inspector configs into one added
#16

Updated by Pavel Kácha 4 months ago

  • Status changed from Resolved to In Review
#17

Updated by Jan Mach 4 months ago

  • Status changed from In Review to Closed

Merged into devel.

#18

Updated by Pavel Kácha 4 months ago

  • Status changed from Closed to In Review
  • Assignee changed from Pavel Kácha to Rajmund Hruska

Seems there is still something fishy (500 on event search on mentat-alt), reopening.

#19

Updated by Rajmund Hruska 4 months ago

The problem was that the database wasn't migrated. When I ran migration manually by alembic upgrade head, the problem was resolved. Is database migration part of the build at alchemist? I can't find that quickly.

#20

Updated by Pavel Kácha 4 months ago

  • To be discussed set to Yes
#21

Updated by Pavel Kácha 4 months ago

  • To be discussed deleted (Yes)
#22

Updated by Rajmund Hruska 26 days ago

During the meeting, we noticed that there are some indexes with the old name (they contain _cesnet). I updated the migration.

#23

Updated by Pavel Kácha 25 days ago

  • Status changed from In Review to Closed
#24

Updated by Rajmund Hruska 21 days ago

I noticed that there are still old index names in 'Upgrading PostgreSQL from 12.x to 13.x' section in the documentation so I updated the names.

#25

Updated by Pavel Kácha 3 days ago

  • Status changed from Closed to In Progress

Reopening to not forget it on 2.8 release.

#26

Updated by Rajmund Hruska 3 days ago

  • Status changed from In Progress to Resolved

As it was discussed during the meeting on 2021-06-18, I added the note about upgrading to the latest version of Mentat prior to the sections "Upgrading PostgreSQL from 11.x to 12.x" and "Upgrading PostgreSQL from 12.x to 13.x".

Also available in: Atom PDF