Project

General

Profile

Actions

Feature #6800

closed

Rename _CESNET key to _Mentat (or make it configurable)

Added by Pavel Kácha over 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Category:
-
Target version:
Start date:
11/19/2020
Due date:
% Done:

100%

Estimated time:
To be discussed:

Description

Historically we save Mentat specific keys (abuse contacts, classes, enrichments, ...) within _CESNET namespace in Idea. That does not make sense for instances running in different organisations. We should thus rename the key (potentially to _Mentat), or even make the name configurable.


Related issues

Related to Warden - Feature #6799: Implement filtering of keys on outgoing eventsClosedPavel Kácha11/19/2020

Actions
Related to Mentat - Feature #4231: Implement and make use of cleanup mechanism for "_" namespaces from IDEA messages.ClosedRajmund Hruška07/27/2018

Actions
Blocks Mentat - Config #6949: Merge inspector configs into oneClosedRajmund Hruška01/22/2021

Actions
Actions #1

Updated by Pavel Kácha over 3 years ago

  • Related to Feature #6799: Implement filtering of keys on outgoing events added
Actions #2

Updated by Pavel Kácha over 3 years ago

  • Related to Feature #4231: Implement and make use of cleanup mechanism for "_" namespaces from IDEA messages. added
Actions #3

Updated by Rajmund Hruška over 3 years ago

  • Status changed from New to In Progress
  • To be discussed changed from No to Yes
Actions #4

Updated by Pavel Kácha over 3 years ago

Will need (quite long) transition period, as we have Idea events, which are quite old (over year for CESNET networks).
Correct way would be to allow for "envelope" for Idea events, passing through the system, with metadata fields, however that would need quite a redesign. So we'll stay with current model, "injecting" our own fields into suitable place within Idea. Let's start with simply renaming to _Mentat, we might move some fields into more "standardized" positions later.

However it would be nice to also rename database fields, let's drop the "cesnet_" prefix in column names altogether to prevent another possible renames in the future.

As for transition/compatibility, the aim is to start adding fields into new (_Mentat prefixed) names in Enricher, Inspector and Storage, however consider both namespaces (_CESNET and _Mentat) when reading. That could be implemented in various ways (everywhere on use, translation in database layer, replacement by generated fields from metadata table, some combination thereof).

Actions #5

Updated by Rajmund Hruška over 3 years ago

  • Status changed from In Progress to Feedback
  • Assignee changed from Rajmund Hruška to Pavel Kácha

I noticed that in pynspect, there are also _CESNET keys in the test files. Should those keys be renamed to _Mentat?

Actions #6

Updated by Rajmund Hruška over 3 years ago

  • % Done changed from 0 to 80
Actions #7

Updated by Rajmund Hruška over 3 years ago

Also, in lib/mentat/stats/idea.py there is some work done with _CESNET (lines 99-103). In my previous commit, I just renamed the key but I am not sure whether it should be also compatible with the _CESNET or not.

Actions #8

Updated by Rajmund Hruška over 3 years ago

  • % Done changed from 80 to 50
Actions #9

Updated by Rajmund Hruška over 3 years ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Pavel Kácha to Rajmund Hruška
  • To be discussed deleted (Yes)
Actions #10

Updated by Pavel Kácha over 3 years ago

Rajmund Hruska wrote in #note-7:

Also, in lib/mentat/stats/idea.py there is some work done with _CESNET (lines 99-103). In my previous commit, I just renamed the key but I am not sure whether it should be also compatible with the _CESNET or not.

From live video: I'm afraid it will have to be more clever - just adding new key would add two types of stats. Solutions (from the top of my head) to be considered:

  • If LIST_AGGREGATIONS are considered linearly everywhere, solution might be just to add second line, like:
    [ST_SKEY_ABUSES,     '_CESNET.ResolvedAbuses',        KEY_UNKNOWN],
    [ST_SKEY_ABUSES,     '_Mentat.ResolvedAbuses',        KEY_UNKNOWN],
  • Or make middle term multiple and modify statistical logic:
    [ST_SKEY_ABUSES,     ('_CESNET.ResolvedAbuses', '_Mentat.ResolvedAbuses'),        KEY_UNKNOWN],
  • Or - of course - hardwire somehow.
Actions #11

Updated by Pavel Kácha over 3 years ago

Rajmund Hruska wrote in #note-5:

I noticed that in pynspect, there are also _CESNET keys in the test files. Should those keys be renamed to _Mentat?

From live video: They are concerning only tests, only example Idea, and not probably really used anywhere. Would make sense to rename them or get rid of the altogether, but I'd do that after we're ready on our side.

Actions #12

Updated by Pavel Kácha over 3 years ago

From live video: Also, still to do is
  • database schema change
  • Alembic migration
Actions #13

Updated by Rajmund Hruška over 3 years ago

Pavel Kácha wrote in #note-10:

Rajmund Hruska wrote in #note-7:

Also, in lib/mentat/stats/idea.py there is some work done with _CESNET (lines 99-103). In my previous commit, I just renamed the key but I am not sure whether it should be also compatible with the _CESNET or not.

From live video: I'm afraid it will have to be more clever - just adding new key would add two types of stats. Solutions (from the top of my head) to be considered:

  • If LIST_AGGREGATIONS are considered linearly everywhere, solution might be just to add second line, like:

[...]

  • Or make middle term multiple and modify statistical logic:

[...]

  • Or - of course - hardwire somehow.

I decided to make tuples - 38233d03

Actions #14

Updated by Rajmund Hruška over 3 years ago

  • Status changed from In Progress to Resolved
  • Assignee changed from Rajmund Hruška to Pavel Kácha
  • % Done changed from 50 to 100

Pavel Kácha wrote in #note-12:

From live video: Also, still to do is

I think all is done now.

  • database schema change

78149a32

  • Alembic migration

cb23deef

Actions #15

Updated by Pavel Kácha over 3 years ago

  • Blocks Config #6949: Merge inspector configs into one added
Actions #16

Updated by Pavel Kácha over 3 years ago

  • Status changed from Resolved to In Review
Actions #17

Updated by Jan Mach over 3 years ago

  • Status changed from In Review to Closed

Merged into devel.

Actions #18

Updated by Pavel Kácha over 3 years ago

  • Status changed from Closed to In Review
  • Assignee changed from Pavel Kácha to Rajmund Hruška

Seems there is still something fishy (500 on event search on mentat-alt), reopening.

Actions #19

Updated by Rajmund Hruška over 3 years ago

The problem was that the database wasn't migrated. When I ran migration manually by alembic upgrade head, the problem was resolved. Is database migration part of the build at alchemist? I can't find that quickly.

Actions #20

Updated by Pavel Kácha over 3 years ago

  • To be discussed set to Yes
Actions #21

Updated by Pavel Kácha over 3 years ago

  • To be discussed deleted (Yes)
Actions #22

Updated by Rajmund Hruška about 3 years ago

During the meeting, we noticed that there are some indexes with the old name (they contain _cesnet). I updated the migration.

Actions #23

Updated by Pavel Kácha about 3 years ago

  • Status changed from In Review to Closed
Actions #24

Updated by Rajmund Hruška about 3 years ago

I noticed that there are still old index names in 'Upgrading PostgreSQL from 12.x to 13.x' section in the documentation so I updated the names.

Actions #25

Updated by Pavel Kácha almost 3 years ago

  • Status changed from Closed to In Progress

Reopening to not forget it on 2.8 release.

Actions #26

Updated by Rajmund Hruška almost 3 years ago

  • Status changed from In Progress to Resolved

As it was discussed during the meeting on 2021-06-18, I added the note about upgrading to the latest version of Mentat prior to the sections "Upgrading PostgreSQL from 11.x to 12.x" and "Upgrading PostgreSQL from 12.x to 13.x".

Actions #27

Updated by Rajmund Hruška almost 3 years ago

  • Status changed from Resolved to In Review

Merged into devel.

Actions #28

Updated by Pavel Kácha almost 3 years ago

  • Status changed from In Review to Closed
Actions

Also available in: Atom PDF