Project

General

Profile

Actions

Feature #7653

closed

Implement Negistry-like JSON dump API for exporting group/network/abuse contact data

Added by Pavel Kácha about 1 year ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Category:
Development - Core
Target version:
Start date:
05/10/2023
Due date:
% Done:

100%

Estimated time:
To be discussed:
No

Description

As non-customer groups/networks are now inconsistently managed partially in Negistry, partially in Mentat as "manual" networks, we essentially want to take over management of non-CESNET organisations/networks from Negistry.

In Mentat those are: , , ,

In Negistry: mnt-by VUTBR-MNT, mnt-by VUT-BATCH-MNT

The line of thought is to automate fetching/updating of related data from RIPE as much as we can and make them available to the third party users (OTRS, PV, ...) through API.

For import we are mostly ready as we should be able to use existing mechanism for importing from Negistry (mentat-netmngr.py ... --whois-file source.json --source SRC), where each "set" or "feed" would be imported as separate SOURCE, all this wrapped

What remains is to make group/network information available, preferably as JSON file similar to Negistry's [1], [2], or some simplified amalgamate thereof.

(Remaining three groups in Mentat (GRID, Sner, ) are CESNET specific, in the long term should end up in NetBox and synced to Mentat by the same (mentat-netmgr) mechanism.)

Actions #1

Updated by Pavel Kácha about 1 year ago

Note from 2023-06-22:

Would be nice to have both full and segmented export, however if coding the segmented one would be to onerous, full is enough - providing the resolution algorithm is documented, unambiguous and all necessary info is in full export.

Actions #2

Updated by Pavel Kácha about 1 year ago

Notes from 2023-06-29:

  • We have a bit of collision in "source" field, as the info on the API will overlap with info from Negistry. However, the aim is to make Mentat's API the central hub of group info (to keep Mentat reports, OTRS reports and others aligned), so it seems we could use a source naming convention like today – negistry/ripe_cesnet, and then mentat/ripe_casablanka for example, and just pass source as is.
  • This info is not limited for users (API tokens), but for specific machines, so we'd like it semi-public, with access limited only by Apache config (specific IPs, maybe X509). However, if too inconsistent with the rest of Mentat API, it's not a big deal.
  • We need to add something to hold CID ("customer id"). Let's keep it general in Mentat (not tied to our CESNET internal names), and use something like "local_id", where Mentat users (as in organisations) can impose their own sense.
  • Comparing to Negistry export, we don't need "primary_key", "first" and "last" (numeric). We might add a key with Mentat group name.
Actions #3

Updated by Rajmund Hruška about 1 year ago

  • Target version changed from Backlog to 2.12
Actions #4

Updated by Pavel Kácha about 1 year ago

There exist abuse groups, which contain multiple network with different CIDs – so we are not able to reliably assign CID to the group. Main causes are:

  • connecting network (2 addresses) – the devices belong to CESNET, so abuse contact is , while the network itself is assigned to the organization itself and bears its CID
  • subparts of organisations – with their own different CIDs, but the same abuse contact

So back to the drawing board a bit – lets put the extension local_id not only to the group, but also to the networks

Actions #5

Updated by Rajmund Hruška about 1 year ago

Pavel Kácha wrote in #note-2:

  • Comparing to Negistry export, we don't need "primary_key", "first" and "last" (numeric). We might add a key with Mentat group name.

So, should there be only ip4_start, ip4_end (resp. ip6_start, ip6_end)?

Currently, the export looks like like:

{
      "client_id": null,
      "descr": "Second demonstration IPv4 network",
      "netname": "NETNAME2",
      "primary_key": "195.113.144.0/24",
      "rank": null,
      "resolved_abuses": {
        "low": [
          "abuse@bogus-domain.org" 
        ]
      },
      "source": "manual" 
    },

Actions #6

Updated by Pavel Kácha about 1 year ago

Rajmund Hruška wrote in #note-5:

Pavel Kácha wrote in #note-2:

  • Comparing to Negistry export, we don't need "primary_key", "first" and "last" (numeric). We might add a key with Mentat group name.

So, should there be only ip4_start, ip4_end (resp. ip6_start, ip6_end)?

Yes. primary_key is closely tied to RIPE object model, which we don't need, first and last were probably simplifications for dumb clients, but as far as I know, nobody uses them, and if so, they can parse IP data into whatever they want. But foremost, it's the same data duplication. So let's keep only ip4_start, ip4_end, ip6_addr, ip6_prefix.

Actions #7

Updated by Rajmund Hruška about 1 year ago

  • Status changed from New to In Progress
  • Assignee set to Rajmund Hruška
  • % Done changed from 0 to 80
Actions #8

Updated by Rajmund Hruška 12 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 80 to 100
Actions #9

Updated by Rajmund Hruška 11 months ago

  • Status changed from Resolved to In Review
Actions #10

Updated by Rajmund Hruška 4 months ago

  • Status changed from In Review to Closed
Actions

Also available in: Atom PDF