Feature #7653
closedImplement Negistry-like JSON dump API for exporting group/network/abuse contact data
100%
Description
As non-customer groups/networks are now inconsistently managed partially in Negistry, partially in Mentat as "manual" networks, we essentially want to take over management of non-CESNET organisations/networks from Negistry.
In Mentat those are: abuse@casablanca.cz, abuse@cdt.cz, ip@cdt.cz, abuse@upjs.sk
In Negistry: mnt-by VUTBR-MNT, mnt-by VUT-BATCH-MNT
The line of thought is to automate fetching/updating of related data from RIPE as much as we can and make them available to the third party users (OTRS, PV, ...) through API.
For import we are mostly ready as we should be able to use existing mechanism for importing from Negistry (mentat-netmngr.py ... --whois-file source.json --source SRC), where each "set" or "feed" would be imported as separate SOURCE, all this wrapped
What remains is to make group/network information available, preferably as JSON file similar to Negistry's [1], [2], or some simplified amalgamate thereof.
(Remaining three groups in Mentat (GRID, Sner, ubik@cesnet.cz) are CESNET specific, in the long term should end up in NetBox and synced to Mentat by the same (mentat-netmgr) mechanism.)
Updated by Pavel Kácha about 1 year ago
Note from 2023-06-22:
Would be nice to have both full and segmented export, however if coding the segmented one would be to onerous, full is enough - providing the resolution algorithm is documented, unambiguous and all necessary info is in full export.
Updated by Pavel Kácha about 1 year ago
Notes from 2023-06-29:
- We have a bit of collision in "source" field, as the info on the API will overlap with info from Negistry. However, the aim is to make Mentat's API the central hub of group info (to keep Mentat reports, OTRS reports and others aligned), so it seems we could use a source naming convention like today – negistry/ripe_cesnet, and then mentat/ripe_casablanka for example, and just pass source as is.
- This info is not limited for users (API tokens), but for specific machines, so we'd like it semi-public, with access limited only by Apache config (specific IPs, maybe X509). However, if too inconsistent with the rest of Mentat API, it's not a big deal.
- We need to add something to hold CID ("customer id"). Let's keep it general in Mentat (not tied to our CESNET internal names), and use something like "local_id", where Mentat users (as in organisations) can impose their own sense.
- Comparing to Negistry export, we don't need "primary_key", "first" and "last" (numeric). We might add a key with Mentat group name.
Updated by Rajmund Hruška about 1 year ago
- Target version changed from Backlog to 2.12
Updated by Pavel Kácha about 1 year ago
There exist abuse groups, which contain multiple network with different CIDs – so we are not able to reliably assign CID to the group. Main causes are:
- connecting network (2 addresses) – the devices belong to CESNET, so abuse contact is abuse@cesnet.cz, while the network itself is assigned to the organization itself and bears its CID
- subparts of organisations – with their own different CIDs, but the same abuse contact
So back to the drawing board a bit – lets put the extension local_id not only to the group, but also to the networks
Updated by Rajmund Hruška about 1 year ago
Pavel Kácha wrote in #note-2:
- Comparing to Negistry export, we don't need "primary_key", "first" and "last" (numeric). We might add a key with Mentat group name.
So, should there be only ip4_start, ip4_end (resp. ip6_start, ip6_end)?
Currently, the export looks like like:
{
"client_id": null,
"descr": "Second demonstration IPv4 network",
"netname": "NETNAME2",
"primary_key": "195.113.144.0/24",
"rank": null,
"resolved_abuses": {
"low": [
"abuse@bogus-domain.org"
]
},
"source": "manual"
},
Updated by Pavel Kácha about 1 year ago
Rajmund Hruška wrote in #note-5:
Pavel Kácha wrote in #note-2:
- Comparing to Negistry export, we don't need "primary_key", "first" and "last" (numeric). We might add a key with Mentat group name.
So, should there be only ip4_start, ip4_end (resp. ip6_start, ip6_end)?
Yes. primary_key is closely tied to RIPE object model, which we don't need, first and last were probably simplifications for dumb clients, but as far as I know, nobody uses them, and if so, they can parse IP data into whatever they want. But foremost, it's the same data duplication. So let's keep only ip4_start, ip4_end, ip6_addr, ip6_prefix.
Updated by Rajmund Hruška about 1 year ago
- Status changed from New to In Progress
- Assignee set to Rajmund Hruška
- % Done changed from 0 to 80
Updated by Rajmund Hruška about 1 year ago
- Status changed from In Progress to Resolved
- % Done changed from 80 to 100
Updated by Rajmund Hruška about 1 year ago
- Status changed from Resolved to In Review