Mentat

I checked the configuration file for the whois service and I found out that this module uses /var/mentat/whois-exceptions.json file. I looked at the mentat-alt server and it seems that this file contains quite a lot of networks. What should be done with those networks?

From the 2021-06-24 call:

The data from /var/mentat/whois-exceptions.json seems to be outdated. There will be new information coming from Negistry after #6227 is implemented so it is OK to remove FileWhoisModule.

While checking lib/mentat/module/netmngr.py in order to finish #6227 I noticed that this module also works with the whois exceptions. In the documentation there is written this:

The exception file is an ordinary text file containing single IPv(4|6)
address|network|range on each line. Blank lines and lines beginning with ``#``
are ignored. Whois exception files are very easy to be generated and they are meant
for specifying whois resolving exceptions. For example you may use it to describe
hosts with addresses from the domain of one particular abuse group, but actually
belonging to different group. This might be the case of routers belonging to service
provider but residing within the network address space of the customer. Another
example may be nodes of some cloud computing service that have addresses from
address space of the cloud computing organization member.

The new Mentat resolving (#6227) takes rank of the network into consideration so the networks can be stored in a normal whois file with higher rank. The other option is adding those network records manually.

Should I let the netmngr.py be as it is or should I change also this module? I would say that the feature of whois exceptions is no longer needed.

Pavel Kácha wrote in #note-5:

Rajmund Hruska wrote in #note-4:

Should I let the netmngr.py be as it is or should I change also this module? I would say that the feature of whois exceptions is no longer needed.

I checked the whois exception file - lots of those exceptions are outright wrong (attributed to cuni.cz), which was probably typo when generating this file, however as majority of this file are errors, I guess there's nothing to lose to stop using it. I've removed it from both mentat-alt and mentat-hub.

If it makes things simpler or makes code changes simpler/more safe, I'm all for full remove.

I think it will make code simpler, so I will remove it from netmngr.py.

I removed the support of whois-exceptions from mentat-netmngr.py module.

I also changed the whois service, so now it returns mentat.internal.sqldb.NetworkModel objects instead of mentat.datatype.internal.NetworkRecord. Using whois/search in hawat seems to be twice as slow now, though. I feel like there is some room for improvements as I only tried to make as few changes as possible.

I ran the profiler and I found out that the most time consuming part is indeed the initialization of whois service. I generated a couple hundreds of events and then ran the reporting script and both seem to be reasonably fast. So the only slow place seems to be whois module in the (hawat) web interface.

Ye

Pavel Kácha wrote in #note-11:

Rajmund Hruska wrote in #note-9:

I ran the profiler and I found out that the most time consuming part is indeed the initialization of whois service. I generated a couple hundreds of events and then ran the reporting script and both seem to be reasonably fast. So the only slow place seems to be whois module in the (hawat) web interface.

As this lingers here quite long, here's a thought (feel free to correct me where I'm wrong in assumptions or conclusions):

• we now have only one "source of truth" about networks - network database. No RIPE, no obscure JSON file.
• we have quite consistent interface for searching through network database at /mentat/networks/list
• if we extend this search to be able to show supersets of ips/networks/ranges (which would be lovely anyway), we would get pretty much the same functionality, as is provided by /mentat/whois/search
• and we could get rid of /mentat/whois specific code completely.

Yes, I totally agree. In addition, this would simplify both the GUI and the codebase.