Project

General

Profile

Actions

Feature #7439

open

Remove FileWhoisModule from whois service

Added by Rajmund Hruska over 1 year ago. Updated 2 months ago.

Status:
In Progress
Priority:
Normal
Category:
-
Target version:
Start date:
06/18/2021
Due date:
% Done:

90%

Estimated time:
To be discussed:
Yes

Description

Currently, the whois service is able to use 2 modules - SqldbWhoisModule and FileWhoisModule. Using the FileWhoisModule might result in resolving abuse groups and networks which are not known (not stored in the database).

Also, the whois module returns the resolved abuses as a simple list of strings. The return type should be more complex, e. g. mentat.datatype.sqldb.NetworkModel.


Files

7439_profile.txt (54.2 KB) 7439_profile.txt Rajmund Hruska, 01/12/2022 05:13 PM
Actions #1

Updated by Rajmund Hruska over 1 year ago

  • Status changed from New to In Progress
  • To be discussed changed from No to Yes

I checked the configuration file for the whois service and I found out that this module uses /var/mentat/whois-exceptions.json file. I looked at the mentat-alt server and it seems that this file contains quite a lot of networks. What should be done with those networks?

Actions #2

Updated by Rajmund Hruska over 1 year ago

I tried resolving networks from /var/mentat/whois-exceptions.json with the new data from Negistry. All of the networks were successfully resolved, but most of the resolved abuses differ. For example in /var/mentat/whois-exceptions.json IP address 195.113.87.41 has resolved abuse '' but in the new data from Negistry the resolved abuses are '', '' and ''.

So, the /var/mentat/whois-exceptions.json can be replaced by using the new data from Negistry provided that the resolved abuses using data from Negistry are correct.

Actions #3

Updated by Rajmund Hruska over 1 year ago

  • To be discussed changed from Yes to No

From the 2021-06-24 call:

The data from /var/mentat/whois-exceptions.json seems to be outdated. There will be new information coming from Negistry after #6227 is implemented so it is OK to remove FileWhoisModule.

Actions #4

Updated by Rajmund Hruska over 1 year ago

  • To be discussed changed from No to Yes

While checking lib/mentat/module/netmngr.py in order to finish #6227 I noticed that this module also works with the whois exceptions. In the documentation there is written this:

The exception file is an ordinary text file containing single IPv(4|6)
address|network|range on each line. Blank lines and lines beginning with ``#``
are ignored. Whois exception files are very easy to be generated and they are meant
for specifying whois resolving exceptions. For example you may use it to describe
hosts with addresses from the domain of one particular abuse group, but actually
belonging to different group. This might be the case of routers belonging to service
provider but residing within the network address space of the customer. Another
example may be nodes of some cloud computing service that have addresses from
address space of the cloud computing organization member.

The new Mentat resolving (#6227) takes rank of the network into consideration so the networks can be stored in a normal whois file with higher rank. The other option is adding those network records manually.

Should I let the netmngr.py be as it is or should I change also this module? I would say that the feature of whois exceptions is no longer needed.

Actions #5

Updated by Pavel Kácha over 1 year ago

Rajmund Hruska wrote in #note-4:

Should I let the netmngr.py be as it is or should I change also this module? I would say that the feature of whois exceptions is no longer needed.

I checked the whois exception file - lots of those exceptions are outright wrong (attributed to cuni.cz), which was probably typo when generating this file, however as majority of this file are errors, I guess there's nothing to lose to stop using it. I've removed it from both mentat-alt and mentat-hub.

If it makes things simpler or makes code changes simpler/more safe, I'm all for full remove.

Actions #6

Updated by Rajmund Hruska over 1 year ago

  • To be discussed changed from Yes to No

Pavel Kácha wrote in #note-5:

Rajmund Hruska wrote in #note-4:

Should I let the netmngr.py be as it is or should I change also this module? I would say that the feature of whois exceptions is no longer needed.

I checked the whois exception file - lots of those exceptions are outright wrong (attributed to cuni.cz), which was probably typo when generating this file, however as majority of this file are errors, I guess there's nothing to lose to stop using it. I've removed it from both mentat-alt and mentat-hub.

If it makes things simpler or makes code changes simpler/more safe, I'm all for full remove.

I think it will make code simpler, so I will remove it from netmngr.py.

Actions #7

Updated by Rajmund Hruska 12 months ago

  • Status changed from In Progress to Feedback
  • % Done changed from 0 to 90
  • To be discussed changed from No to Yes

I removed the support of whois-exceptions from mentat-netmngr.py module.

I also changed the whois service, so now it returns mentat.internal.sqldb.NetworkModel objects instead of mentat.datatype.internal.NetworkRecord. Using whois/search in hawat seems to be twice as slow now, though. I feel like there is some room for improvements as I only tried to make as few changes as possible.

Actions #8

Updated by Pavel Kácha 11 months ago

Rajmund Hruska wrote in #note-7:

I removed the support of whois-exceptions from mentat-netmngr.py module.

I also changed the whois service, so now it returns mentat.internal.sqldb.NetworkModel objects instead of mentat.datatype.internal.NetworkRecord. Using whois/search in hawat seems to be twice as slow now, though. I feel like there is some room for improvements as I only tried to make as few changes as possible.

It might be wise to run at least some basic profile or time check, to find out what's the culprit. It might be db, but it also might be class instantiations or some method unexpected to be called too often. However, if twice is in order of seconds, I wouldn't bother too much.

Actions #9

Updated by Rajmund Hruska 11 months ago

I ran the profiler and I found out that the most time consuming part is indeed the initialization of whois service. I generated a couple hundreds of events and then ran the reporting script and both seem to be reasonably fast. So the only slow place seems to be whois module in the (hawat) web interface.

Actions #10

Updated by Rajmund Hruska 10 months ago

  • Status changed from Feedback to In Progress
  • To be discussed deleted (Yes)
Actions #11

Updated by Pavel Kácha 4 months ago

Rajmund Hruska wrote in #note-9:

I ran the profiler and I found out that the most time consuming part is indeed the initialization of whois service. I generated a couple hundreds of events and then ran the reporting script and both seem to be reasonably fast. So the only slow place seems to be whois module in the (hawat) web interface.

As this lingers here quite long, here's a thought (feel free to correct me where I'm wrong in assumptions or conclusions):

  • we now have only one "source of truth" about networks - network database. No RIPE, no obscure JSON file.
  • we have quite consistent interface for searching through network database at /mentat/networks/list
  • if we extend this search to be able to show supersets of ips/networks/ranges (which would be lovely anyway), we would get pretty much the same functionality, as is provided by /mentat/whois/search
  • and we could get rid of /mentat/whois specific code completely.
Actions #12

Updated by Rajmund Hruska 2 months ago

  • To be discussed set to Yes

Ye

Pavel Kácha wrote in #note-11:

Rajmund Hruska wrote in #note-9:

I ran the profiler and I found out that the most time consuming part is indeed the initialization of whois service. I generated a couple hundreds of events and then ran the reporting script and both seem to be reasonably fast. So the only slow place seems to be whois module in the (hawat) web interface.

As this lingers here quite long, here's a thought (feel free to correct me where I'm wrong in assumptions or conclusions):

  • we now have only one "source of truth" about networks - network database. No RIPE, no obscure JSON file.
  • we have quite consistent interface for searching through network database at /mentat/networks/list
  • if we extend this search to be able to show supersets of ips/networks/ranges (which would be lovely anyway), we would get pretty much the same functionality, as is provided by /mentat/whois/search
  • and we could get rid of /mentat/whois specific code completely.

Yes, I totally agree. In addition, this would simplify both the GUI and the codebase.

Actions #13

Updated by Pavel Kácha 2 months ago

Just a note - I guess it's used in the popup of IP addresses in reports and in event view, and I'm not sure whether networks list data are available to common Mentat users.

Actions

Also available in: Atom PDF