Project

General

Profile

Actions

Feature #7439

open

Remove FileWhoisModule from whois service

Added by Rajmund Hruska about 2 months ago. Updated about 1 month ago.

Status:
In Progress
Priority:
Normal
Category:
-
Target version:
Start date:
06/18/2021
Due date:
% Done:

0%

Estimated time:
To be discussed:
No

Description

Currently, the whois service is able to use 2 modules - SqldbWhoisModule and FileWhoisModule. Using the FileWhoisModule might result in resolving abuse groups and networks which are not known (not stored in the database).

Also, the whois module returns the resolved abuses as a simple list of strings. The return type should be more complex, e. g. mentat.datatype.sqldb.NetworkModel.

Actions #1

Updated by Rajmund Hruska about 2 months ago

  • Status changed from New to In Progress
  • To be discussed changed from No to Yes

I checked the configuration file for the whois service and I found out that this module uses /var/mentat/whois-exceptions.json file. I looked at the mentat-alt server and it seems that this file contains quite a lot of networks. What should be done with those networks?

Actions #2

Updated by Rajmund Hruska about 1 month ago

I tried resolving networks from /var/mentat/whois-exceptions.json with the new data from Negistry. All of the networks were successfully resolved, but most of the resolved abuses differ. For example in /var/mentat/whois-exceptions.json IP address 195.113.87.41 has resolved abuse '' but in the new data from Negistry the resolved abuses are '', '' and ''.

So, the /var/mentat/whois-exceptions.json can be replaced by using the new data from Negistry provided that the resolved abuses using data from Negistry are correct.

Actions #3

Updated by Rajmund Hruska about 1 month ago

  • To be discussed changed from Yes to No

From the 2021-06-24 call:

The data from /var/mentat/whois-exceptions.json seems to be outdated. There will be new information coming from Negistry after #6227 is implemented so it is OK to remove FileWhoisModule.

Actions #4

Updated by Rajmund Hruska about 1 month ago

  • To be discussed changed from No to Yes

While checking lib/mentat/module/netmngr.py in order to finish #6227 I noticed that this module also works with the whois exceptions. In the documentation there is written this:

The exception file is an ordinary text file containing single IPv(4|6)
address|network|range on each line. Blank lines and lines beginning with ``#``
are ignored. Whois exception files are very easy to be generated and they are meant
for specifying whois resolving exceptions. For example you may use it to describe
hosts with addresses from the domain of one particular abuse group, but actually
belonging to different group. This might be the case of routers belonging to service
provider but residing within the network address space of the customer. Another
example may be nodes of some cloud computing service that have addresses from
address space of the cloud computing organization member.

The new Mentat resolving (#6227) takes rank of the network into consideration so the networks can be stored in a normal whois file with higher rank. The other option is adding those network records manually.

Should I let the netmngr.py be as it is or should I change also this module? I would say that the feature of whois exceptions is no longer needed.

Actions #5

Updated by Pavel Kácha about 1 month ago

Rajmund Hruska wrote in #note-4:

Should I let the netmngr.py be as it is or should I change also this module? I would say that the feature of whois exceptions is no longer needed.

I checked the whois exception file - lots of those exceptions are outright wrong (attributed to cuni.cz), which was probably typo when generating this file, however as majority of this file are errors, I guess there's nothing to lose to stop using it. I've removed it from both mentat-alt and mentat-hub.

If it makes things simpler or makes code changes simpler/more safe, I'm all for full remove.

Actions #6

Updated by Rajmund Hruska about 1 month ago

  • To be discussed changed from Yes to No

Pavel Kácha wrote in #note-5:

Rajmund Hruska wrote in #note-4:

Should I let the netmngr.py be as it is or should I change also this module? I would say that the feature of whois exceptions is no longer needed.

I checked the whois exception file - lots of those exceptions are outright wrong (attributed to cuni.cz), which was probably typo when generating this file, however as majority of this file are errors, I guess there's nothing to lose to stop using it. I've removed it from both mentat-alt and mentat-hub.

If it makes things simpler or makes code changes simpler/more safe, I'm all for full remove.

I think it will make code simpler, so I will remove it from netmngr.py.

Actions

Also available in: Atom PDF