Project

General

Profile

Actions
Copy link

Bug #7548

closed

mentat-enricher.py error in Vagrant box

Added by Rajmund Hruška almost 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Jan Mach
Category:
-
Target version:
2.9
Start date:
01/27/2022
Due date:
% Done:

100%

Estimated time:
To be discussed:
No

Description

When I executed mentat-controller.py --command start the following exception occurred.

Traceback (most recent call last):
  File "/var/mentat/venv/bin/mentat-enricher.py", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/vagrant/bin/mentat-enricher.py", line 66, in <module>
    mentat.module.enricher.MentatEnricherDaemon().run()
  File "/var/mentat/venv/lib/python3.7/site-packages/pyzenkit/baseapp.py", line 1553, in run
    self._stage_setup()
  File "/var/mentat/venv/lib/python3.7/site-packages/pyzenkit/baseapp.py", line 1409, in _stage_setup
    self._sub_stage_setup()
  File "/var/mentat/venv/lib/python3.7/site-packages/pyzenkit/zendaemon.py", line 743, in _sub_stage_setup
    component.setup(self)
  File "/vagrant/lib/mentat/daemon/component/enricher.py", line 182, in setup
    self._setup_plugins(daemon)
  File "/vagrant/lib/mentat/daemon/component/enricher.py", line 159, in _setup_plugins
    self._setup_plugin(daemon, plugin_conf)
  File "/vagrant/lib/mentat/daemon/component/enricher.py", line 147, in _setup_plugin
    plugin.setup(daemon, plugin_conf.get('config', None))
  File "/vagrant/lib/mentat/plugin/enricher/geoip.py", line 65, in setup
    self.geoip_service = geoip_manager.service()
  File "/vagrant/lib/mentat/services/geoip.py", line 235, in service
    self._service.setup()
  File "/vagrant/lib/mentat/services/geoip.py", line 91, in setup
    self.asndb = geoip2.database.Reader(self.fn_asndb)
  File "/var/mentat/venv/lib/python3.7/site-packages/geoip2/database.py", line 85, in __init__
    self._db_reader = maxminddb.open_database(fileish, mode)
  File "/var/mentat/venv/lib/python3.7/site-packages/maxminddb/__init__.py", line 41, in open_database
    return maxminddb.reader.Reader(database, mode)
  File "/var/mentat/venv/lib/python3.7/site-packages/maxminddb/reader.py", line 51, in __init__
    with open(database, 'rb') as db_file:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/GeoIP/GeoLite2-ASN.mmdb'

Related issues

Related to Mentat - Task #7041: Use Vagrant as alternative tool for local developmentClosedJan Mach02/12/2021

Actions
Actions
Copy link
 #1

Updated by Rajmund Hruška almost 3 years ago

  • Related to Task #7041: Use Vagrant as alternative tool for local development added
Actions
Copy link
 #2

Updated by Radko Krkoš almost 3 years ago

  • To be discussed changed from No to Yes

This is due to the GeoLite2 database license restrictions, as was discussed previously. A possible solution appeared on the mailing list on January 25th 2022 6:16 PM, a free 1:1 replacement. The following links are of interest:
https://github.com/adulau/mmdb-server/blob/main/db/GeoOpen-country.mmd
https://hdoc.csirt-tooling.org/GwZ3IOgPQXWRY5RA4Qm1pQ#

Actions
Copy link
 #3

Updated by Jan Mach almost 3 years ago

  • Status changed from New to Feedback
  • Assignee changed from Jan Mach to Rajmund Hruška

Rajmund Hruška, please have you followed these instructions: https://alchemist.cesnet.cz/mentat/doc/development/html/_doclib/development.html#development-with-vagrant ? I am unable to reproduce your error, perhaps I am doing something wrong. Have you tried vagrant destroy, then vagrant up to rebuild your box from scratch to make sure there are no weird artifacts breaking the provisioning? In another words: does the issue appear, when you try it on freshly cloned repository?

This looks to me like the configuration file for GeoIP was not placed into correct location prior to vagrant up as instructed in the manual and Vagrant is not able to perform the provisioning. By design it does not raise error, because in certain cases you might know, what you are doing.

There can also be an error when trying to build Vagrant box when you have previously used the repository for development the old way. For example everything breaks or Vagrant when toplevel .env file is present, because for previous development workflow it contains chroot path and breaks everything.

Actions
Copy link
 #4

Updated by Rajmund Hruška almost 3 years ago

  • Assignee changed from Rajmund Hruška to Jan Mach

I tried destroying the vagrant

Jan Mach wrote in #note-3:

Rajmund Hruška, please have you followed these instructions: https://alchemist.cesnet.cz/mentat/doc/development/html/_doclib/development.html#development-with-vagrant ? I am unable to reproduce your error, perhaps I am doing something wrong. Have you tried vagrant destroy, then vagrant up to rebuild your box from scratch to make sure there are no weird artifacts breaking the provisioning? In another words: does the issue appear, when you try it on freshly cloned repository?

This looks to me like the configuration file for GeoIP was not placed into correct location prior to vagrant up as instructed in the manual and Vagrant is not able to perform the provisioning. By design it does not raise error, because in certain cases you might know, what you are doing.

There can also be an error when trying to build Vagrant box when you have previously used the repository for development the old way. For example everything breaks or Vagrant when toplevel .env file is present, because for previous development workflow it contains chroot path and breaks everything.

I followed the mentioned instructions. I tried vagrant destroy then vagrant up and it seems to be working now.

Actions
Copy link
 #5

Updated by Jan Mach almost 3 years ago

  • To be discussed changed from Yes to No
Actions
Copy link
 #6

Updated by Rajmund Hruška almost 3 years ago

  • To be discussed changed from No to Yes

I deleted the virtual machine and the cloned repository. Then I cloned the repository again, I switched to devel, edited Makefile where I put the full address of mentat-alt server and I executed make data-fetch-config. Afterwards, I executed vagrant up followed by vagrant ssh. I switched to Mentat user by sm and entered the virtual environment by ve. Finally, I executed mentat-controller.py --command start and the same error appeared on my screen.

Then after recreating the virtual machine by vagrant destroy and vagrant up, mentat-controller.py --command start works fine.

Actions
Copy link
 #7

Updated by Jan Mach almost 3 years ago

  • Status changed from Feedback to In Progress
  • Target version changed from Backlog to 2.9
  • % Done changed from 0 to 50
  • To be discussed changed from Yes to No

Rajmund Hruska wrote in #note-6:

I deleted the virtual machine and the cloned repository. Then I cloned the repository again, I switched to devel, edited Makefile where I put the full address of mentat-alt server and I executed make data-fetch-config. Afterwards, I executed vagrant up followed by vagrant ssh. I switched to Mentat user by sm and entered the virtual environment by ve. Finally, I executed mentat-controller.py --command start and the same error appeared on my screen.

Then after recreating the virtual machine by vagrant destroy and vagrant up, mentat-controller.py --command start works fine.

I see the problem. The geoipupdate tool for vagrant is downloading files to different directory than for which the mentat-enricher is configured.

Actions
Copy link
 #8

Updated by Jan Mach almost 3 years ago

  • Status changed from In Progress to In Review
  • % Done changed from 50 to 100

Attached commit resolves the issue. There was a problem with installation of geoipupdate tool, which removed prepared directory symlink. As a result database files were installed into different directory and mentat-enricher was unable to find them.

Actions
Copy link
 #9

Updated by Pavel Kácha almost 3 years ago

  • Status changed from In Review to Closed
Actions
Copy link

Also available in: Atom PDF