Bug #7548
closedmentat-enricher.py error in Vagrant box
100%
Description
When I executed mentat-controller.py --command start the following exception occurred.
Traceback (most recent call last):
File "/var/mentat/venv/bin/mentat-enricher.py", line 7, in <module>
exec(compile(f.read(), __file__, 'exec'))
File "/vagrant/bin/mentat-enricher.py", line 66, in <module>
mentat.module.enricher.MentatEnricherDaemon().run()
File "/var/mentat/venv/lib/python3.7/site-packages/pyzenkit/baseapp.py", line 1553, in run
self._stage_setup()
File "/var/mentat/venv/lib/python3.7/site-packages/pyzenkit/baseapp.py", line 1409, in _stage_setup
self._sub_stage_setup()
File "/var/mentat/venv/lib/python3.7/site-packages/pyzenkit/zendaemon.py", line 743, in _sub_stage_setup
component.setup(self)
File "/vagrant/lib/mentat/daemon/component/enricher.py", line 182, in setup
self._setup_plugins(daemon)
File "/vagrant/lib/mentat/daemon/component/enricher.py", line 159, in _setup_plugins
self._setup_plugin(daemon, plugin_conf)
File "/vagrant/lib/mentat/daemon/component/enricher.py", line 147, in _setup_plugin
plugin.setup(daemon, plugin_conf.get('config', None))
File "/vagrant/lib/mentat/plugin/enricher/geoip.py", line 65, in setup
self.geoip_service = geoip_manager.service()
File "/vagrant/lib/mentat/services/geoip.py", line 235, in service
self._service.setup()
File "/vagrant/lib/mentat/services/geoip.py", line 91, in setup
self.asndb = geoip2.database.Reader(self.fn_asndb)
File "/var/mentat/venv/lib/python3.7/site-packages/geoip2/database.py", line 85, in __init__
self._db_reader = maxminddb.open_database(fileish, mode)
File "/var/mentat/venv/lib/python3.7/site-packages/maxminddb/__init__.py", line 41, in open_database
return maxminddb.reader.Reader(database, mode)
File "/var/mentat/venv/lib/python3.7/site-packages/maxminddb/reader.py", line 51, in __init__
with open(database, 'rb') as db_file:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/GeoIP/GeoLite2-ASN.mmdb'
Related issues
Updated by Rajmund Hruška almost 3 years ago
- Related to Task #7041: Use Vagrant as alternative tool for local development added
Updated by Radko Krkoš almost 3 years ago
- To be discussed changed from No to Yes
This is due to the GeoLite2 database license restrictions, as was discussed previously. A possible solution appeared on the mailing list on January 25th 2022 6:16 PM, a free 1:1 replacement. The following links are of interest:
https://github.com/adulau/mmdb-server/blob/main/db/GeoOpen-country.mmd
https://hdoc.csirt-tooling.org/GwZ3IOgPQXWRY5RA4Qm1pQ#
Updated by Jan Mach almost 3 years ago
- Status changed from New to Feedback
- Assignee changed from Jan Mach to Rajmund Hruška
Rajmund Hruška, please have you followed these instructions: https://alchemist.cesnet.cz/mentat/doc/development/html/_doclib/development.html#development-with-vagrant ? I am unable to reproduce your error, perhaps I am doing something wrong. Have you tried vagrant destroy, then vagrant up to rebuild your box from scratch to make sure there are no weird artifacts breaking the provisioning? In another words: does the issue appear, when you try it on freshly cloned repository?
This looks to me like the configuration file for GeoIP was not placed into correct location prior to vagrant up as instructed in the manual and Vagrant is not able to perform the provisioning. By design it does not raise error, because in certain cases you might know, what you are doing.
There can also be an error when trying to build Vagrant box when you have previously used the repository for development the old way. For example everything breaks or Vagrant when toplevel .env file is present, because for previous development workflow it contains chroot path and breaks everything.
Updated by Rajmund Hruška almost 3 years ago
- Assignee changed from Rajmund Hruška to Jan Mach
I tried destroying the vagrant
Jan Mach wrote in #note-3:
Rajmund Hruška, please have you followed these instructions: https://alchemist.cesnet.cz/mentat/doc/development/html/_doclib/development.html#development-with-vagrant ? I am unable to reproduce your error, perhaps I am doing something wrong. Have you tried
vagrant destroy, thenvagrant upto rebuild your box from scratch to make sure there are no weird artifacts breaking the provisioning? In another words: does the issue appear, when you try it on freshly cloned repository?This looks to me like the configuration file for GeoIP was not placed into correct location prior to
vagrant upas instructed in the manual and Vagrant is not able to perform the provisioning. By design it does not raise error, because in certain cases you might know, what you are doing.There can also be an error when trying to build Vagrant box when you have previously used the repository for development the old way. For example everything breaks or Vagrant when toplevel
.envfile is present, because for previous development workflow it contains chroot path and breaks everything.
I followed the mentioned instructions. I tried vagrant destroy then vagrant up and it seems to be working now.
Updated by Rajmund Hruška almost 3 years ago
- To be discussed changed from No to Yes
I deleted the virtual machine and the cloned repository. Then I cloned the repository again, I switched to devel, edited Makefile where I put the full address of mentat-alt server and I executed make data-fetch-config. Afterwards, I executed vagrant up followed by vagrant ssh. I switched to Mentat user by sm and entered the virtual environment by ve. Finally, I executed mentat-controller.py --command start and the same error appeared on my screen.
Then after recreating the virtual machine by vagrant destroy and vagrant up, mentat-controller.py --command start works fine.
Updated by Jan Mach almost 3 years ago
- Status changed from Feedback to In Progress
- Target version changed from Backlog to 2.9
- % Done changed from 0 to 50
- To be discussed changed from Yes to No
Rajmund Hruska wrote in #note-6:
I deleted the virtual machine and the cloned repository. Then I cloned the repository again, I switched to
devel, editedMakefilewhere I put the full address ofmentat-altserver and I executedmake data-fetch-config. Afterwards, I executedvagrant upfollowed byvagrant ssh. I switched to Mentat user bysmand entered the virtual environment byve. Finally, I executedmentat-controller.py --command startand the same error appeared on my screen.Then after recreating the virtual machine by
vagrant destroyandvagrant up,mentat-controller.py --command startworks fine.
I see the problem. The geoipupdate tool for vagrant is downloading files to different directory than for which the mentat-enricher is configured.
Updated by Jan Mach almost 3 years ago
- Status changed from In Progress to In Review
- % Done changed from 50 to 100
Attached commit resolves the issue. There was a problem with installation of geoipupdate tool, which removed prepared directory symlink. As a result database files were installed into different directory and mentat-enricher was unable to find them.