Task #7041: Use Vagrant as alternative tool for local development - Mentat - Homeproj: Redmine for CESNET

Actions

Copy link

#1

Updated by Jan Mach almost 4 years ago

Status changed from New to In Progress
Assignee changed from Jan Mach to Mentat - Core

Redmine behavior test.

Actions

Copy link

#2

Updated by Jan Mach almost 4 years ago

Assignee changed from Mentat - Core to Jan Mach

Redmine behavior test.

Actions

Copy link

#3

Updated by Jan Mach over 3 years ago

Subject changed from Use Docker as alternative tool for local development to Use Vagrant as alternative tool for local development
% Done changed from 0 to 50
To be discussed changed from No to Yes

After much work and consideration I have concluded, that it will be much better to use Vagrant and therefore full featured virtual machine for local Mentat development due to the complexity of the whole environment. It would be somewhat possible to do this with Docker, as was originally intended, but the result would be either completely hacked and abused Docker concept, or really complex network of multiple Docker containers.

First commit introduces working, but unpolished and somewhat rough on the edges proof of concept for setting up and using development environment. There is still lots of work, but I am confident the result will be much more pleasant to work with and as a side effect many utility scripts will receive another look and rework.

Actions

Copy link

#4

Updated by Jan Mach over 3 years ago

To be discussed changed from Yes to No

Actions

Copy link

#5

Updated by Jan Mach over 3 years ago

Target version changed from 2.8 to 2.9

Actions

Copy link

#6

Updated by Jan Mach about 3 years ago

% Done changed from 50 to 80
To be discussed changed from No to Yes

In repository is a branch mach-task-#7041:"https://homeproj.cesnet.cz/projects/mentat/repository/mentat-ng?utf8=%E2%9C%93&rev=mach-task-%237041", in which you may now preview and test local development using prepared Vagrant box. In documentation file doc/sphinx/_doclib/development.rst there is a paragraph "Development with Vagrant" with basic how to. Any feedback will be much appreciated.

Actions

Copy link

#7

Updated by Jan Mach about 3 years ago

Documentation regarding development with Vagrant grows slowly here: doc/sphinx/_doclib/development.rst.

Vagrant up command takes roughly 15 mins, mostly due to the provisioning of the database (download time of the snapshot from mentat-alt is not included, because it is not necessary to do it every time). This can be improved by making the development snapshot of the database smaller.

Actions

Copy link

#8

Updated by Jan Mach about 3 years ago

Confirmed to work with VirtualBox 6.1.26 and Vagrant 2.2.18.

Actions

Copy link

#9

Updated by Pavel Kácha about 3 years ago

Notes from today's meeting:

~~please, do not incorporate (un)relocation patches just yet, make notes - separate issue, perhaps?~~
we would like parts of this work to be reusable on integration testing (Mek: noted)
if possible, test also with Docker backend (as GitLab uses Docker as default)
~~put in only a handful of necessary fixtures, make import from another live systems available on explicit action~~
- or maybe no fixtures yet, just empty functioning system, and let us make this small "devel" fixture set gradually during integration testing implementation? I mean - it does not have to be within scope of this issue, just make the wiring ready and useful contents will grow based on later needs.
everybody: please test (and note functioning versions of Vagrand and VirtualBox)

Actions

Copy link

#10

Updated by Radko Krkoš about 3 years ago

Pavel Kácha wrote in #note-9:

(thinking after meeting:) or maybe no fixtures yet, just empty functioning system, and let us make this small "devel" fixture set gradually during integration testing implementation? I mean - it does not have to be within scope of this issue, just make the wiring ready and useful contents will grow based on later needs.

I concur this.

everybody: please test (and note functioning versions of Vagrand and VirtualBox)

Tested working:
VirtualBox 6.1.26 with Vagrant 2.2.15
Tested not-working:
VirtualBox 6.1.26 with Vagrant 1.8.7
VirtualBox 6.1.22 with Vagrant 1.8.7

Maybe we could create a table in wiki? I will look into that.

Actions

Copy link

#11

Updated by Radko Krkoš about 3 years ago

Radko Krkoš wrote in #note-10:

Maybe we could create a table in wiki? I will look into that.

Here is the result matrix: Vagrant_VirtualBox. All previously mentioned results were added.

Actions

Copy link

#12

Updated by Jan Mach about 3 years ago

Summary of drawbacks of using current opproach with lightweight chroot and maximal application portability:

Configuration files for Babel translations need to be generated to provide the necessary portability. Opposing argument is, that this is not relevant, because babel is misused in this use case anyway and the translations of report contents should be handled differently.
There are many text (JSON) configuration files, which need a separate copy in case of local development within the Git repository and deployment on the server. It is untrivial matter to keep all instances of all configuration files in sync. Currently this is resolved by using .env files, but this solution brings is too magical to be clear.
...

Actions

Copy link

#13

Updated by Jan Mach about 3 years ago

I have just merged this branch into devel. It was possible, because I have successfully removed all breaking changes. Now the new Vagrant configurations are just non breaking additions. This means, that now it will be possible to test the new Vagrant development workflow more easily.

Actions

Copy link

#14

Updated by Jan Mach about 3 years ago

Pavel, I have a heretic question regarding installation of Warden client on development Vagrant box. Current provisioning procedure already requires to fetch configuration file /etc/GeoIP.conf from mentat-alt, so that the developer does not have to register account on MaxMind site. Without this configuration file the provisioning succeeds, but system is not usable right away, because working geolocation service is a prerequisite. That`s why I opted to download it from mentat-alt. To clarify, this download is not part of the provisioning script, it is a separate step before vagrant up, a prerequisite.

And now for my question. Would you consider using same approach for warden client? Would it be possible to download a prepared set of certificate and key for warden client and use the same pair as a default for all developers on all Vagrant boxes? This would further greatly simplify the installation process. Of course I would suggest to implement it as optional feature, otherwise the Vagrant box would be too much tied to CESNET and Warden.

Actions

Copy link

#15

Updated by Jan Mach about 3 years ago

Pavel, another note: Docker backend is currently not available for debian/contrib-buster64 Vagrant box, so I am not able to perform the testing you requested.

Actions

Copy link

#16

Updated by Jan Mach about 3 years ago

Status changed from In Progress to Feedback
Assignee changed from Jan Mach to Radko Krkoš

While working on scripts for dumping database for development and importing it back later I have come across interesting issue, that have multiple solutions and I would like to brainstorm it with other developers and most importantly with Radko.

The issue lies in different ordering of columns on Vagrant box and our production and staging servers. This is caused by the fact, that when the Vagrant box is provisioned the database schema is created immediately to the latest state, while the production and staging servers are running and therefore database schema is maintained by migrations. Newly added columns are appended to the end of the table.

Currently I am using equivalent of the following command to do the time based dump of some unnecessarily big tables like reports and then import it back:

sudo -u mentat psql mentat_main -c "COPY (SELECT * FROM ${table_name} WHERE createtime >= '${BACKUP_DATE_FROM}'::timestamptz) TO stdout" > ${BACKUP_DIR}/${table_name}.dat
sudo -u postgres psql mentat_main -c "COPY FROM stdin;" < /vagrant/data/devsnapshot/chunks/reports_events.dat

I am using COPY command, because to my knowledge pg_dump does not support performing WHERE on the dumped tables.

I have considered following solutions, but I do not like any of them:

Change ordering of the columns on our servers. PostgreSQL does not support this natively, but there are workarounds, for example by creating new table and copying data between them, then dropping the former and renaming the latter. This however is not permanent fix, another migration can break it again.
List the columns in the select, so that they are in order expected for Vagrant box. This however would need to be maintained every time we add or remove column.
Use different dumping and importing mechanism that supports where to restrict the number of fetched rows by createtime.

I would like to kindly ask Radko for help with this, I am sure he has some neat trick right up his sleeve.

Side note: It is currently possible to run /etc/mentat/scripts/devsnapshot.sh script on mentat-alt server to prepare fresh snapshot and then experiment with this from within the Vagrant box with command make data-import-dbsnapshot, which currently fails due to the error discussed above.

!DEV! vagrant@mentat-devel /vagrant $ sudo -u postgres psql mentat_main -c "COPY reports_events FROM stdin;" < /vagrant/data/devsnapshot/chunks/reports_events.dat
ERROR:  invalid input syntax for type integer: "2021-10-26 04:40:03.114034" 
CONTEXT:  COPY reports_events, line 1, column id: "2021-10-26 04:40:03.114034"

Actions

Copy link

#17

Updated by Radko Krkoš about 3 years ago

Assignee changed from Radko Krkoš to Jan Mach

Jan Mach wrote in #note-16:

While working on scripts for dumping database for development and importing it back later I have come across interesting issue, that have multiple solutions and I would like to brainstorm it with other developers and most importantly with Radko.

The issue lies in different ordering of columns on Vagrant box and our production and staging servers. This is caused by the fact, that when the Vagrant box is provisioned the database schema is created immediately to the latest state, while the production and staging servers are running and therefore database schema is maintained by migrations. Newly added columns are appended to the end of the table.

Currently I am using equivalent of the following command to do the time based dump of some unnecessarily big tables like reports and then import it back:

I am using COPY command, because to my knowledge pg_dump does not support performing WHERE on the dumped tables.

I have considered following solutions, but I do not like any of them:

Change ordering of the columns on our servers. PostgreSQL does not support this natively, but there are workarounds, for example by creating new table and copying data between them, then dropping the former and renaming the latter. This however is not permanent fix, another migration can break it again.

List the columns in the select, so that they are in order expected for Vagrant box. This however would need to be maintained every time we add or remove column.

Use different dumping and importing mechanism that supports where to restrict the number of fetched rows by createtime.

I would like to kindly ask Radko for help with this, I am sure he has some neat trick right up his sleeve.

Yes, this is a tricky one. I see two possible solutions. First one is organizational: always add the new columns at the end when changing schema. Of course that has enforcement problems and certainly cannot help now.
As for your list, number two is the most versatile and universally accepted. The drawback of maintenance could be lifted as it is possible to get the existing ordering from the source table and exploit that COPY allows to define the column ordering even in its FROM variant. Therefore I would go with something like this:

sudo -u mentat psql mentat_main -c "COPY (SELECT string_agg(data.column_name, ',') FROM (SELECT column_name FROM information_schema.columns WHERE table_name = '${table_name}' ORDER BY ordinal_position) AS data) TO stdout" > ${BACKUP_DIR}/${table_name}.cols
sudo -u mentat psql mentat_main -c "COPY (SELECT * FROM ${table_name} WHERE createtime >= '${BACKUP_DATE_FROM}'::timestamptz) TO stdout" > ${BACKUP_DIR}/${table_name}.dat

sudo -u postgres psql mentat_main -c "COPY ${table_name} ($(cat ${BACKUP_DIR}/${table_name}.cols)) FROM stdin;" < ${BACKUP_DIR}/${table_name}.dat

Is that what you were looking for?

As a side note, my first idea was to define a VIEW with the WHERE filtering and use pg_dump on that (so a variant of option 3), but the tool seems to handle views differently and only export their definition (what kind of makes sense), so that is a no go.

Actions

Copy link

#18

Updated by Jan Mach about 3 years ago

Status changed from Feedback to In Progress
% Done changed from 80 to 90

Radko Krkoš wrote in #note-17:

Yes, this is a tricky one. I see two possible solutions. First one is organizational: always add the new columns at the end when changing schema. Of course that has enforcement problems and certainly cannot help now.
As for your list, number two is the most versatile and universally accepted. The drawback of maintenance could be lifted as it is possible to get the existing ordering from the source table and exploit that COPY allows to define the column ordering even in its FROM variant. Therefore I would go with something like this:

[...]

Is that what you were looking for?

As a side note, my first idea was to define a VIEW with the WHERE filtering and use pg_dump on that (so a variant of option 3), but the tool seems to handle views differently and only export their definition (what kind of makes sense), so that is a no go.

Thank you, that is perfect! Again, you didn't disappoint.

Actions

Copy link

#19

Updated by Jan Mach about 3 years ago

Note from today`s meeting:

Make sure Vagrant provisioning is usable for users without access to our mentat-alt server.
Create separate certificate/key pair to Warden server and use it for our core developers to enable easy access to life events.

Actions

Copy link

#20

Updated by Rajmund Hruška about 3 years ago

Jan Mach wrote in #note-13:

I have just merged this branch into devel. It was possible, because I have successfully removed all breaking changes. Now the new Vagrant configurations are just non breaking additions. This means, that now it will be possible to test the new Vagrant development workflow more easily.

I am not sure if something changed since this comment, but I have a trouble launching mentat-controller.py --command start. Apparently, now it requires user account mentat. Here is the error message - CRITICAL ERROR: "Requested unknown user account 'mentat'". When I create user mentat, I have some issues with permissions. Changing permissions of mentat-ng repository is not enough.

Actions

Copy link

#21

Updated by Rajmund Hruška about 3 years ago

Rajmund Hruska wrote in #note-20:

Jan Mach wrote in #note-13:

I have just merged this branch into devel. It was possible, because I have successfully removed all breaking changes. Now the new Vagrant configurations are just non breaking additions. This means, that now it will be possible to test the new Vagrant development workflow more easily.

I am not sure if something changed since this comment, but I have a trouble launching mentat-controller.py --command start. Apparently, now it requires user account mentat. Here is the error message - CRITICAL ERROR: "Requested unknown user account 'mentat'". When I create user mentat, I have some issues with permissions. Changing permissions of mentat-ng repository is not enough.

This is actually related to #7121 and not this issue.

Actions

Copy link

#22

Updated by Jan Mach almost 3 years ago

Rajmund Hruska wrote in #note-20:

Jan Mach wrote in #note-13:

I have just merged this branch into devel. It was possible, because I have successfully removed all breaking changes. Now the new Vagrant configurations are just non breaking additions. This means, that now it will be possible to test the new Vagrant development workflow more easily.

I am not sure if something changed since this comment, but I have a trouble launching mentat-controller.py --command start. Apparently, now it requires user account mentat. Here is the error message - CRITICAL ERROR: "Requested unknown user account 'mentat'". When I create user mentat, I have some issues with permissions. Changing permissions of mentat-ng repository is not enough.

I have investigated this furthrer and came to the conclusion, that your issue was due to the wrong usage and perhaps misleading documentation. I had no troubles executing mentat-controller.py --command start when doing it under the mentat system user as it should be. I have improved the documentation to be more clear, but the short summary is this: there are two main system users: vagrant and mentat. the vagrant is deeply baked into the Vagrant system and acts as de facto root for the box. It would not be practical to change it to something else. The mentat user is there to mimic the production enviroment and is intended to run Mentat system services. Project files are mounted and owned by vagrant user and by default you are also connected as vagrant user, so that you may freely work with the files. When you need to execute Mentat, you need to switch to mentat system user, because everything is preconfigured for that.

Actions

Copy link

#23

Updated by Jan Mach almost 3 years ago

Status changed from In Progress to In Review
% Done changed from 90 to 100

Actions

Copy link

#24

Updated by Rajmund Hruška almost 3 years ago

Based on the 2021-01-06 meeting, I am writing a clarification.

I am not using Vagrant for development. I just installed Mentat from Git, as described in https://alchemist.cesnet.cz/mentat/doc/development/html/_doclib/installation.html#section-installation-git, which is referred to in https://alchemist.cesnet.cz/mentat/doc/development/html/_doclib/development.html#getting-the-code. In the documentation for development it was never said that I must have user Mentat.

I am not sure if my issue was caused by this task or #7121 but I think it was the latter one in 1c633b1.

Actions

Copy link

#25

Updated by Jan Mach almost 3 years ago

Note: Ok, so now the problem is that the default configuration was changed to hard-code the usage of mentat:mentat system user and group to solve the issue with directories being occasionally created with incorrect permissions. I am wondering, if the best approach at this point shouldn`t be to stop supporting arbitrary installations for development. After all the whole purpose of having and using a tool like Vagrant is to be able to stop worrying with broken and/or hard installations.

Actions

Copy link

#26

Updated by Rajmund Hruška almost 3 years ago

Jan Mach wrote in #note-12:

Summary of drawbacks of using current opproach with lightweight chroot and maximal application portability:

Configuration files for Babel translations need to be generated to provide the necessary portability. Opposing argument is, that this is not relevant, because babel is misused in this use case anyway and the translations of report contents should be handled differently.

There are many text (JSON) configuration files, which need a separate copy in case of local development within the Git repository and deployment on the server. It is untrivial matter to keep all instances of all configuration files in sync. Currently this is resolved by using .env files, but this solution brings is too magical to be clear.

...

Also, there is a couple of files (mostly configuration) which I modified in order to be able to run some modules:

conf/core/reporting.json.conf
conf/core/statistics.json.conf
- changed paths in both files to chroot
conf/mentat-hawat.py.conf
- enabled 'hawat.blueprints.auth_dev'
conf/mentat-ideagen.py.conf
- added all starting IP addresses of networks from negistry
scripts/sqldb-migrate.sh
- changed path /etc/mentat/migrations-events to /home/rajmund/mentat-ng/migrations-events

Actions

Copy link

#27

Updated by Jan Mach almost 3 years ago

To be discussed changed from Yes to No

Actions

Copy link

#28

Updated by Jan Mach almost 3 years ago

Status changed from In Review to In Progress

Actions

Copy link

#29

Updated by Rajmund Hruška almost 3 years ago

Related to Bug #7548: mentat-enricher.py error in Vagrant box added

Actions

Copy link

#30

Updated by Rajmund Hruška almost 3 years ago

Related to Bug #7552: Tests fail in Vagrant box added

Actions

Copy link

#31

Updated by Rajmund Hruška almost 3 years ago

To be discussed changed from No to Yes

When I was trying to replicate the issue described in #7548 I came upon another issue.

I deleted the virtual machine and the cloned repository. Then I cloned the repository again and switched to devel branch. When I executed make data-fetch-config the following error was printed on my terminal screen:

ssh: Could not resolve hostname mentat-alt: Temporary failure in name resolution
make: *** [Makefile:486: data-fetch-config] Error 1

I checked the Makefile and there is indeed only mentat-alt and not the full address. Am I doing something wrong?

Actions

Copy link

#32

Updated by Jan Mach almost 3 years ago

Status changed from In Progress to In Review
To be discussed changed from Yes to No

Rajmund Hruska wrote in #note-31:

When I was trying to replicate the issue described in #7548 I came upon another issue.

I deleted the virtual machine and the cloned repository. Then I cloned the repository again and switched to devel branch. When I executed make data-fetch-config the following error was printed on my terminal screen:
[...]

I checked the Makefile and there is indeed only mentat-alt and not the full address. Am I doing something wrong?

No, you are not doing anything wrong. Locally I use the SSH`s config file and I automatically put the shortcut to SCP path. I have not discovered this error before, bacause it just worked for me correctly. I have put full URLs to Makefile, so it should be resolved now.

Actions

Copy link

#33

Updated by Pavel Kácha almost 3 years ago

Status changed from In Review to Closed

Project

General

Profile

Mentat

Custom queries

Task #7041

Use Vagrant as alternative tool for local development

Updated by Jan Mach almost 4 years ago

Updated by Jan Mach almost 4 years ago

Updated by Jan Mach over 3 years ago

Updated by Jan Mach over 3 years ago

Updated by Jan Mach over 3 years ago

Updated by Jan Mach about 3 years ago

Updated by Jan Mach about 3 years ago

Updated by Jan Mach about 3 years ago

Updated by Pavel Kácha about 3 years ago

Updated by Radko Krkoš about 3 years ago

Updated by Radko Krkoš about 3 years ago

Updated by Jan Mach about 3 years ago

Updated by Jan Mach about 3 years ago

Updated by Jan Mach about 3 years ago

Updated by Jan Mach about 3 years ago

Updated by Jan Mach about 3 years ago

Updated by Radko Krkoš about 3 years ago

Updated by Jan Mach about 3 years ago

Updated by Jan Mach about 3 years ago

Updated by Rajmund Hruška about 3 years ago

Updated by Rajmund Hruška about 3 years ago

Updated by Jan Mach almost 3 years ago

Updated by Jan Mach almost 3 years ago

Updated by Rajmund Hruška almost 3 years ago

Updated by Jan Mach almost 3 years ago

Updated by Rajmund Hruška almost 3 years ago

Updated by Jan Mach almost 3 years ago

Updated by Jan Mach almost 3 years ago

Updated by Rajmund Hruška almost 3 years ago

Updated by Rajmund Hruška almost 3 years ago

Updated by Rajmund Hruška almost 3 years ago

Updated by Jan Mach almost 3 years ago

Updated by Pavel Kácha almost 3 years ago