Project

General

Profile

Task #4216

Make Mentat executable from within the cloned Git repository

Added by Jan Mach over 1 year ago. Updated 9 months ago.

Status:
Closed
Priority:
High
Assignee:
Category:
Documentation
Target version:
Start date:
07/27/2018
Due date:
% Done:

100%

Estimated time:
To be discussed:
No

Description

It would be very useful for future contributors if it was possible to develop and execute Mentat modules from within the cloned Git repository. Currently the development process is little bit awkward. Part of this task is also writing the manual for developer and contributors.

Associated revisions

Revision 1003d42b (diff)
Added by Jan Mach about 1 year ago

Removed unnecessary shell script.

(Redmine issue: #4216)

Revision d1672c44 (diff)
Added by Jan Mach about 1 year ago

Make the whole project more developer friendly.

Following big improvements were made to make the development more easier:

  • Created the 'setup.py’ script to support native Python packaging with setuptools and pip.
  • There is a new target 'make develop’, that will bootstrap the Python virtual environment, install all production and development requirements and install the whole project in editable mode.
  • Revised the dependency installation targets.
  • Big facelift of makefile help target.
  • Added target for build environment cleanup.
  • Added target for initialization of new translations of Mentat and Hawat.
  • Added target for building locally Python packages.
  • Added target for building locally Debian packages.
  • Updated documentation page for developers.

(Redmine issue: #4216,#3387,#3361)

Revision 8ee65f54 (diff)
Added by Jan Mach about 1 year ago

Fixes and improvements in master makefile.

  • Merged pylint and pyflakes tasks for simplicity,
  • Fixed broken clean task.
  • Updated development documentation to include Grunt as prerequisite.
  • Few additional minor tweaks.

(Redmine issue: #4216,#3387,#3361)

Revision 307c703f (diff)
Added by Jan Mach about 1 year ago

Changed the setup.py to load development requirements from file.

(Redmine issue: #4216,#3387)

Revision 22d814dd (diff)
Added by Jan Mach about 1 year ago

Removed manually created API documentation files and used sphinx-apidoc instead.

Previously all API documentation pages were manually created, which resulted is errors. (Redmine issue: #4216,#3361,#3387)

Revision 17df7221 (diff)
Added by Jan Mach about 1 year ago

Removed unnecessary tasks and functions from Gruntfile.

There were deprecated deployment related tasks in Gruntfile related to previous package building and deployment mechanism. There are no longer valid and used, so I have removed them to make the file more clear. All package build and deployment related tasks should be moved to master makefile and Grunt should be used only to the stuff related to web interface. So the next task will be to rewrite the Debian package generation into the master makefile. (Redmine issue: #4216,#3387)

Revision 8ea53be9 (diff)
Added by Jan Mach about 1 year ago

Huge revision of master makefile and development documentation page.

Documentation, build procedures...almost everything was revised and rethought. (Redmine issue: #4216,#3387,#3361)

Revision 43bc8eaf (diff)
Added by Jan Mach about 1 year ago

Moved part of the Makefile functionality to included library.

Reason is better reuse in other projects. (Redmine issue: #4216,#3387)

Revision 72b7e4dc (diff)
Added by Jan Mach about 1 year ago

Rewritten and enhanced the Hawat logging setup.

The logging setup is now more capable and configurable. (Redmine issue: #4216,#3443)

Revision de483ec4 (diff)
Added by Jan Mach about 1 year ago

Implemented Hawat command line interface based on flask.cli and Click.

Currently only built-in Flask commands run, shell, etc. are available. It can be used to launch development web server, which is now documented in development documentation page. (Redmine issue: #4216)

Revision cc0e4e71 (diff)
Added by Jan Mach about 1 year ago

Huge amount of work towards making Mentat executable from within the cloned Git repository.

This commit brings in considerable amount of work. When working on this issue everything was so tied together, that I have abandoned all attempts to split it into multiple commits. There are following changes and more:

  • Implementation of lightweight chroot for containing the application in single directory tree.
  • Design changes in scripts and daemons to implement the lightweight chroot.
  • Many configuration file changes to make everything work properly.
  • Makefile additions, new targets related to running
  • Updated development documentation page.

(Redmine issue: #4216)

Revision 1f61c8dc (diff)
Added by Jan Mach about 1 year ago

Moved metadata database migrations from configuration to Python module namespace.

(Redmine issue: #3387,#4216)

Revision 097d3a7b (diff)
Added by Jan Mach about 1 year ago

Implemented the pybabel translation commands as built-in Flask CLI commands.

Pybabel translations are now implemented as built-in Flask CLI commands and accessible from 'hawat-cli intl’ utility. This approach enables users to easily customize translations for their custom Mentat system installations, or adding new languages. The drawback of this approach is, that the commands are executed within the Flask`s application context, so the application must be properly installed on target system for everything to work properly. I have kept original translation commands in the makefile for future reference.

(Redmine issue: #4789,#4216)

Revision 7c94f02c (diff)
Added by Jan Mach about 1 year ago

Changed realpath() to relpath() when calculating paths for pybabel.

Problem with the previous solution was, that full path to the extracted strings was visible in the message .po files. So the exact path to the folder from which the developer is working was visible, which is bad for so many reasons. (Redmine issue: #4789,#4216)

Revision 77710096 (diff)
Added by Jan Mach about 1 year ago

Minor documentation improvements regarding translations.

(Redmine issue: #4789,#4216)

Revision 6e9bf33c (diff)
Added by Jan Mach about 1 year ago

Master makefile targets for message catalog translations now use pybabel directly again.

Previous approach, while better in some aspects caused troubles on our automated build system, because it required for the Mentat to be correctly installed for hawat-cli to be available. (Redmine issue: #3387,#4216)

Revision 07c3f135 (diff)
Added by Jan Mach about 1 year ago

Makefile and buildprocedure imporvements.

  • The creation of Python virtual environment is now a separate makefile target.
  • The 'develop’ target now reuses already defined targets.
  • Using python-dotenv to local configuration customizations.

(Redmine issue: #3387,#4216)

Revision e1c72d3a (diff)
Added by Jan Mach about 1 year ago

Fix: Fixed makefile to display names of executed commands for tests.

(Redmine issue: #3387,#4216)

Revision 8cfc1309 (diff)
Added by Jan Mach about 1 year ago

Rewritten the Debian package building mechanism.

New mechanism for building Debian packages makes better use of debhelper. Previous solution with using Grunt for building the package was abandoned entirely. (Redmine issue: #3387,#4216)

Revision 3fb0e3b4 (diff)
Added by Jan Mach 12 months ago

Fix: Fixed broken application path calculations for Hawat.

Hawat contains several application path calculations relative to application root path. The mechanism was broken, because these configurations were evaluated on compilation time and the APP_ROOT_PATH had then no effect. (Redmine issue: #4216)

Revision 287910e9 (diff)
Added by Jan Mach 12 months ago

Updated the installation manual with Git installation.

The installation manual was enhanced with the option of installing the project directly from Git repository. This option overlaps with the installation process for local development, so the development documentation page was also updated accordingly. (Redmine issue: #4216)

Revision 04185551 (diff)
Added by Jan Mach 12 months ago

Fix: Added local cron.d directory to lightweight chroot.

(Redmine issue: #4216)

Revision 97df5a36 (diff)
Added by Jan Mach 11 months ago

Improved the installation process from Git repository.

The makefile target 'develop’ now pre-builds web interface artifacts. There is a new example configuration file for Apache2 that demonstrates the necessary configuration for Git installations. There is a custom WSGI script file for easier web interface installation from Git. (Redmine issue: #4216)

Revision e61f36f5 (diff)
Added by Jan Mach 11 months ago

Fix: Added the missing event migrations to Debian package.

(Redmine issue: #3361,#3387,#4216)

Revision 45f315ae (diff)
Added by Jan Mach 11 months ago

Improvements in documentation of installation instructions.

(Redmine issue: #4216)

History

#1 Updated by Jan Mach over 1 year ago

  • Target version changed from Future to 2.2

#2 Updated by Jan Mach over 1 year ago

  • Target version changed from 2.2 to 2.3

#3 Updated by Jan Mach over 1 year ago

  • Target version changed from 2.3 to 2.4

#4 Updated by Jan Mach about 1 year ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 10

#5 Updated by Jan Mach 11 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 10 to 100

#6 Updated by Jan Mach 11 months ago

  • Status changed from Closed to Feedback
  • Assignee changed from Jan Mach to Radko Krkoš
  • % Done changed from 100 to 90

Radko, could you please walk through these installation instructions and attempt to install working Mentat instance from Debian package and local Git repository to verify the installation instructions. I am striving to make the installation process as simple as possible and if you are not able to have working instance in half an hour this task should be improved further.

I would like another set of eyes to look through it, because after working on it I am biased.

Please install development version, necessary code changes are not yet merged to the master/production branch.

#7 Updated by Jan Mach 11 months ago

  • Priority changed from Normal to High

#8 Updated by Radko Krkoš 11 months ago

I have tested the instructions for manual installation.

Here are some other remarks:

  • What I am missing is a mention that the steps have to be run as root. Using sudo (in various places where required) will not result in a working installation as the environment is very different from what seems to be expected. (I have not tested this thoroughly, I might have been trying to install the stable version [2.3.2] at the time).
  • I advise to include the installation commands for postgresql and ip4r and the mention that the server must listen on port 5432. For a clean installation, it would be highly beneficial. We should not expect the Mentat admin to be a PostgreSQL admin also (or to read the documentation to a third-party extension).
  • The development documentation should probably reference the development repositories for mentat, not the stable ones (steps 2 and 3).
  • In step 9, the /etc/mentat/scripts/sqldb-migrate.sh stamp head command has to be run inside the venv, otherwise it fails with alembic: command not found

Unfortunately it took me quite a bit longer than the agreed limit of half and hour to get the mentat working.
I have not yet tried the git repository instructions.

#9 Updated by Jan Mach 11 months ago

Radko Krkoš wrote:

I have tested the instructions for manual installation.

Here are some other remarks:

  • What I am missing is a mention that the steps have to be run as root. Using sudo (in various places where required) will not result in a working installation as the environment is very different from what seems to be expected. (I have not tested this thoroughly, I might have been trying to install the stable version [2.3.2] at the time).

Done.

  • I advise to include the installation commands for postgresql and ip4r and the mention that the server must listen on port 5432. For a clean installation, it would be highly beneficial. We should not expect the Mentat admin to be a PostgreSQL admin also (or to read the documentation to a third-party extension).

Done, added section about installation of prerequisites.

  • The development documentation should probably reference the development repositories for mentat, not the stable ones (steps 2 and 3).

Sadly, this is not possible. To my knowledge and according to my research RST does not allow variable substitutions in code blocks.

  • In step 9, the /etc/mentat/scripts/sqldb-migrate.sh stamp head command has to be run inside the venv, otherwise it fails with alembic: command not found

Done.

Unfortunately it took me quite a bit longer than the agreed limit of half and hour to get the mentat working.
I have not yet tried the git repository instructions.

#10 Updated by Jan Mach 11 months ago

  • Assignee changed from Radko Krkoš to Pavel Kácha
  • % Done changed from 90 to 100

Pavel, during our conference you have volunteered to perform another test installation. Do you still have time to do it? If not, we can close the issue a polish the instructions later. I think, that all changes at this point should be purely cosmetic, as I have already tested the installation many times.

#11 Updated by Jan Mach 9 months ago

  • Status changed from Feedback to Closed

Ok, let`s close this issue, so we can close the whole version.

Also available in: Atom PDF