Project

General

Profile

Actions

Task #7539

closed

Migrate from Buildbot to GitLab

Added by Pavel Kácha over 2 years ago. Updated 11 days ago.

Status:
Closed
Priority:
Normal
Category:
Development - Tools
Target version:
Start date:
11/18/2021
Due date:
% Done:

80%

Estimated time:
To be discussed:

Description

This is task for discussions/planning of migration, other more specific tasks may stem from this.

Final result should be that all Mentat related repos from Buildbot (and some external) should be migrated to GitLab, with adapted CI stage, pip upload and deb generation.


Related issues

Related to Mentat - Bug #7718: Gitlab CI/CD doesn't push the built package to PyPIClosedRajmund Hruška03/19/2024

Actions
Related to Mentat - Config #7719: In GitLab CI/CD GeoLite2 databases are downloaded for almost every job ClosedRajmund Hruška03/19/2024

Actions
Actions #1

Updated by Pavel Kácha over 2 years ago

  • Description updated (diff)
Actions #2

Updated by Pavel Kácha over 2 years ago

From discussion with Martin Dušička:

  • There are 4 runners for general use directly on GitLab server
  • GitLab Pages not working yet, Martin has to decide where (and how) to make them accessible
    • Pages would allow to publish docs and probably repos
  • Wiki will be migrated as is, we will cleanup later (so we will have history on GitLab)
Actions #3

Updated by Pavel Kácha over 2 years ago

Question is how to manage PGP keys for deb package signing. Possibilities (already sorted by viability):

  1. Extra repo with keyring and limited access (can be common for more projects, we don't mind binary contents as it gets changed rarely and never edited by hand)
  2. Private and public key in GitLab variables (signing script must always reinitialize all new keyring, must be in variables in every project concerned)
  3. Docker image with ready keyring (problem of encrypted contents on public place or need to keep own image resource, cumbersome and nonversioned updates of the keyring)

First one seems plausible.

Actions #4

Updated by Rajmund Hruška over 2 years ago

  • Status changed from New to Feedback
  • To be discussed changed from No to Yes

I think I managed to make signing work on typedcols repository using the first option. I will just polish some things and then I will create the configuration file for other repositories.

There is one thing which I don't really understand. I tried verifying gpg signature of files at typecols. Version 0.1.13 is working just fine but verifying previous 2 versions results in BAD signature and I don't know why.

Actions #5

Updated by Rajmund Hruška over 2 years ago

  • Status changed from Feedback to Waiting
  • % Done changed from 0 to 20
  • To be discussed changed from Yes to No

Rajmund Hruska wrote in #note-4:

There is one thing which I don't really understand. I tried verifying gpg signature of files at typecols. Version 0.1.13 is working just fine but verifying previous 2 versions results in BAD signature and I don't know why.

So, seems like for some older versions the files were replaced but the signatures weren't and that's why the verification of signature failed and the sha256sums were different.

Next thing to figure out is publishing documentation and signed files. So we are waiting until GitLab Pages is set and running.

Actions #6

Updated by Rajmund Hruška about 2 years ago

  • Status changed from Waiting to In Progress

The GitLab Pages are finally set. The next thing to figure out is how to publish signed packages.

Actions #7

Updated by Rajmund Hruška about 2 years ago

  • Target version changed from Backlog to 2.10
  • % Done changed from 20 to 50
  • To be discussed changed from No to Yes

So, I did manage to make both documentation and signed files publicly available. I am still working on how to share the cached files between branches which I thought I had already done, but it's not quite working yet.

Here are the example Pages for Pynspect project, master branch.
https://709.gitlab-pages-test.cesnet.cz/hruska/pynspect/master/html/manual.html
https://709.gitlab-pages-test.cesnet.cz/hruska/pynspect/master/files/

Actions #8

Updated by Rajmund Hruška about 2 years ago

  • To be discussed deleted (Yes)
Actions #9

Updated by Rajmund Hruška about 2 years ago

  • % Done changed from 50 to 70
  • To be discussed set to Yes

Pynspect, Pyzenkit, typedcols, ipranges and idea are already at gitlab.cesnet.cz and CI/CD seems to be working. I only changed the version of Pynspect, all other projects have the same version number as before.

I didn't finish Pydgets yet, as I suspect the code in GitHub repository is not up to date. I wrote an email to Jan Mach regarding this repository.

https://709.gitlab-pages.cesnet.cz/idea/ipranges/master/html/manual.html
https://709.gitlab-pages.cesnet.cz/idea/idea/master/html/manual.html
https://709.gitlab-pages.cesnet.cz/idea/typedcols/master/html/manual.html
https://709.gitlab-pages.cesnet.cz/mentat/pynspect/master/html/manual.html
https://709.gitlab-pages.cesnet.cz/mentat/pyzenkit/master/html/manual.html

https://709.gitlab-pages.cesnet.cz/idea/ipranges/master/files/
https://709.gitlab-pages.cesnet.cz/idea/idea/master/files/
https://709.gitlab-pages.cesnet.cz/idea/typedcols/master/files/
https://709.gitlab-pages.cesnet.cz/mentat/pynspect/master/files/
https://709.gitlab-pages.cesnet.cz/mentat/pyzenkit/master/files/

By changing master to devel in the URL, it is possible to access the development documentation and signed files.

So apart from Pydgets, the next step is Mentat repository.

Actions #10

Updated by Rajmund Hruška almost 2 years ago

  • To be discussed changed from Yes to No

Pydgets is now also ready.

All of those smaller projects used simple python docker image. For Mentat it would be nice to have a better Docker image. Maybe this one could do the work, although it doesn't contain PostgreSQL. https://hub.docker.com/r/nikolaik/python-nodejs

Actions #11

Updated by Pavel Kácha almost 2 years ago

Wouldn't it make sense to make our own and push it to the hub? Seems creating Docker images based on pre-existing ones needs might need just short Dockerfile.

https://linuxhint.com/automatically-build-docker-images-in-debian-10-buster/

Actions #12

Updated by Pavel Kácha almost 2 years ago

  • Target version changed from 2.10 to 2.11
Actions #13

Updated by Rajmund Hruška over 1 year ago

  • Status changed from In Progress to Feedback
  • To be discussed changed from No to Yes

I lost my notes from the meeting where we discussed this issue. So, I should create a job which downloads all the files at a given URL address? If I remember correctly, we want to get the older files from https://alchemist.cesnet.cz/pyzenkit/files/development/ and also have some kind of backup if we lose data from GitLab.

Actions #14

Updated by Pavel Kácha over 1 year ago

Rajmund Hruska wrote in #note-13:

I lost my notes from the meeting where we discussed this issue. So, I should create a job which downloads all the files at a given URL address? If I remember correctly, we want to get the older files from https://alchemist.cesnet.cz/pyzenkit/files/development/ and also have some kind of backup if we lose data from GitLab.

We wanted to solve two main problems:

  • Loss of continuity – something in Gitlab goes awry or human makes an error, and we thus lose the continually reloaded artifact containing older releases
  • Migration of older data from Redmine

If memory serves me well (sorry, too much parity errors these days), we came to rather simple sidestep – define GitLab variable, which, if set to a bunch of URLs, would cause build script to download contents of all the URLs and populate the "history" artifact (or maybe one URL with expected zipfile? Just what's simpler). That would allow to inject initial history (from Redmine), which we are not able to reconstruct, and also to possibly inject history from backups if data end up in a blue smoke.

Also, we should actively backup those released files, for example by regular download to backed tree at Mentat-alt or somewhere.

Actions #15

Updated by Rajmund Hruška over 1 year ago

Pavel Kácha wrote in #note-14:

Also, we should actively backup those released files, for example by regular download to backed tree at Mentat-alt or somewhere.

So, the backup is set up on mentat-alt. If I understand it correctly I can keep the built packages in /var/backups/ .

Actions #16

Updated by Pavel Kácha over 1 year ago

Rajmund Hruska wrote in #note-15:

So, the backup is set up on mentat-alt. If I understand it correctly I can keep the built packages in /var/backups/ .

Yup, or some convenient subdirectory thereof.

Actions #17

Updated by Rajmund Hruška about 1 year ago

  • To be discussed deleted (Yes)
Actions #18

Updated by Rajmund Hruška 11 months ago

  • Status changed from Feedback to In Review
Actions #19

Updated by Rajmund Hruška 11 months ago

  • % Done changed from 70 to 80

The repository is created here: https://gitlab.cesnet.cz/713/mentat/mentat.

The next step is to set up mentat-alt to install mentat from deb packages created in GitLab.

Actions #20

Updated by Pavel Kácha 10 months ago

From 2023-06-22 meeting:

  • GitLab expects specific suffixes as a part of version.
  • GitLab extracts version of the package from the changelog.

Our old packages (generated on Buildbot) don't satisfy neither GitLab requirement - although we have the mechanism to upload old packages along with new generated ones, packages are in wrong branch with wrong (all the same) fabricated version.

To add insult to injury, our deb packages are de facto envelope around venv and pip install, and keep static URL to the Buildbot.

Conclusion - attempts to incorporate old packages onto GitLab along with new ones would take too much effort with inadequate results. Let's not try unify old and new packages and let's keep URL to Buildbot on the web as obsolete and keep running Buildbot (maybe without Alchemist) for some time (a year maybe?). And then ditch it altogether.

Actions #21

Updated by Rajmund Hruška 10 months ago

  • Target version changed from 2.11 to Future
Actions #22

Updated by Rajmund Hruška 8 months ago

mentat-alt is now installed and upgraded from GitLab packages. There is no need to push to buildbot/alchemist anymore.

Actions #23

Updated by Pavel Kácha 8 months ago

🎆 👏 🥂

Actions #24

Updated by Rajmund Hruška about 1 month ago

  • Related to Bug #7718: Gitlab CI/CD doesn't push the built package to PyPI added
Actions #25

Updated by Rajmund Hruška about 1 month ago

  • Related to Config #7719: In GitLab CI/CD GeoLite2 databases are downloaded for almost every job added
Actions #26

Updated by Pavel Kácha 11 days ago

  • Status changed from In Review to Closed

CI/CD and package/repo creation complete. Big thanks again.

Actions

Also available in: Atom PDF