Project

General

Profile

Actions

Task #7539

open

Migrate from Buildbot to GitLab

Added by Pavel Kácha about 1 year ago. Updated about 17 hours ago.

Status:
Feedback
Priority:
Normal
Category:
Development - Tools
Target version:
Start date:
11/18/2021
Due date:
% Done:

70%

Estimated time:
To be discussed:
Yes

Description

This is task for discussions/planning of migration, other more specific tasks may stem from this.

Final result should be that all Mentat related repos from Buildbot (and some external) should be migrated to GitLab, with adapted CI stage, pip upload and deb generation.

Actions #1

Updated by Pavel Kácha about 1 year ago

  • Description updated (diff)
Actions #2

Updated by Pavel Kácha about 1 year ago

From discussion with Martin Dušička:

  • There are 4 runners for general use directly on GitLab server
  • GitLab Pages not working yet, Martin has to decide where (and how) to make them accessible
    • Pages would allow to publish docs and probably repos
  • Wiki will be migrated as is, we will cleanup later (so we will have history on GitLab)
Actions #3

Updated by Pavel Kácha about 1 year ago

Question is how to manage PGP keys for deb package signing. Possibilities (already sorted by viability):

  1. Extra repo with keyring and limited access (can be common for more projects, we don't mind binary contents as it gets changed rarely and never edited by hand)
  2. Private and public key in GitLab variables (signing script must always reinitialize all new keyring, must be in variables in every project concerned)
  3. Docker image with ready keyring (problem of encrypted contents on public place or need to keep own image resource, cumbersome and nonversioned updates of the keyring)

First one seems plausible.

Actions #4

Updated by Rajmund Hruska 12 months ago

  • Status changed from New to Feedback
  • To be discussed changed from No to Yes

I think I managed to make signing work on typedcols repository using the first option. I will just polish some things and then I will create the configuration file for other repositories.

There is one thing which I don't really understand. I tried verifying gpg signature of files at typecols. Version 0.1.13 is working just fine but verifying previous 2 versions results in BAD signature and I don't know why.

Actions #5

Updated by Rajmund Hruska 12 months ago

  • Status changed from Feedback to Waiting
  • % Done changed from 0 to 20
  • To be discussed changed from Yes to No

Rajmund Hruska wrote in #note-4:

There is one thing which I don't really understand. I tried verifying gpg signature of files at typecols. Version 0.1.13 is working just fine but verifying previous 2 versions results in BAD signature and I don't know why.

So, seems like for some older versions the files were replaced but the signatures weren't and that's why the verification of signature failed and the sha256sums were different.

Next thing to figure out is publishing documentation and signed files. So we are waiting until GitLab Pages is set and running.

Actions #6

Updated by Rajmund Hruska 10 months ago

  • Status changed from Waiting to In Progress

The GitLab Pages are finally set. The next thing to figure out is how to publish signed packages.

Actions #7

Updated by Rajmund Hruska 10 months ago

  • Target version changed from Backlog to 2.10
  • % Done changed from 20 to 50
  • To be discussed changed from No to Yes

So, I did manage to make both documentation and signed files publicly available. I am still working on how to share the cached files between branches which I thought I had already done, but it's not quite working yet.

Here are the example Pages for Pynspect project, master branch.
https://709.gitlab-pages-test.cesnet.cz/hruska/pynspect/master/html/manual.html
https://709.gitlab-pages-test.cesnet.cz/hruska/pynspect/master/files/

Actions #8

Updated by Rajmund Hruska 8 months ago

  • To be discussed deleted (Yes)
Actions #9

Updated by Rajmund Hruska 8 months ago

  • % Done changed from 50 to 70
  • To be discussed set to Yes

Pynspect, Pyzenkit, typedcols, ipranges and idea are already at gitlab.cesnet.cz and CI/CD seems to be working. I only changed the version of Pynspect, all other projects have the same version number as before.

I didn't finish Pydgets yet, as I suspect the code in GitHub repository is not up to date. I wrote an email to Jan Mach regarding this repository.

https://709.gitlab-pages.cesnet.cz/idea/ipranges/master/html/manual.html
https://709.gitlab-pages.cesnet.cz/idea/idea/master/html/manual.html
https://709.gitlab-pages.cesnet.cz/idea/typedcols/master/html/manual.html
https://709.gitlab-pages.cesnet.cz/mentat/pynspect/master/html/manual.html
https://709.gitlab-pages.cesnet.cz/mentat/pyzenkit/master/html/manual.html

https://709.gitlab-pages.cesnet.cz/idea/ipranges/master/files/
https://709.gitlab-pages.cesnet.cz/idea/idea/master/files/
https://709.gitlab-pages.cesnet.cz/idea/typedcols/master/files/
https://709.gitlab-pages.cesnet.cz/mentat/pynspect/master/files/
https://709.gitlab-pages.cesnet.cz/mentat/pyzenkit/master/files/

By changing master to devel in the URL, it is possible to access the development documentation and signed files.

So apart from Pydgets, the next step is Mentat repository.

Actions #10

Updated by Rajmund Hruska 7 months ago

  • To be discussed changed from Yes to No

Pydgets is now also ready.

All of those smaller projects used simple python docker image. For Mentat it would be nice to have a better Docker image. Maybe this one could do the work, although it doesn't contain PostgreSQL. https://hub.docker.com/r/nikolaik/python-nodejs

Actions #11

Updated by Pavel Kácha 7 months ago

Wouldn't it make sense to make our own and push it to the hub? Seems creating Docker images based on pre-existing ones needs might need just short Dockerfile.

https://linuxhint.com/automatically-build-docker-images-in-debian-10-buster/

Actions #12

Updated by Pavel Kácha 4 months ago

  • Target version changed from 2.10 to 2.11
Actions #13

Updated by Rajmund Hruska 2 months ago

  • Status changed from In Progress to Feedback
  • To be discussed changed from No to Yes

I lost my notes from the meeting where we discussed this issue. So, I should create a job which downloads all the files at a given URL address? If I remember correctly, we want to get the older files from https://alchemist.cesnet.cz/pyzenkit/files/development/ and also have some kind of backup if we lose data from GitLab.

Actions #14

Updated by Pavel Kácha 2 months ago

Rajmund Hruska wrote in #note-13:

I lost my notes from the meeting where we discussed this issue. So, I should create a job which downloads all the files at a given URL address? If I remember correctly, we want to get the older files from https://alchemist.cesnet.cz/pyzenkit/files/development/ and also have some kind of backup if we lose data from GitLab.

We wanted to solve two main problems:

  • Loss of continuity – something in Gitlab goes awry or human makes an error, and we thus lose the continually reloaded artifact containing older releases
  • Migration of older data from Redmine

If memory serves me well (sorry, too much parity errors these days), we came to rather simple sidestep – define GitLab variable, which, if set to a bunch of URLs, would cause build script to download contents of all the URLs and populate the "history" artifact (or maybe one URL with expected zipfile? Just what's simpler). That would allow to inject initial history (from Redmine), which we are not able to reconstruct, and also to possibly inject history from backups if data end up in a blue smoke.

Also, we should actively backup those released files, for example by regular download to backed tree at Mentat-alt or somewhere.

Actions #15

Updated by Rajmund Hruska 2 days ago

Pavel Kácha wrote in #note-14:

Also, we should actively backup those released files, for example by regular download to backed tree at Mentat-alt or somewhere.

So, the backup is set up on mentat-alt. If I understand it correctly I can keep the built packages in /var/backups/ .

Actions #16

Updated by Pavel Kácha about 17 hours ago

Rajmund Hruska wrote in #note-15:

So, the backup is set up on mentat-alt. If I understand it correctly I can keep the built packages in /var/backups/ .

Yup, or some convenient subdirectory thereof.

Actions

Also available in: Atom PDF