Overall migration status (periodically updated)¶

• (DONE) Install new base server
  • New server mentat-alt.cesnet.cz is ready and installed using Ansible
• (DONE) Install Mentat system and Warden client on the server
  • Development version of Mentat system is installed on the server using Debian package system
  • Warden client is installed on the server and connected to production instance of Warden server
• (IN PROGRESS) Perform data and service migration to new server
  • Prepare database migration scripts
  • Prepare filesystem migration scripts
  • Prepare utility migration scripts
  • Migrate the Mentat service
• Verify functionality

General guidelines for migration process¶

1. Day before migration lower the TTL of relevant DNS records for mentat-hub.cesnet.cz and mentat-alt.cesnet.cz servers
2. Presynchronize filesystem data (rsync), so that the actual migration will be much quicker later.
   1. report attachments
   2. RRD databases and chart images
   3. cache files
   4. persistent state files
   5. runlog files? (maybe not necessary)
   6. log files? (maybe not necessary)
3. Shut down Warden client on mentat-hub.cesnet.cz and mentat-alt.cesnet.cz servers and let Mentat empty all queues.
4. Shut down Mentat systems on mentat-hub.cesnet.cz and mentat-alt.cesnet.cz servers
5. Perform database migration
   1. users
   2. groups
   3. filters
   4. networks
   5. reports
   6. event statistics
6. Perform filesystem migration - same data as above
7. Perform configuration migration
   1. synchronize content of /etc/mentat configuration directory
8. Switch warden client certificates between mentat-hub.cesnet.cz and mentat-alt.cesnet.cz servers
9. Switch shibd configuration between mentat-hub.cesnet.cz and mentat-alt.cesnet.cz servers
10. Switch hostnames and IP addresses between mentat-hub.cesnet.cz and mentat-alt.cesnet.cz servers
11. Reboot both servers and pray to your favorite god, or as an atheist sit quietly with your hands in your lap
12. Login to new mentat-hub.cesnet.cz and launch everything
    1. Launch Mentat backend services (daemons and scripts)
    2. Launch Warden client and verify messages are being stored into database
    3. Verify that the web interface is accessible
13. Synchronize crontab for root

Event migration might not be necessary. If the Mentat will be running on new server for some time and without any downtimes, we could skip slow migration of events from MongoDB to PostgreSQL.

Migration process checklist¶

Trial period before migration¶

During this period the mentat-alt.cesnet.cz server works as independent and fully operational instance of Mentat system, which can be used for testing and development purposes.

1. (DONE) mentat-alt.cesnet.cz: Install base server.
2. mentat-alt.cesnet.cz: Configure server monitoring with Nagios.
3. mentat-alt.cesnet.cz: Configure server backup.
4. (DONE) mentat-alt.cesnet.cz: Install development version of Mentat system. Keep it running and updated during trial period.
5. (DONE) mentat-hub.cesnet.cz: Write script for periodical dump of MongoDB.
   1. Script is called /root/mentatdb-dump-all.sh.
   2. Verified, that script is working properly.
6. (DONE) mentat-alt.cesnet.cz: Write script for periodical import of MongoDB dumps from mentat-hub.cesnet.cz.
   1. Script is called /root/mentat-sync-mongodb.sh.
   2. Verified, that script is working properly.
7. (DONE) mentat-hub.cesnet.cz: Install cronjob for script /root/mentat-sync-mongodb.sh to periodically test the import process.
   1. Installed with following root crontab record: 5 */4 * * * /root/mentat-sync-mongodb.sh
   2. The script will perform fresh dump using /root/mentatdb-dump-all.sh on mentat-hub.cesnet.cz, fetch the result and import it to local MongoDB instance.
   3. Verified, that cronjob is working properly.

From day before migration until migration time.¶

After this period the mentat-alt.cesnet.cz is getting ready for migration process. All Mentat modules will be stopped and data will be synchronized to the local filesystem. Only web interface will be operational to some extend and can be used to verify, that migrated data will be accessible.

1. (DONE) Stop all Warden client daemons.
2. (DONE) Stop all Mentat modules.
3. (DONE) mentat-alt.cesnet.cz: Write script for periodical Mentat filesystem data synchronization.
   1. Script is called /root/mentat-sync-files.sh.
   2. Verified, that script is working properly.
4. (DONE) mentat-hub.cesnet.cz: Install cronjob for script /root/mentat-sync-files.sh to periodically prefetch filesystem data to target server.
   1. Installed with following root crontab record: 35 * * * * /root/mentat-sync-files.sh --skip-install
   2. Verified, that cronjob is working properly.
5. (DONE) mentat-alt.cesnet.cz: Prepare new networking configuration into file /etc/network/interfaces.new, backup current setting into file /etc/networking/interfaces.old.
   1. New networking configuration configuration can be enabled by following command cp /etc/networking/interfaces.new /etc/networking/interfaces and restarting the networking service.
6. (DONE) mentat-alt.cesnet.cz: Write script for quick renaming of the server to different name.
   1. Script will replace all ocurences of mentat-alt with mentat-hub in list of selected configuration files.
   2. Script is called /root/system-rename.sh.
   3. Verified, that script is working properly.
7. (DONE) mentat-alt.cesnet.cz: Write script for quick switching of most important configurations.
   1. Send various configuration files to source server and fetch corresponding ones from it.
   2. Configurations like server certificates, shibboleth configurations, Warden client configurations, etc.
   3. Script is called /root/mentat-sync-config.sh.
   4. Verified, that script is working properly.
8. (DONE) Perform initial database migration to be more efficient and less time consuming later.
   1. Fetched current database with /root/mentat-sync-mongodb.sh.
   2. Launch migration with /etc/mentat/scripts/sqldb-migrate-data.py --drop in tmux terminal.

Actual migration process¶

# Disable utility migration scripts that were installed before so that they do not mess with migration process.

root@mentat-alt$ /root/mentat-sync-config.sh
root@mentat-alt$ /root/mentat-sync-files.sh
root@mentat-alt$ /root/mentat-sync-mongodb.sh
root@mentat-alt$ /etc/mentat/scripts/sqldb-migrate-data.py --clear --from-timestamp 1532304000

2018-07-24 13:59:48,096 sqldb-migrate-data.py INFO: Data migration results:
2018-07-24 13:59:48,096 sqldb-migrate-data.py INFO: --------------------------------------------------
2018-07-24 13:59:48,102 sqldb-migrate-data.py INFO: User count:                       133
2018-07-24 13:59:48,105 sqldb-migrate-data.py INFO: Group count:                      295
2018-07-24 13:59:48,107 sqldb-migrate-data.py INFO: Network count:                  1,833
2018-07-24 13:59:48,110 sqldb-migrate-data.py INFO: Filter count:                      55
2018-07-24 13:59:48,114 sqldb-migrate-data.py INFO: Setting count:                    295
2018-07-24 13:59:48,157 sqldb-migrate-data.py INFO: Event reports count:          163,900
2018-07-24 13:59:48,240 sqldb-migrate-data.py INFO: Event stats count:            375,573
2018-07-24 13:59:48,240 sqldb-migrate-data.py INFO: --------------------------------------------------
2018-07-24 13:59:48,240 sqldb-migrate-data.py INFO: Migration started at:  2018-07-24 13:15:49.078338
2018-07-24 13:59:48,240 sqldb-migrate-data.py INFO: Migration finished at: 2018-07-24 13:59:48.096645
2018-07-24 13:59:48,240 sqldb-migrate-data.py INFO: Migration duration:    0:43:59.018307

root@mentat-alt$ psql -f mentat-tweakdb.sql mentat_main

root@mentat-alt$ /root/system-rename.sh

# Configure all Mentat modules by comparing configuration files form old production server.
# Configure all Warden modules, switch warden client certificates.
# Reconfigure IP address settings to values of mentat-hub.cesnet.cz.

root@mentat-alt$ reboot

After rebooting bring the whole system back up:

root@mentat-alt$ mentat-controller.py --command start
root@mentat-alt$ mentat-controller.py --command enable
root@mentat-alt$ /etc/init.d/warden_filer_receiver start
root@mentat-alt$ /etc/init.d/warden_filer_sender start
root@mentat-alt$ update-rc.d warden_filer_receiver defaults
root@mentat-alt$ update-rc.d warden_filer_sender defaults