Project

General

Profile

Task #3362

Updated by Jan Mach over 5 years ago

h2. Overall migration status (periodically updated) 

 * *(DONE)* Install new base server 
 ** New server mentat-alt.cesnet.cz is ready and installed using Ansible 
 * *(DONE)* Install Mentat system and Warden client on the server 
 ** Development version of Mentat system is installed on the server using Debian package system 
 ** Warden client is installed on the server and connected to production instance of Warden server 
 * *(IN PROGRESS)* Perform data and service migration to new server 
 ** Prepare database migration scripts 
 ** Prepare filesystem migration scripts 
 ** Prepare utility migration scripts 
 ** Migrate the Mentat service 
 * Verify functionality 

 h2. General guidelines for migration process 

 # Day before migration lower the TTL of relevant DNS records for mentat-hub.cesnet.cz and mentat-alt.cesnet.cz servers 
 # Presynchronize filesystem data (rsync), so that the actual migration will be much quicker later. 
 ## report attachments 
 ## RRD databases and chart images 
 ## cache files 
 ## persistent state files 
 ## runlog files? (maybe not necessary) 
 ## log files? (maybe not necessary) 
 # Shut down Warden client on mentat-hub.cesnet.cz and mentat-alt.cesnet.cz servers and let Mentat empty all queues. 
 # Shut down Mentat systems on mentat-hub.cesnet.cz and mentat-alt.cesnet.cz servers 
 # Perform database migration 
 ## users 
 ## groups 
 ## filters 
 ## networks 
 ## reports 
 ## event statistics 
 # Perform filesystem migration - same data as above 
 # Perform configuration migration 
 ## synchronize content of @/etc/mentat@ configuration directory 
 # Switch warden client certificates between mentat-hub.cesnet.cz and mentat-alt.cesnet.cz servers 
 # Switch shibd configuration between mentat-hub.cesnet.cz and mentat-alt.cesnet.cz servers 
 # Switch hostnames and IP addresses between mentat-hub.cesnet.cz and mentat-alt.cesnet.cz servers 
 # Reboot both servers and pray to your favorite god, or as an atheist sit quietly with your hands in your lap ;-) 
 # Login to new mentat-hub.cesnet.cz and launch everything 
 ## Launch Mentat backend services (daemons and scripts) 
 ## Launch Warden client and verify messages are being stored into database 
 ## Verify that the web interface is accessible 
 # Synchronize crontab for root 

 Event migration might not be necessary. If the Mentat will be running on new server for some time and without any downtimes, we could skip slow migration of events from MongoDB to PostgreSQL. 

 h2. Migration process checklist 

 h3. Trial period before migration 

 During this period the mentat-alt.cesnet.cz server works as independent and fully operational instance of Mentat system, which can be used for testing and development purposes. 

 # *(DONE)* *mentat-alt.cesnet.cz*: Install base server. 
 # *mentat-alt.cesnet.cz*: Configure server monitoring with Nagios. 
 # *mentat-alt.cesnet.cz*: Configure server backup. 
 # *(DONE)* *mentat-alt.cesnet.cz*: Install development version of Mentat system. Keep it running and updated during trial period. 
 # *(DONE)* *mentat-hub.cesnet.cz*: Write script for periodical dump of MongoDB. 
 ## Script is called @/root/mentatdb-dump-all.sh@. 
 ## Verified, that script is working properly. 
 # *(DONE)* *mentat-alt.cesnet.cz*: Write script for periodical import of MongoDB dumps from mentat-hub.cesnet.cz. 
 ## Script is called @/root/mentat-sync-mongodb.sh@. 
 ## Verified, that script is working properly. 
 # *(DONE)* *mentat-hub.cesnet.cz*: Install cronjob for script @/root/mentat-sync-mongodb.sh@ to periodically test the import process. 
 ## Installed with following root crontab record: @5 */4 * * * /root/mentat-sync-mongodb.sh@ 
 ## The script will perform fresh dump using @/root/mentatdb-dump-all.sh@ on mentat-hub.cesnet.cz, fetch the result and import it to local MongoDB instance. 
 ## Verified, that cronjob is working properly. 

 h3. From day before migration until migration time. 

 After this period the mentat-alt.cesnet.cz is getting ready for migration process. All Mentat modules will be stopped and data will be synchronized to the local filesystem. Only web interface will be operational to some extend and can be used to verify, that migrated data will be accessible. 

 # *(DONE)* Stop all Warden client daemons. 
 # *(DONE)* Stop all Mentat modules. 
 # *(DONE)* *mentat-alt.cesnet.cz*: Write script for periodical Mentat filesystem data synchronization. 
 ## Script is called @/root/mentat-sync-files.sh@. 
 ## Verified, that script is working properly. 
 # *(DONE)* *mentat-hub.cesnet.cz*: Install cronjob for script @/root/mentat-sync-files.sh@ to periodically prefetch filesystem data to target server. 
 ## Installed with following root crontab record: @35 * * * * /root/mentat-sync-files.sh --skip-install@ 
 ## Verified, that cronjob is working properly. 
 # *(DONE)* *mentat-alt.cesnet.cz*: Prepare new networking configuration into file @/etc/network/interfaces.new@, backup current setting into file @/etc/networking/interfaces.old@. 
 ## New networking configuration configuration can be enabled by following command @cp /etc/networking/interfaces.new /etc/networking/interfaces@ and restarting the networking service. 
 # *(DONE)* *mentat-alt.cesnet.cz*: Write script for quick renaming of the server to different name. 
 ## Script will replace all ocurences of @mentat-alt@ with @mentat-hub@ in list of selected configuration files. 
 ## Script is called @/root/system-rename.sh@. 
 ## Verified, that script is working properly. 
 # *(DONE)* *mentat-alt.cesnet.cz*: Write script for quick switching of most important configurations. 
 ## Send various configuration files to source server and fetch corresponding ones from it. 
 ## Configurations like server certificates, shibboleth configurations, Warden client configurations, etc. 
 ## Script is called @/root/mentat-sync-config.sh@. 
 ## Verified, that script is working properly. 
 # *(DONE)* Perform initial database migration to be more efficient and less time consuming later. 
 ## Fetched current database with @/root/mentat-sync-mongodb.sh@. 
 ## Launch migration with @/etc/mentat/scripts/sqldb-migrate-data.py --drop@ in @tmux@ terminal. 

 h3. Actual migration process 


 notice. The migration was initiated at 2018-07-27T13:00:00+0200, so any timestamp values were relevant to that date. 

 <pre> 
 # Disable utility migration scripts that were installed before so that they do not mess with migration process. 

 root@mentat-alt$ /root/mentat-sync-config.sh 
 root@mentat-alt$ /root/mentat-sync-files.sh 
 root@mentat-alt$ /root/mentat-sync-mongodb.sh 
 root@mentat-alt$ /etc/mentat/scripts/sqldb-migrate-data.py --clear --from-timestamp 1532304000 

 root@mentat-alt$ /root/system-rename.sh 
 </pre>

Back