https://homeproj.cesnet.cz/https://homeproj.cesnet.cz/httpauth-login/favicon.ico?16194486082018-05-31T11:57:56ZHomeproj: Redmine for CESNETMentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=141942018-05-31T11:57:56ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Priority</strong> changed from <i>Low</i> to <i>High</i></li></ul><p>Development version of Mentat system is installed on new hardware. Currently it is being used for debugging and testing purposes before releasing new stable version. Database and filesystem migration scripts are ready, but might need one more revision.</p> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=141952018-05-31T11:58:07ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Related to</strong> <i><a class="issue tracker-5 status-5 priority-4 priority-default closed" href="/issues/3752">Task #3752</a>: Migration from MongoDB to PostgreSQL</i> added</li></ul> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=141972018-05-31T11:58:22ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Related to</strong> <i><a class="issue tracker-5 status-5 priority-4 priority-default closed child" href="/issues/3734">Task #3734</a>: Migrate Hawat web user inteface from Perl-base to Python-based Mentat framework</i> added</li></ul> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=141992018-05-31T11:58:32ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Related to</strong> <i><a class="issue tracker-5 status-5 priority-5 priority-high3 closed parent" href="/issues/3374">Task #3374</a>: Migrate all core modules from legacy Mentat</i> added</li></ul> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=142012018-05-31T12:18:51ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/14201/diff?detail_id=14532">diff</a>)</li><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li><li><strong>Assignee</strong> changed from <i>Jan Mach</i> to <i>Pavel Kácha</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>30</i></li></ul> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=142022018-05-31T14:01:31ZPavel Káchaph@cesnet.cz
<ul><li><strong>Assignee</strong> changed from <i>Pavel Kácha</i> to <i>Jan Mach</i></li></ul><blockquote>
<p>The actual process of migration will be done according to the following checklist:</p>
</blockquote>
<p>Hint: Set short (~minutes) TTL on all related A/AAAA/CNAME/PTR RRs.</p>
<blockquote>
<ol>
<li>Presynchronize filesystem data (rsync), so that the actual migration will be much quicker.</li>
</ol>
</blockquote>
<p>Except db perhaps?</p>
<blockquote>
<ol>
<li>Shut down Mentat and Warden systems on mentat-hub.cesnet.cz and mentat-alt.cesnet.cz servers</li>
</ol>
</blockquote>
<p>Hint: <strong>Disable</strong> automatic start of whatever does state changes - warden-filer, cron scripts, automatic downloads, etc.<br />Hint: Also <strong>disable</strong> start of Mentat itself...</p>
<blockquote>
<ol>
<li>Perform database migration</li>
</ol>
</blockquote>
<p>So real migration of data or just run with month of already saved data? (No hard opinion here, we can import older data later if we find it important.)</p>
<blockquote>
<ol>
<li>Perform filesystem migration</li>
</ol>
</blockquote>
<p>rsync again? Or do you mean something else?</p>
<blockquote>
<ol>
<li>Perform configuration migration</li>
<li>Switch warden client certificates between mentat-hub.cesnet.cz and mentat-alt.cesnet.cz servers</li>
<li>Switch hostnames and IP addresses between mentat-hub.cesnet.cz and mentat-alt.cesnet.cz servers</li>
<li>Reboot both servers and pray to your favorite god, or as an atheist sit quietly with your hands in your lap <span class="wiking smiley smiley-wink" title=";-)"></span></li>
</ol>
</blockquote>
<p>Ph'nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn!</p>
<blockquote>
<ol>
<li>Login to new mentat-hub.cesnet.cz, launch all services</li>
</ol>
</blockquote>
<p>Hint: If only basic system started automatically, daemons start can be tested by hand from the end (starting from storage), and data inflow (warden-filer) and disruptive scripts can be started only when everything is checked as ok.</p> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=142152018-06-01T15:45:10ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/14215/diff?detail_id=14553">diff</a>)</li><li><strong>Status</strong> changed from <i>Feedback</i> to <i>In Progress</i></li></ul> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=144692018-07-09T12:48:34ZREST Automat Admin
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/14469/diff?detail_id=14856">diff</a>)</li></ul> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=144712018-07-09T13:01:34ZREST Automat Admin
<ul></ul><a name="Remarks-regarding-database-migration"></a>
<h1 >Remarks regarding database migration<a href="#Remarks-regarding-database-migration" class="wiki-anchor">¶</a></h1>
<p>MongoDB database dump on server <strong>mentat-hub.cesnet.cz</strong>:</p>
<ul>
<li><code>/root/mentatdb-dump-all.sh</code> (dump script for Mentat databases)</li>
</ul>
<p>MongoDB database restore on server <strong>mentat-alt.cesnet.cz</strong>:</p>
<ul>
<li><code>/root/mentat-sync-db.sh</code> (executed regularly at 8am by cron to verify functionality)</li>
</ul>
<p>MongoDB -> PostgreSQL database migration on server <strong>mentat-alt.cesnet.cz</strong>:</p>
<ul>
<li><code>/etc/mentat/scripts/sqldb-migrate-data.py</code> (migrate metadata database containing users, groups, reports, statistics, etc.)</li>
<li><code>/etc/mentat/scripts/sqldb-migrate-events.py</code> (migrate IDEA messages, might not be necessary)</li>
</ul>
<p>At this point database migration should be ready.</p>
<a name="Remarks-regarding-data-migration"></a>
<h1 >Remarks regarding data migration<a href="#Remarks-regarding-data-migration" class="wiki-anchor">¶</a></h1>
<p>Migrate data:</p>
<ul>
<li><code>rsync --archive --update --delete --progress /var/mentat root@target:/var</code></li>
</ul>
<p>Cleanup runlogs and logs (might cause issue with new version):</p>
<ul>
<li><code>find /var/mentat/log -name=*.log* -delete</code></li>
<li><code>find /var/mentat/run -name=*.runlog -delete</code></li>
<li><code>find /var/mentat/run -name=*.pstate -delete</code></li>
<li><code>find /var/mentat/run -name=*.state -delete</code></li>
</ul> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=144962018-07-11T12:14:27ZPavel Káchaph@cesnet.cz
<ul></ul><p>user#373 wrote:</p>
<blockquote>
<p>At this point database migration should be ready.</p>
</blockquote>
<p><strong>should</strong> implies it might not. What if something goes awry? <span class="wiking smiley smiley-smiley" title=":)"></span></p>
<blockquote>
<p>Cleanup runlogs and logs (might cause issue with new version):</p>
</blockquote>
<p>What issue? Something critical?</p> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=144982018-07-12T11:59:46ZJan Machjan.mach@cesnet.cz
<ul></ul><p><a class="user active" href="https://homeproj.cesnet.cz/users/17">Pavel Kácha</a> wrote:</p>
<blockquote>
<p>user#373 wrote:</p>
<blockquote>
<p>At this point database migration should be ready.</p>
</blockquote>
<p><strong>should</strong> implies it might not. What if something goes awry? <span class="wiking smiley smiley-smiley" title=":)"></span></p>
</blockquote>
<p>You can never be 100% sure <span class="wiking smiley smiley-wink" title=";-)"></span> I have tested that many many times, so that the <strong>should</strong> is as close to <strong>will</strong> as possible <span class="wiking smiley smiley-wink" title=";-)"></span>.</p>
<blockquote><blockquote>
<p>Cleanup runlogs and logs (might cause issue with new version):</p>
</blockquote>
<p>What issue? Something critical?</p>
</blockquote>
<p>Some modules have additional runlog attributes. Everything is written with backwards compatibility in mind, but some really old runlogs could cause problems. However these problems will only show when evaluating runlogs using <code>--action=runlogs-evaluate</code> module action. So these possible problems are not critical, they just make the deloper look bad.</p> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=145082018-07-13T18:32:51ZJan Machjan.mach@cesnet.cz
<ul></ul><p>New Mentat installation guide in official documentation:</p>
<p><a class="external" href="https://alchemist.cesnet.cz/mentat/doc/development/html/_doclib/installation.html">https://alchemist.cesnet.cz/mentat/doc/development/html/_doclib/installation.html</a></p>
<p>New Mentat migration guide in official documentation:</p>
<p><a class="external" href="https://alchemist.cesnet.cz/mentat/doc/development/html/_doclib/migration.html">https://alchemist.cesnet.cz/mentat/doc/development/html/_doclib/migration.html</a></p>
<p>New Mentat reporting guide in official documentation:</p>
<p><a class="external" href="https://alchemist.cesnet.cz/mentat/doc/development/html/_doclib/reporting.html">https://alchemist.cesnet.cz/mentat/doc/development/html/_doclib/reporting.html</a></p> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=145942018-07-20T14:52:01ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/14594/diff?detail_id=15016">diff</a>)</li></ul> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=146072018-07-23T12:31:43ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/14607/diff?detail_id=15030">diff</a>)</li></ul> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=146092018-07-23T17:32:42ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/14609/diff?detail_id=15032">diff</a>)</li></ul> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=146102018-07-23T18:36:29ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/14610/diff?detail_id=15033">diff</a>)</li></ul> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=146122018-07-24T11:28:00ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/14612/diff?detail_id=15037">diff</a>)</li><li><strong>Category</strong> changed from <i>Installation</i> to <i>Documentation</i></li></ul> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=146132018-07-24T12:09:43ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/14613/diff?detail_id=15039">diff</a>)</li></ul> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=146192018-07-24T13:29:12ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/14619/diff?detail_id=15048">diff</a>)</li></ul> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=146582018-07-27T09:51:16ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Category</strong> changed from <i>Documentation</i> to <i>Installation</i></li><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li><li><strong>% Done</strong> changed from <i>30</i> to <i>100</i></li></ul><p>Migration was successfully performed on 24.7. 2018. Waiting for any feedback from users before closing as successfull.</p> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=146872018-07-27T12:33:44ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Related to</strong> <i><a class="issue tracker-5 status-5 priority-5 priority-high3 closed behind-schedule" href="/issues/4210">Task #4210</a>: Release and deploy Mentat package version 2.0</i> added</li></ul> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=149592018-09-03T07:36:21ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>In Progress</i></li></ul><ul>
<li>All Ansible roles related to Mentat server management were improved and polished.</li>
<li>Automated build system Alchemist received big overhaul and is now back online. It provides building packages of newly introduced <strong>release</strong> suite, which is something in between of <strong>development</strong> and <strong>production</strong>. This is going to enable us test the Mentat code in our production environment before releasing it as true <strong>production</strong> level code.</li>
<li>I am now waiting for confirmation from the manager of our monitoring system based on Nagios, that he updated the monitoring configuration according to new requirements.</li>
</ul> Mentat - Task #3362: Migrate Mentat system to new hardwarehttps://homeproj.cesnet.cz/issues/3362?journal_id=150262018-09-11T08:36:48ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Closed</i></li></ul><p>Migration complete, all Nagios monitoring scripts are fixed, up and running. Closing issue as resolved, this also completes the work on version 2.0.</p>