Consolidation of MongoDB persistent storage¶

• Table of contents
• Consolidation of MongoDB persistent storage
  • Consolidation without service interruption
  • Consolidation with service interruption

Back to TOC

Consolidation without service interruption¶

Consolidation with service interruption¶

1. Prepare another server (BACKUP) with sufficient data storage and MongoDB
2. Open firewall on both servers, so that they can talk to each other
3. Mount the BACKUP storage using sshfs:
   

   # Directory mongo-rep is the location of MongoDB database files
   sshfs -o idmap=user root@void.cesnet.cz:/mnt/vol/mongo-rep/ /media/void
   

4. Perform the initial copy of database files using cp or rsync (This step turned out to be useless, because database is constantly touching data files, so the rsync is always copying everything. The -c option for rsync is also useless, because hash calculations take too much time.)
   

   cp -R /var/lib/mongodb/* /media/void/
   # OR
   rsync -vahP /var/lib/mongodb/ /media/void
   # OR
   rsync -vahP /var/lib/mongodb/ root@void.cesnet.cz:/mnt/vol/mongo-rep/
   

1. Stop the mentat-storage daemon and database on MASTER server
   

   mentat-controller stop mentat-storage
   service mongo stop
   

2. Perform resynchronization of database files using rsync
   

   rsync -vahP /var/lib/mongodb/ /media/void
   # OR
   rsync -vahP /var/lib/mongodb/ root@void.cesnet.cz:/mnt/vol/mongo-rep/
   

3. Start MongoDB on BACKUP server, initiate replication and set it as primary replica (man1)
   

   # Make sure there is no local configuration
   rm /mnt/vol/mongo-rep/local.*
   # Remove the lock file, if present
   rm /mnt/vol/mongo-rep/mongod.lock
   # Make sure the database files have appropriate ownerships
   chown -R mongodb:mongodb /mnt/vol/mongo-rep/
   # Start the MongoDB service
   service mongodb start
   # Open the mongo shell and initiate the replication according to the manual
   mongo
   

4. Remove database files on MASTER server
   

   rm -rf /var/lib/mongodb/*
   

5. Start MongoDB on MASTER server, initiate replication and let the system perform initial sync (and repair at once) (man1)
   

   service mongodb start
   

   
   

   rs.status()
   rs.add("mentat.cesnet.cz")
   rs.status()
   cfg = rs.conf()
   cfg.members[0].priority = 2
   cfg.members[1].priority = 0
   rs.reconfig(cfg)
   rs.conf()
   rs.status()
   

6. Stop MongoDB on MASTER server, remove it from replication and restart (man2, man3, man4)
   

   service mongodb stop
   rm -rf /var/lib/mongodb/local.*
   echo Comment the --replSet option in configuration file
   service mongodb start
   

7. Start all mentat services again

Back to TOC