Consolidation of MongoDB persistent storage

Back to TOC

Consolidation without service interruption

Consolidation with service interruption

  1. Prepare another server (BACKUP) with sufficient data storage and MongoDB
  2. Open firewall on both servers, so that they can talk to each other
  3. Mount the BACKUP storage using sshfs:
    # Directory mongo-rep is the location of MongoDB database files
    sshfs -o idmap=user /media/void
  4. Perform the initial copy of database files using cp or rsync (This step turned out to be useless, because database is constantly touching data files, so the rsync is always copying everything. The -c option for rsync is also useless, because hash calculations take too much time.)
    cp -R /var/lib/mongodb/* /media/void/
    # OR
    rsync -vahP /var/lib/mongodb/ /media/void
    # OR
    rsync -vahP /var/lib/mongodb/
Beginning of service interruption
  1. Stop the mentat-storage daemon and database on MASTER server
    mentat-controller stop mentat-storage
    service mongo stop
  2. Perform resynchronization of database files using rsync
    rsync -vahP /var/lib/mongodb/ /media/void
    # OR
    rsync -vahP /var/lib/mongodb/
  3. Start MongoDB on BACKUP server, initiate replication and set it as primary replica (man1)
    # Make sure there is no local configuration
    rm /mnt/vol/mongo-rep/local.*
    # Remove the lock file, if present
    rm /mnt/vol/mongo-rep/mongod.lock
    # Make sure the database files have appropriate ownerships
    chown -R mongodb:mongodb /mnt/vol/mongo-rep/
    # Start the MongoDB service
    service mongodb start
    # Open the mongo shell and initiate the replication according to the manual
  4. Remove database files on MASTER server
    rm -rf /var/lib/mongodb/*
  5. Start MongoDB on MASTER server, initiate replication and let the system perform initial sync (and repair at once) (man1)
    service mongodb start

    cfg = rs.conf()
    cfg.members[0].priority = 2
    cfg.members[1].priority = 0
  6. Stop MongoDB on MASTER server, remove it from replication and restart (man2, man3, man4)
    service mongodb stop
    rm -rf /var/lib/mongodb/local.*
    echo Comment the --replSet option in configuration file
    service mongodb start
  7. Start all mentat services again
End of service interruption
Perhaps it would be much better to drop the indexes on the replica, perform the replication to the master server, then build only the most necessary indexes, launch the system and finish building of other indexes during production mode.

Back to TOC