GYMINUTES

Migrating Parse MongoDB to MongoRocks on AWS

Advertisements

GYMINUTES, uses parse to backup our users data. Gyminutes community has lifted more than 8.5 Million pounds till date. Thats a lot of weight and a lot of data for us to store.

We used parse as our backend because it reduced our time to market. But now Parse is going down. So we decided to migrate to AWS.

Here is a step-by-step guide to help anyone who is looking to migrate their app away from PARSE to AWS. Why only AWS, if you may ask? Because its STABLE and unlike Parse to Facebook, AWS is actually bringing in $$. So thats why. I am sure other services are great but I don’t plan to use them PERIOD. Now lets get to it.

Migrating from Parse is a simple 2 step process. They are highlighted here. (https://github.com/ParsePlatform/parse-server/wiki/Migrating-an-Existing-Parse-App)

Step 1: Migrate Mongo DB

Step 2: Migrate Parse Server. 

This post will specifically cover step-by-step guide to migrate MongoDB (Step 1). More specifically MongoDB to MongoRock.

Why mongoRock ?

Read here. The description is excellent (https://github.com/ParsePlatform/parse-server/wiki/MongoRocks)

TL;DR:

  1. Its cheap for small apps as data is compressed to roughly 10x. This means, you save $$.
  2. If it is good for parse to use it, it is good for you too.

STEP 1: MIGRATING MONGODB TO AWS

I am going to assume that you are new to AWS, Mongo and MongoRock. So I have divided this into multiple steps. Even if you are familiar with AWS and Mongo, I recommend you follow the guide below. It will reduce your debugging time.

  1. Bringing up AWS instance.
  2. Installing MongoRock on AWS
  3. Migrating data from Parse to MongoRock.
  4. Testing the data

Step 1.1 Bringing up AWS instance.

If you know how to set-up instance on AWS, jump to step 1.1.11, as its required for mongoRock.

Step 1.1.1 Login/Signup into AWS. (aws.amazon.com)

Step 1.1.2 Click “Network & Security” > “Key Pair”

Step 1.1.3 Click “Create Key Pair” on top Left tab.

Step 1.1.4 Give a name and click “Create”.

Step 1.1.5 It will prompt to download the key, save it somewhere safe. This is your private key that you will use to log into the server.

Step 1.1.6 Click “INSTANCES” > “Instance” from left menu. 

Step 1.1.7 Click “Launch Instance”

Step 1.1.8 This will bring up list of OS that you can install it on. We will be using Ubuntu. Click “Select” on ubuntu tab.

Step 1.1.9: Select the instance type. Now I am assuming your app is small. If it is big, you should size them accordingly. For setup  purpose, choose “t2.micro” and click “Next: Configure Instance Details”

Step 1.1.10: You will see the screen below. Leave everything as is and select “Protect against accidental termination” and “Enable CloudWatch detailed Monitoring” and click “Next: Add Storage”

 

Step 1.1.11: This is important. For MongoRock, you need to mount 2 more volumes. Click “Add New Volume” and add 2 volumes. It should look like screen below. Once done, click “Next: tag Instance”

Step 1.1.12: Don’t change anything unless you know what you are doing. Click  “Next: Configure Security Group”

Step 1.1.13: You would need to create a security group. Trust me this is for your own good. Just add 1 more rule to the table so we can allow external applications like parse server to talk to our mongoRock DB. It should look like this.

Step 1.1.14: Click “Review and Launch”. Verify everything and then click “LAUNCH”. As soon as you do that, a pop up will appear asking for the key. Select the key we created in step 1.1.3. Click “I ack….” and select “Launch Instance”. Click “View Instance” and wait for it to launch. 

After initialization, you should have the instance “Public IP” and DNS”.

 

Step 2.1 Installing MongoRock on AWS instance.

Step 2.1.1: Login using your private key that you downloaded in step 1.1.3 and the public ip of your new instance. As we created the ubuntu instance, the username is “ubuntu”. Open a terminal and execute the command,

#> “sudo chmod 600 mongo-key”

#> “ssh -i mongo-key ubuntu@<xx.xx.xx.xx>”

This should log you in the server. Changing permission for key is required otherwise you get error message “key too open. Permission denied”

Step 2.1.2: After you log in, execute following command in this order.

#> sudo apt-get update

#> sudo apt-get install mdadm

#> sudo mdadm –create /dev/md0 –level=stripe –raid-devices=2 /dev/xvdb /dev/xvdc
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

#> sudo mkfs -t ext4 /dev/md0
mke2fs 1.42.9 (4-Feb-2014)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=128 blocks, Stripe width=256 blocks
1048576 inodes, 4194048 blocks
209702 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
128 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

 

#> sudo mkdir -p /var/lib/mongodb

#> sudo mkdir -p /var/lib/mongodb

#> sudo mount /dev/md0 /var/lib/mongodb

#> sudo apt-key adv –keyserver keys.gnupg.net –recv-keys 1C4CBDCDCD2EFD2A
Executing: gpg –ignore-time-conflict –no-options –no-default-keyring –homedir /tmp/tmp.Iiwijlcf0j –no-auto-check-trustdb –trust-model always –keyring /etc/apt/trusted.gpg –primary-keyring /etc/apt/trusted.gpg –keyserver keys.gnupg.net –recv-keys 1C4CBDCDCD2EFD2A
gpg: requesting key CD2EFD2A from hkp server keys.gnupg.net
gpg: key CD2EFD2A: public key “Percona MySQL Development Team <mysql-dev@percona.com>” imported
gpg: Total number processed: 1
gpg: imported: 1

#> echo “deb http://repo.percona.com/apt “$(lsb_release -sc)” main” | sudo tee /etc/apt/sources.list.d/percona.list
deb http://repo.percona.com/apt trusty main

#> sudo apt-get update

#> sudo apt-get install percona-server-mongodb

..

* Starting database mongod [ OK ]
Setting up percona-server-mongodb-tools (3.0.10-1.5.trusty) …
Processing triggers for ureadahead (0.100.0-16) …
Setting up percona-server-mongodb (3.0.10-1.5.trusty) …

#> service mongod status
* Checking status of database mongod [ OK ]
ubuntu@ip-172-31-48-243:~$

If you reached here, Congratulations. Your mongo is installed.

BUT DONT START MIGRATION YET.

Step 2.1.3: Now we need to enable MongoRock. MongoRock is a compression engine that is already compiled with the mongo instance we just installed. So now we need to enable that. Perform Following steps

#> sudo service mongod stop

#> sudo vim /etc/mongod.conf

//uncomment the like “engine: rocksdb”. The config file should look like this. Save and quit

#> sudo service mongod start
* Starting database mongod [fail]
This will most likely fail. Why? Because we already had a mongo instance created before. So a mongo database do exist. And since we are new to installation and bringing up rockdb, we need to clear the directory containing the OLD database. I am assuming you don’t have any data in that database.

#> cd /var/lib/mongodb/

#> ls

#> sudo rm -rf *.*

#> sudo service mongod start
* Starting database mongod [ OK ]

WOO HOO... you are in BUSINESS.

Step 2.1.4: Check if mongo is up and running.

#> mongo
MongoDB shell version: 3.0.10-1.5
connecting to: test
Welcome to the MongoDB shell.
For interactive help, type “help”.
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
Server has startup warnings:
2016-04-28T19:18:15.702+0000 I CONTROL [initandlisten]
2016-04-28T19:18:15.702+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is ‘always’.
2016-04-28T19:18:15.702+0000 I CONTROL [initandlisten] ** We suggest setting it to ‘never’
2016-04-28T19:18:15.702+0000 I CONTROL [initandlisten]
2016-04-28T19:18:15.702+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is ‘always’.
2016-04-28T19:18:15.702+0000 I CONTROL [initandlisten] ** We suggest setting it to ‘never’
2016-04-28T19:18:15.702+0000 I CONTROL [initandlisten]
> db.serverStatus()[“rocksdb”]
{
“stats” : [
“”,
“** Compaction Stats [default] **”,
“Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop”,
“———————————————————————————————————————————————————————“,
” Sum 0/0 0.00 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 0″,
” Int 0/0 0.00 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 0″,

This means we have mongo up and running. If you want to setup a web access, do following steps:

#> sudo vim /etc/mongod.conf

Remove “bindip: 127.0.0.1” and add

http:
enabled: true
RESTInterfaceEnabled: true

The file should look like this now…

#> sudo service mongod start

You should now be able to go to your browser and type

http://<xx.xx.xx.xx&gt;:27017/ and see the message

This means your mongo is up. If you want to see actual some stats, go to your “Security group” and add new rule “All traffic”. It will look like this.

Beware that you don’t want to have this open rule. As this is just for testing, make sure you don’t have this rule when it goes to production.

BUT DONT MIGRATE YOUR DATA YET.

If you look at parse recommendation, the suggest you add following information to config file.

#> sudo service mongod stop

#> sudo vim /etc/mongod.conf

setParameter:
  internalQueryExecYieldPeriodMS: 1000
  internalQueryExecYieldIterations: 100000

Make sure you keep the indentation as it looks like above. Mongo is PICKY.

#> sudo service mongod start

Step 2.1.5: Restart the box.

#> sudo reboot

Why? Because sooner or late your box will go down. It may go down because of IT fault or because you want to resize the instance. In any case, its better if we test this now? You are now assuming that if you restart the box, mongo will comeback up and start running.

WRONG. This is what happens when it comes back up again.

#> mongo

MongoDB shell version: 3.0.10-1.5
connecting to: test
2016-04-28T22:45:42.763+0000 W NETWORK Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
2016-04-28T22:45:42.764+0000 E QUERY Error: couldn’t connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed
at connect (src/mongo/shell/mongo.js:179:14)
at (connect):1:6 at src/mongo/shell/mongo.js:179
exception: connect failed

now even if you try

#> sudo service mongod start
* Starting database mongod [fail]

it will fail. Lets look at logs.

#> tail -f /var/log/mongodb/mongod.log
2016-04-28T22:48:34.626+0000 I CONTROL ***** SERVER RESTARTED *****
2016-04-28T22:48:34.628+0000 I ACCESS Initialized External Auth Session
2016-04-28T22:48:34.630+0000 I CONTROL [initandlisten] MongoDB starting : pid=1342 port=27017 dbpath=/var/lib/mongodb 64-bit host=ip-172-31-48-243
2016-04-28T22:48:34.630+0000 I CONTROL [initandlisten] db version v3.0.10-1.5
2016-04-28T22:48:34.630+0000 I CONTROL [initandlisten] git version: 376fb629b3911b74ee040029dd2bdc57fb2927af
2016-04-28T22:48:34.630+0000 I CONTROL [initandlisten] build info: Linux vps-trusty-x64-01 2.6.32-042stab112.15 #1 SMP Tue Oct 20 17:22:56 MSK 2015 x86_64 BOOST_LIB_VERSION=1_49
2016-04-28T22:48:34.630+0000 I CONTROL [initandlisten] allocator: tcmalloc
2016-04-28T22:48:34.630+0000 I CONTROL [initandlisten] options: { config: “/etc/mongod.conf”, net: { http: { RESTInterfaceEnabled: true, enabled: true }, port: 27017 }, processManagement: { fork: true, pidFilePath: “/var/run/mongod.pid” }, setParameter: { internalQueryExecYieldIterations: “100000”, internalQueryExecYieldPeriodMS: “1000” }, storage: { dbPath: “/var/lib/mongodb”, engine: “rocksdb”, journal: { enabled: true } }, systemLog: { destination: “file”, logAppend: true, path: “/var/log/mongodb/mongod.log” } }
2016-04-28T22:48:34.651+0000 I STORAGE [initandlisten] exception in initAndListen: 98 Unable to create/open lock file: /var/lib/mongodb/mongod.lock errno:13 Permission denied Is a mongod instance already running?, terminating
2016-04-28T22:48:34.651+0000 I CONTROL [initandlisten] dbexit: rc: 100

There is a permission issue. Since we create stuff using sudo user, we still need to give permission to mongod user.

#>sudo chown -R mongod /var/lib/mongodb/

#> sudo service mongod start
* Starting database mongod [ OK ]

Thats it. You are all set.

Now reboot again and see if mongod comes up.

Step 3: Migrating data from Parse to MongoRock

Parse has made this step dead simple. So I am not going to repeat that here because that is redundant.

Step 4: Testing your data.

Never finalize anything without migrating.

The next part of Tutorial will cover bringing up parse-server and connecting that to MongoRock.

If you have any comments or suggestions please share below.

Advertisements