elastic search and spark

How come these stupid people use java as an underlining technology of stuff? Java sucks rocks. I hate java.

Neverless, wanted to get elastic search on  a server with low memory for testing purposes. Java stupid idiots don't seem to understand that things should work well with low amounts of resources. Idiots.

Anyways,
https://gist.github.com/dominicsayers/8319752
and I set
-Xms50m
-Xmx50m

in jvm.options file. 

from http://devdocs.magento.com/guides/v2.1/config-guide/elasticsearch/es-overview.html#install-prereq-java-ubuntu

to install oracle java -- yuck. 

add-apt-repository -y ppa:webupd8team/java
apt-get -y update
apt-get install -y oracle-java8-installer

https://vexxhost.com/resources/tutorials/how-to-install-elasticsearch-on-ubuntu-14-04/

It took a little bit to manage the memory for it, but I got it started. I don't think I want to use it because I don't want a hadoop cluster. yuck.

http://stackoverflow.com/questions/36152152/elasticsearch-how-to-integrate-with-mysql
https://www.cloudways.com/blog/setup-elasticsearch-with-mysql/
https://www.quora.com/How-do-I-use-MySQL-with-Elasticsearch

https://bitquabit.com/post/having-fun-python-and-elasticsearch-part-1/
https://www.unixmen.com/install-elasticsearch-ubuntu-14-04/

apt-get install python-pip

pip install elasticsearch

http://www.elasticsearchtutorial.com/elasticsearch-in-5-minutes.html

#------------------------------------------------------------------------------------

into apache spark, I hope I can avoid elastic search
https://www.linkedin.com/pulse/getting-started-apache-spark-ubuntu-1404-myles-harrison

had to set in /etc/hosts the loopback to the hostname, since aws doesn't include the hostname in /etc/hosts, stupid.

spark looks promising:
https://www.percona.com/blog/2016/08/17/apache-spark-makes-slow-mysql-queries-10x-faster/
obviously I will use the python interface

Mongo 3.4 installation

1. https://www.mongodb.com/collateral/mongodb-performance-best-practices
2. https://www.mongodb.com/blog/post/6-rules-of-thumb-for-mongodb-schema-design-part-2?_ga=1.58772142.2085632332.1453495037
3. https://blog.engineyard.com/2011/mongodb-best-practices
4. http://highscalability.com/blog/2014/3/5/10-things-you-should-know-about-running-mongodb-at-scale.html
5. https://www.mongodb.com/mongodb-architecture
6. https://www.mongodb.com/collateral/rdbms-mongodb-migration-guide
7. https://dzone.com/articles/mongodb-atlas-best-practices-part-1
8. https://www.mongodb.com/blog/post/whats-new-mongodb-30-part-3-performance-efficiency-gains-new-storage-architecture
9. https://www.mongodb.com/blog/post/tiered-storage-models-in-mongodb-optimizing

Someone asked me how to setup a replica set and shards on mongo, and its easy. I decided to do it with mongo 3.4 which is a lot more sensitive in the configuration files.

I must have deleted the steps for mongo 3.0 installation. This one will use 3 computers, on different ports, 2 data replica sets, 1 config set, 1 mongos set. So if one server goes down its okay. the config set are in a repica too. And the 2 data replica sets will be sharded.

So 3 times, install mongo 3.4 as from the website, using rpm or ubuntu or debian packages.
Install configs for mongo data replica set 1. at /etc/mongod1.conf

dbpath=/var/lib/mongodb1
logpath=/var/log/mongodb/mongod1.log
logappend=true
port = 27117
replSet=r1


directoryperdb = 1
profile = 1
smallfiles = 1
oplogSize = 16
noprealloc = 1
#not sure if this should be 16
nssize = 1
shardsvr=1

-----------------------------------------------------------

one per server, make this config file at /etc/mongo2.conf

dbpath=/var/lib/mongodb2
logpath=/var/log/mongodb/mongod2.log
logappend=true
port = 27118
replSet=r2


directoryperdb = 1
profile = 1
smallfiles = 1
oplogSize = 16
noprealloc = 1
#not sure if this should be 16
nssize = 1

shardsvr=1

-------------------------------------------

one per server, the config file for the config servers at /etc/mongod_config1.conf

dbpath=/var/lib/mongodb_config1

logpath=/var/log/mongodb/mongod_config1.log

logappend=true

#bind_ip = 127.0.0.1

port = 27217

replSet=c1

configsvr = 1

----------------------------------------------------

Now  commands to make the directories and stuff

# cd /var/lib; rm -rf mongo*

mkdir -p /var/log/mongodb/

mkdir -p /var/lib/mongodb_config1

mkdir -p /var/lib/mongodb1

mkdir -p /var/lib/mongodb2

chown -R mongodb.mongodb /var/log/mongodb /var/lib/mongo*

------------------------------------------------------------

Now the rest of the files are in /etc/init

The mongod start files, mongo config, and mongos.

These are just modifications to the files that come with normal mongo installation. 

------------------------------------------

# monogdb1

limit fsize unlimited unlimited

limit cpu unlimited unlimited

limit as unlimited unlimited

limit nofile 64000 64000

limit rss unlimited unlimited

limit nproc 32000 32000

kill timeout 300 # wait 300s between SIGTERM and SIGKILL.

pre-start script

  DAEMONUSER=${DAEMONUSER:-mongodb}

  if [ ! -d /var/lib/mongodb1 ]; then

    mkdir -p /var/lib/mongodb1 && chown mongodb:mongodb /var/lib/mongodb1

  fi

  if [ ! -d /var/log/mongodb1 ]; then

    mkdir -p /var/log/mongodb1 && chown mongodb:mongodb /var/log/mongodb1

  fi

  touch /var/run/mongodb1.pid

  chown $DAEMONUSER /var/run/mongodb1.pid

end script

start on runlevel [2345]

stop on runlevel [06]

script

  ENABLE_MONGOD="yes"

  CONF=/etc/mongod1.conf

  DAEMON=/usr/bin/mongod

  DAEMONUSER=${DAEMONUSER:-mongodb}

  if [ -f /etc/default/mongod ]; then . /etc/default/mongod; fi

  # Handle NUMA access to CPUs (SERVER-3574)

  # This verifies the existence of numactl as well as testing that the command works

  NUMACTL_ARGS="--interleave=all"

  if which numactl >/dev/null 2>/dev/null && numactl $NUMACTL_ARGS ls / >/dev/null 2>/dev/null

  then

    NUMACTL="$(which numactl) -- $NUMACTL_ARGS"

    DAEMON_OPTS=${DAEMON_OPTS:-"--config $CONF"}

  else

    NUMACTL=""

    DAEMON_OPTS="-- "${DAEMON_OPTS:-"--config $CONF"}

  fi

  if [ "x$ENABLE_MONGOD" = "xyes" ]

  then

    exec start-stop-daemon --start \

        --chuid $DAEMONUSER \

        --pidfile /var/run/mongodb1.pid \

        --make-pidfile \

        --exec $NUMACTL $DAEMON $DAEMON_OPTS

  fi

end script

------------------------------------------------------------

# monogdb2

limit fsize unlimited unlimited

limit cpu unlimited unlimited

limit as unlimited unlimited

limit nofile 64000 64000

limit rss unlimited unlimited

limit nproc 32000 32000

kill timeout 300 # wait 300s between SIGTERM and SIGKILL.

pre-start script

  DAEMONUSER=${DAEMONUSER:-mongodb}

  if [ ! -d /var/lib/mongodb2 ]; then

    mkdir -p /var/lib/mongodb2 && chown mongodb:mongodb /var/lib/mongodb2

  fi

  if [ ! -d /var/log/mongodb2 ]; then

    mkdir -p /var/log/mongodb2 && chown mongodb:mongodb /var/log/mongodb2

  fi

  touch /var/run/mongodb2.pid

  chown $DAEMONUSER /var/run/mongodb2.pid

end script

start on runlevel [2345]

stop on runlevel [06]

script

  ENABLE_MONGOD="yes"

  CONF=/etc/mongod2.conf

  DAEMON=/usr/bin/mongod

  DAEMONUSER=${DAEMONUSER:-mongodb}

  if [ -f /etc/default/mongod ]; then . /etc/default/mongod; fi

  # Handle NUMA access to CPUs (SERVER-3574)

  # This verifies the existence of numactl as well as testing that the command works

  NUMACTL_ARGS="--interleave=all"

  if which numactl >/dev/null 2>/dev/null && numactl $NUMACTL_ARGS ls / >/dev/null 2>/dev/null

  then

    NUMACTL="$(which numactl) -- $NUMACTL_ARGS"

    DAEMON_OPTS=${DAEMON_OPTS:-"--config $CONF"}

  else

    NUMACTL=""

    DAEMON_OPTS="-- "${DAEMON_OPTS:-"--config $CONF"}

  fi

  if [ "x$ENABLE_MONGOD" = "xyes" ]

  then

    exec start-stop-daemon --start \

        --chuid $DAEMONUSER \

        --pidfile /var/run/mongodb2.pid \

        --make-pidfile \

        --exec $NUMACTL $DAEMON $DAEMON_OPTS

  fi

end script

#---------------------------

#mongo config

limit fsize unlimited unlimited

limit cpu unlimited unlimited

limit as unlimited unlimited

limit nofile 64000 64000

limit rss unlimited unlimited

limit nproc 32000 32000

kill timeout 300 # wait 300s between SIGTERM and SIGKILL.

pre-start script

  DAEMONUSER=${DAEMONUSER:-mongodb}

  if [ ! -d /var/lib/mongodb_config1 ]; then

    mkdir -p /var/lib/mongodb_config1 && chown mongodb:mongodb /var/lib/mongodb_config1

  fi

  if [ ! -d /var/log/mongodb ]; then

    mkdir -p /var/log/mongodb && chown mongodb:mongodb /var/log/mongodb

  fi

  touch /var/run/mongodb_config1.pid

  chown $DAEMONUSER /var/run/mongodb_config1.pid

end script

start on runlevel [2345]

stop on runlevel [06]

script

  ENABLE_MONGOD="yes"

  CONF=/etc/mongod_config1.conf

  DAEMON=/usr/bin/mongod

  DAEMONUSER=${DAEMONUSER:-mongodb}

  if [ -f /etc/default/mongod ]; then . /etc/default/mongod; fi

  # Handle NUMA access to CPUs (SERVER-3574)

  # This verifies the existence of numactl as well as testing that the command works

  NUMACTL_ARGS="--interleave=all"

  if which numactl >/dev/null 2>/dev/null && numactl $NUMACTL_ARGS ls / >/dev/null 2>/dev/null

  then

    NUMACTL="$(which numactl) -- $NUMACTL_ARGS"

    DAEMON_OPTS=${DAEMON_OPTS:-"--config $CONF"}

  else

    NUMACTL=""

    DAEMON_OPTS="-- "${DAEMON_OPTS:-"--config $CONF"}

  fi

  if [ "x$ENABLE_MONGOD" = "xyes" ]

  then

    exec start-stop-daemon --start \

        --chuid $DAEMONUSER \

        --pidfile /var/run/mongodb_config1.pid \

        --make-pidfile \

        --exec $NUMACTL $DAEMON $DAEMON_OPTS

  fi

end script

#---------------------------------------------------------------------

# mongos
# Change [IP] to your 3 ip addresses for the mongo config servers with ports, 

limit fsize unlimited unlimited
limit cpu unlimited unlimited
limit as unlimited unlimited
limit nofile 64000 64000
limit rss unlimited unlimited
limit nproc 32000 32000

kill timeout 300 # wait 300s between SIGTERM and SIGKILL.

pre-start script
  DAEMONUSER=${DAEMONUSER:-mongodb}
  touch /var/run/mongos1.pid
  chown $DAEMONUSER /var/run/mongos1.pid
end script

start on runlevel [2345]
stop on runlevel [06]

script
  CONFIG=" c1/[IP]:27217,[IP]:27217,[IP]:27217"
  DAEMON=/usr/bin/mongos
  DAEMONUSER=${DAEMONUSER:-mongodb}

  start-stop-daemon --start \
        --chuid $DAEMONUSER \
        --pidfile /var/run/mongos1.pid \
        --make-pidfile \
        --exec $DAEMON -- --configdb $CONFIG --port 27017  --logpath  /var/log/mongodb/mongos1.log

end script

#-----------------------------------


0. Copy all of it to the 3 servers.
1. Setup replica set  r1.
2. Setup replica set  r2

3. Setup config servers in replica set.
4. Setup mongos --- each server should have a mongos.

#--------------------------------------------------------------------------
On all three servers:
1. start mongodb1
2. start mongodb1
3. start mongodb_config1

#------------------------------
1. connect to mongodb1: mongo --port  27117
2. Setup replication
rs.initiate(
   {
      _id: "r1",
      version: 1,
      members: [
         { _id: 0, host : "IP1:27117" },
         { _id: 1, host : "IP2:27117" },
         { _id: 2, host : "IP3:27117" }
      ]
   }
)
3. After a minute it should make itself primary. Do rs.config() to see if the other servers connected.

#-------------------------------------------------------
4. connect to mongodb1: mongo --port  27118
5. Setup replication
rs.initiate(
   {
      _id: "r2",
      version: 1,
      members: [
         { _id: 0, host : "IP1:27118" },
         { _id: 1, host : "IP2:27118" },
         { _id: 2, host : "IP3:27118" }
      ]
   }
)
6. After a minute it should make itself primary. Do rs.config() to see if the other servers connected.

#-----------------------------------------------------
# now the config servers.
7. connect to mongodb1: mongo --port  27118
8. Setup replication
rs.initiate(
   {
      _id: "r2",
      version: 1,
      members: [
         { _id: 0, host : "IP1:27217" },
         { _id: 1, host : "IP2:27217" },
         { _id: 2, host : "IP3:27217" }
      ]
   }
)

9. After a minute it should make itself primary. Do rs.config() to see if the other servers connected.

#---------------------------------------------------------------
# At this point, I would connet to each replica set, 3 of them, do rs.conf() and make sure everything in running. Also, ps auwx | grep mongo helps to see if you have 3 mongo processes running. Check the log files too.

1. Now setup mongos on each server the same way.
2. You should had already installed the mongos init files, just start it.
3. start mongos
4. connect to mongo: mongo
5. Now setup the sharding. If you had problems connecting to mongos, there should be a log file for it.

#---------------------------------------------
1. Connected to mongos using mongo client, do the following commands.

sh.addShard( "r1/[IP]:27117")
sh.addShard( "r2/[IP]:27118")

use testsharddb1
db.testsharddoc1.insert( { _id: 11, junk : "aaa" } )
# You should have been able to insert and make a document. Now look at the sharding setup.
sh.status()
# You should see our database is not sharded yet.
# It should say somewhere: {  "_id" : "testsharddb1",  "primary" : "r2",  "partitioned" : false }

sh.enableSharding("testsharddb1")

# Put we have to specify documents to shard too.
sh.shardCollection("testsharddb1.testsharddoc1", { _id : 1 } )

# Nowif you so sh.status() you should see the database and document sharded and the document should have a range for the key on one of the replica sets. 

aws

I have using AWS for a long time and I decided to do the following after I had a contract gig:
1. Integrate creating/destroying systems with DAD.
2. Run tests on the instance of MySQL RDS, Mongo, or Aurora and then destroy. RDS and Aurora are EXPENSIVE.
3. Add MySQL or Mongo and get replication setup with MySQL or Add a node to Mongo if needed. This one might take a while.
4. When AWS needs to upgrade something, print out cli commands to do the commands. This should not be auto done in general. For most people you don't want that.

#-----------------------------
Making Credentials
1. http://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html
2. Create admin group
3. Create user and assign to group. Save credentials which will be used for api and cli.
4. Add password. Save password ( I think it is also used for api and cli).
5. Install the credentials.
  a. http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html
  b. sudo pip install awscli
  c.  aws help
 d. aws configure: need access key id, secret access key, default zone. If zone is us-west-2c, put in us-west-2. 
  e. TEST: aws ec2 describe-instances
6. Execute "aws iam list-groups" to test if the setup is okay.
aws iam list-groups

{
    "Groups": [
        {
            "Path": "/", 
            "CreateDate": "2014-06-05T20:29:53.622Z", 
            "GroupId":"ABCDEFGHABCDEFGHABCDE", 
            "Arn": "arn:aws:iam::123456789012:group/Admins", 
            "GroupName": "Admins"
        }
    ]
}

7. OPTIONAL: put this stuff on an aws server, perhaps use a limited admin account that can just read stuff. Perhaps have it only print out commands needed. But we have to destroy after an hour. Hmmmmm. 

AWS links for cli I will pay attention to:

1. http://docs.aws.amazon.com/cli/latest/reference/

2. http://docs.aws.amazon.com/cli/latest/reference/ec2/index.html

3. http://docs.aws.amazon.com/cli/latest/reference/rds/index.html

I don't need all the servers I have, so I will manually destroy them, get the cli to make them, and see if they are made with the right server types. Once ec2 works out, then I moved onto to RDS. 

#-----------------------

api -- i did this way before, just writing down the quick steps

1. https://boto3.readthedocs.io/en/latest/guide/quickstart.html

2. pip install boto3

3. apt-get install awscli

4. aws configure --- just for testing, we will be submitting values in DAD not looking at the home users settings
For example:
import boto3

client = boto3.client(
    's3',
    aws_access_key_id=ACCESS_KEY,
    aws_secret_access_key=SECRET_KEY,
    aws_session_token=SESSION_TOKEN,
)

5. https://boto3.readthedocs.io/en/latest/index.html

I care only about EC2 and mysql and aurora stuff right now. 

6. a. https://boto3.readthedocs.io/en/latest/reference/services/ec2.html

  b. https://boto3.readthedocs.io/en/latest/reference/services/es.html

  c. https://boto3.readthedocs.io/en/latest/reference/services/rds.html

   d. 

https://boto3.readthedocs.io/en/latest/reference/services/redshift.html

7. http://boto3.readthedocs.io/en/latest/reference/core/session.html

Setting up a website for little money

Goals
1. Free DNS. Will forward all traffic to an aws server. Forward all email to gmail.
2. Setup website. Basic at first.
3. Setup a gmail account $5 a month for using their office stuff.
4. Setup facebook. Done. But populate it with webpage.
5. Setup Blogger -- DONE. But not going to use it anymore.
6. Setup mongo or couchdb for fun. It will hold the categories of jiujitsu stuff. And secret location of videos downloaded. Webpage to upload secret videos. Might not have enough diskspace for videos.
7. Meetup?

First need DNS.
1a. Setup DNS account on bulkregister.
1b. Setup DNS on free DNS to point to exisitng AWS server and gmail. use namecheap.com
      I had to fight for a while to get this setup. Wait 2 days for DNS to propagate.
2. Setup a website just to say hello so when you choose domain it goes there.

Wait until those two steps are done.

Cassandra Certification, and others.

Update2: I need to merge this with the other certification page I made. 

UPDATE: After the 1st test for cassandra, do AWS certification, then the 2nd test for cassandara. Another thing to do, with redshift and mysql on AWS, learn to make an instance and only use it for an hour. Then destroy it. Perhaps script it to do something then save the output.

You have to go through 2 $300 tests to get the basic certification. There are a lot of classes I need to put up here and study guides. I don't know if there are cheap or free tests to take.

I have one cassandra test to take in May, so I might as well finish it out. I think in order, then, vertica, hadoop, coudbdb (but I consider this minor certifications), and maybe voltdb if it looks like it is economically worth it. If I had known vertica was cheap, I would have done it instead of starting on cassandra.

I just started this, so I apologize there isn't much here. I choose Cassandra because I have reservations about Hadoop. VoltDB and vertica also interest me and I might wants certifications in those, as well as couchDB.

Some quick stuff:

Preparation
There has to more preparation stuff, like youtube.

• Some stuff lined from oreilly
• https://www.webassessor.com/wa.do?page=certInfo&branding=OREILLY&tabs=13

Getting certified

• https://academy.datastax.com/Certifications

Cassandra Tutorials (didn't take these yet) from udemy

•  Learn Apache Cassandra from Scratch
• There are about 3 or 4 more courses at udemy for cassandra. 

------------------------------------------------

Other misc crap that needs its own blog

CouchDB

I am not thrilled by these options. I might look for more. 

• https://www.udemy.com/learn-nosql-database-design-with-couchdb/
• http://www.kerneltraining.com/couchdb
• Not sure if this is expensive. 
• http://www.vskills.in/certification/Certified-Apache-CouchDB-Professional
• I am have no idea if we can even do it. Its in India for the government. 

voltdb

• https://university.voltdb.com/
• $2000 YIKES!!!!

hadoop

• http://hortonworks.com/training/certification/hdpca-certification/
• $250 makes this attractive. Maybe more than cassandra, or both. 

Vertica

• From HP. I suspect these exams are not too expensive. Need to check. This also is very attractive. 
• https://my.vertica.com/resources/certification/
• https://my.vertica.com/resources/certification/client-certification/

AWS https://aws.amazon.com/certification/

With the direction the cloud is going, it is important as a DBA to know AWS in an out to best make recommendations and to handle cloud environments. If you get this, it still works in favor for other cloud environments too. 

• From AWS -- The exams are $150 each I think, but whatever, they are cheap. 
• https://aws.amazon.com/certification/certified-solutions-architect-associate/
• https://aws.amazon.com/certification/certified-solutions-architect-professional/
• https://aws.amazon.com/certification/certified-sysops-admin-associate/
• https://aws.amazon.com/certification/certified-devops-engineer-professional/
• Also, they have practice exams for $20, I heavily recommend these. 
• Udemy -- just cheap classes
• https://www.udemy.com/aws-certified-developer-associate/
• https://www.udemy.com/aws-solutions-architect-associate-certification-course2016/
• https://www.udemy.com/aws-certified-solution-architect-associate-level-training/
• https://www.udemy.com/aws-certified-solutions-architect-associate/
• https://www.udemy.com/aws-certified-sysops-administrator-associate/
• Online documentation
• https://aws.amazon.com/documentation/vpc/
• https://aws.amazon.com/documentation/

Chef certification --- This is good to learn. chef I think is the 2nd most popular one out there.

•  https://training.chef.io/certification
• there are free tutorials

Puppet certification --- the most popular one

•  https://puppet.com/support-services/certification

DAD

Tasks finished.

1. Installed redis
1. sudo apt-get -y install redis-server
2. The default configuration with no password and tied to the loopbask is fine. In production environments, of course add a password. 
3. You may want to download and install the latest version. 
4. https://redislabs.com/python-redis\
1. pip install redis

Well now that I have mongo and postgresql certification, I am ready to start my dad project again. It will initially support MySQL, Mogno, and PostgreSQL without replication. Then add in replication, vertica, voltdb, and cassandra. There are different types of replication, and it can be a pain. I will update the progress to this blog for DAD.

Project list:

Part 1
1. Load from YAML options file what database and location to run DAD from. Initially MySQL, Postgresql, or Mongo.
2. Basic connection for database.
3. Initial DAD setup scripts.
4. Store process information
5. Store OS information
6. Store misc information
7. Store diskspace according to database and according to OS.
8. One module for each database type with the same functions to get data.
9. Each module will have a support module.
10. Modules are dynamically loaded when needed and otherwise abort. Aborting one type of database doesn't kill the scripts from running with the other databases.
11. Log file for errors which rotates.
12. Global module is database independent.
13. All data transfer, inserting and selecting data, will be done by either json or a python dictionary (probably python). Also, printing and inserting data will be checked for on each field so that a print won't abort a server if there is no info, and inserts won't abort if one field is missing. There should never be NULLs in fields. use empty spaces. NULLs mean no data was inserted.

Part 1-2.
1. Make global available dashboard on AWS for systems.

Part 2
1. Automatic dev packages. Dependecy checks and not override later versions.
2. Automatic rpm packages. Dependency checks and not override later versions.
3. Everything in YAML will have variables that will not collide between versions, or minimally. It should work by default if no options are given and state what the default options are.
4. Encrypt data option which is needed at startup by a config file or a loaded program. Have program run only on localhost.

Part 3
1. voltdb
2. couchdb
3. cassandra
4. vertica

Part 4
1. Add in replication and cluster columns for all databases.

Part 5
1. Run DAD from other databases. (have to chose which ones).

Part 5
1. sqllite
2. bsd files with replication

-----------------------------------------------------
Misc stuff I neded to do:

1. Install Percona MySQL 5.7. I used Linux Mint instead of Ubuntu because Ubuntu hasn't figured out in 10 years their stupid Unity desktop sucks rocks. 
1. wget https://repo.percona.com/apt/percona-release_0.1-3.trusty_all.deb
2. sudo dpkg -i percona-release_0.1-3.trusty_all.deb 
3. sudo apt-get update
4. sudo apt-get install percona-server-server-5.7
5. Its uses a plugin system for passwords now, and you have to be as root to login when you don't have a password. You have to change the plugin used for the account and password. Overall its a good move. 

Mongo and postgresql were already installed. For now, just doing no replication dashboard.

The memory overhead for these 3 systems is VERY low. So I just leave them on.

Driver installation:

• postgresql
• http://initd.org/psycopg/docs/install.html
• sudo apt-get install python-psycopg2
• pip install psycopg2
• OR
• apt-get install python-psycopg2
• MySQL
• sudo apt-get install python-mysqldb
• I didn't use the pip version. 
• Mongo
• python -m pip install pymongo

PostgreSQL certification (status: passed)

You can now get postgresql certification online for the associate and then the advanced for $200 each. I am not sure of the process.

• udemy classes --- most of these classes I don't want. Not beginner. 
• https://www.udemy.com/beginners-guide-to-postgresql/
• https://www.udemy.com/python-programming-with-postgresql/
• https://www.udemy.com/learn-database-design-using-postgresql/learn/#/
• https://www.udemy.com/draft/371372/learn/#/
• PostgreSQL: Learn how to deploy Streaming Replication
•  I wanted to learn about the latest replication in postgresql. I don't think it is as good as MySQL. I have to do some tests. 
• Note: http://www.thatguyfromdelhi.com/2014/12/postgresql-certification.html
• Certifications
• PostgreSQL 9.5 Associate Certification
• http://www.enterprisedb.com/store/products/dba-training/01t50000001Nyi7AAC
• PostgreSQL 9.5 Professional Certification
• http://www.enterprisedb.com/store/products/dba-training/01t50000001NyiCAAS
• This might require an expensive ridiculous class. 
• BrainMeasures -- they have two free tests. Might as well take before the others. 
• Login first. 
• http://www.brainmeasures.com/online/newtests/217/postgresql-certification.aspx#qq
• http://www.odesk-tests.net/odesk-test-answers/databases/postgresql-test-answers-2016.html
• You can use that to cheat, but I don't recommend do. 
• http://www.brainmeasures.com/online/newtests/692/postgresql-8-1-3-certification.aspx#qq
• Even though its oracle, postgresql is copied from it a lot:
• http://www.tutorialspoint.com/sql_certificate/index.htm
• Cheating -- not sure which tests these are for, but you should study the things that you didn't know. 
• http://www.odesk-tests.net/odesk-test-answers/databases/postgresql-8_1_3-test-answers-2015/questions.html
• http://www.odesk-tests.net/elance-test-answers/it-programming/postgresql-answers-2015.html
• http://www.odesk-tests.net/odesk-test-answers/databases/postgresql-rdbms-test-answers-2015.html

Mongo certification (status: passed one test, will do the other someday)

UPDATE: Got the mongo dba certifcation. Now I want to get the dev one too.

I will be doing two things here. Listing the urls for sample tests and good reading material. And, setup step by step example of stuff to do to learn it.

NOTE: The mongodb certification exams are once a month. I highly recommend taking the free courses at https://university.mongodb.com/courses/catalog which will help with certification. Here's also a trick. The certification exams are $150. A class is thousands of dollars. Take it, fail it, learn what you were weak on, take it again. That's what I did for the MySQL certifications, but I usually passed them the first time. Out of 6 MySQL exams, I barely didn't pass one, studied it, and took it again. Much cheaper than thousands of dollars on courses. I will do the same for Mongo certifications, so we will see how I do on the exams. I have lots of previous Mongo experience, so that helps.

I will rate these links when I get done with them.

• Mongo study guides (need to rate these two --- the videos and such)
• https://university.mongodb.com/exam/C100DEV/2016_March/guide
• https://university.mongodb.com/exam/C100DBA/2016_March/guide
• http://www.brainmeasures.com/online/tests/2718/mongodb-test.aspx#qqq
• http://www.menprojects.com/class_notes/EXAMS/brainmeasures_notes.txt
• They try to get you to pay $50 for a lot of questions that were taken from the manual. But to give them credit the test is free and good practice. 
• Register first. 
• Take test, copy the image locally, copy the webpage of the test results locally. 
• Don't buy certificate. I don't think $50 is worth. Good free test but not a great certification test. 
• http://www.tutorialspoint.com/mongodb/mongodb_overview.htm
• Good basic review. Go through and do all their examples. 
• http://www.sanfoundry.com/1000-mongodb-questions-answers/
• Good questions. I suggest to go over the questions and focus on topics you are weak at.
• https://quizlet.com/32272213/mongo-flash-cards/
• These aren't that great. 
• Cheating -- not sure which tests these are for. Study what you don't know
• http://www.odesk-tests.net/elance-test-answers/it-programming/mongodb-answers-2015.html
• http://www.odesk-tests.net/elance-test-answers/it-programming/mongodb-answers-2015/questions.html

----------------------------------------------

All of these steps below will be linked to a file. I havent started it yet as I am looking at the links above, but I will note when I start them and put in the link. 

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse"\
 | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
sudo apt-get update
#sudo apt-get install -y mongodb-org
sudo apt-get install -y mongodb-org=3.2.4 mongodb-org-server=3.2.4 mongodb-org-\
shell=3.2.4 mongodb-org-mongos=3.2.4 mongodb-org-tools=3.2.4

Step 1. Setup a standalone mongo on port 3000. 

Step 2. Setup 5 mongo node cluster on same machine. 3 config servers. 1 mongos. 

Step 3. Start practice stuff.
----------------------------------
Other stuff

• Leanr mongo chef. It is free for non commercial use. Might as well. 
• http://3t.io/mongochef/download/
• https://www.mongodb.com/collateral/quick-reference-cards
• Memorize these after a little practice with each of the sections. 
•