4-22-2015: misc garbage over april

Just decided to combine all the useless posts 4-2-15 2 redis classes online 2 centos AWS servers -- should get a third. Need to convert install scripts. Setup fail2ban signed up for hadoop course thing. There are a few of them. Might as well get it over with. headset/mic in phone now I can call from computer, much easier. I have a minipad I can still do conferencing with. started python project on ramsey and DAD Looking at method to separate read/write with redis and to use redis with mongo since mysql just takes up too much ram. Maybe at home I can do it, but I need to setup my third laptop, but it has windows on it which is required for mongo certification --- remote exam and stuff. That sucks. ------------------------------- 4-6-2015 busy busy busy Taking pivotal course on redis. Finished 3rd class on redit from udemy over the weekend. Very very good. I highly recommend "Redis" from udemy. The others are useless. Hopefully I can do more programming for ramsey, DAD, and get redis into DAD as well (basically a one second ttl when getting data in case everybody tries to beat up a server to get data). 4-7-2015 Finished pivotal redis course, got certficate, granted its online. Such a bum for not getting more done. 4-9-2015 Hadoop is understandable, but a big project from the Mapr classes. They will take a long time to do. I won't want it to be endless -- but its good. Need some practice first. Will take some easy free classes at udemy for an hour a day while I work on other stuff. 4-13-2015 1. Singed up for puppet certification for end of month. After dealing with OPS and a person who could never seem to help because he was too distracted and only into himself, I don't want to rely on someone. I want to know if it is so easy that it can be done, and why not. 2. Finished personal crap, but more to do. 3. Need to finish that stupid hadoop course --- with mongodb and puppet certification exams, hadoop will just be casually studied with easy online sources. Exam will have to wait. 4. Still waiting on getting DAD setup with redis (on myself). Its funny how some people are so into themselves they learn technologies so that they have job security but they are not really that great at it, but since they are the only person you can rely on, they get away with murder. 4-14-2015 Starting puppet training course. https://www.udemy.com/learning-puppet/?dtcode=NBTtnMS2K1qq Follow this first: https://www.howtoforge.com/puppet-ubuntu-14.04 which then I have an exam to take a Person Vue for this, and I don't care if I fail the first time. You know what they are asking and can study for it. Often a whole book or course contains lots of useless information and you don't know what is important. I don't recommend "Basic overview of Big Data Hadoop" as it was hard to follow. Giving Hadoop a little break because I was to get certified in puppet, Mongodb, and PostgreSQL first. To me, OPS people take control of your life by running Nagios, Thruk, Graphite, Grafana, and puppet and you can't get anything done because they are too busy. In addition, some people take it on themselves to be experts in those areas and they suck but nobody knows they suck because their boss doesn't know it and he is an idiot. No this isn't about any friend I have (not you who is an expert at quality control). Its about an overworked idiot who specialized in stuff his boss (who couldn't even program) knew nothing about and acted like he was special. The best way to defend against selfish political people is to be able to do what they do so you can show them up. Their lies and stupidity come to a halt. I had a guy who always had to win. I agreed with him 50% of the time, he never agreed with me. He was good most of the time, but sometimes he was incredibly stupid. His boss wasn't any better. Good most of the time, but sometimes just short sighted and dumb. You have to be independent to show people reality sometimes. Its good not to be arrogant or to show you are really better, but plant the idea in their heads and let them take all the credit. They are politicians or demented anyways. 4-16-2015 Puppet uses Ruby for modules. Didn't know that. That explains why our guy made it such a huge pain in the butt to add anything needed for MySQL. He should have told us --- projects were unnecessarily delayed if he would have explained the situation so get more resources. That. Still, his fault. His fault for not telling us that so he could keep control because then I would have helped --- maybe --- I converted from Perl to Python and never looked back, and Ruby is too much like Perl. Will still get Puppet certification, but for my computers I will have them running salt. So in order of certification: Puppet, MongoDB, PostgreSQL, Hadoop. Maybe Cassandra afterwards. During this time, using salt, DAD, graphite grafana, etc. 4-17-2015 I am beginning to realize how immature and crude our setup potentially was with puppet. My theory about centralized configuration might have been correct and it should have been done different. This puppet course is very good, clear, and east to understand. The trick is I watch a little everyday to it doesn't overload the brain. I am going to go with salt anyways, after getting certified by puppet, hopefully if not the end of this month, the end of next month. But, I will leave 3 mysql servers in puppet, and 3 hadoop/other systems in salt for my setup --- just so I am forced to deal with them and keep up to date. 4-20-2015 Living in the dark. Its amazing how much you can learn from getting certified. The certification is not the important part, the learning of various subjects for fun is. Get certified in stuff you like cause its easy to get side tracked. DAD + Mongo (with MMS, ops manager, and salt) and also MaxDB, all with graphite and grafana, and predictive resource growth, is the way to go. I am very impressed with MongoDB 3.0 and the direction its going. Its do funny, but it is so easy to get lost in a fast paced company, you can't see the forest for the trees. Especially with a narrow-minded, unable to program, political, crocodile smiling ex-boss who makes weird ass decisions. 4-22-2015 Immediate:
1. Mongodb 3.0 with ops manager --- use ops manager to create mongodb services.
2. Redis + Mongodb --- just setup.
3. MongoDB certification -- take test next week (study now)
4. Test mnongodb failures from 2.2. Adding server, removing, etc.
5. Test document level transactions. Create a way to undo stuff in a trasnsaction if ti fails.
6. Test a 3rd mongodb server that is all in ram. Compare queries to it and 2nd server. Maybe 4 servers: Master Secondary, Secondary arbiter, add a 4th all memory server that cannot become master (hidden I guess)? Write up an article on this.
7. Test push button upgrade in OPS that is doesn't disconnect or ruin trasnsactions. 

other: MAXdb in front of mariadb galera. Remove percona or just use other 3 servers.

The highest is to get one certification out of the way, mongodb 3.0. Need to get certifications out of the way so I can work on DAD, grafana, etc.

6-3-2015 Didn't know google has dns servers you can use. The ones for my provider at home seems to have slow DNS. Might use it. https://developers.google.com/speed/public-dns/docs/using

Free online classes and cheap exams: for stuff I like

I will be updating this as time goes on. Many classes or exams are not worth it. I am filtering it down to the technologies I am familiar with and I like the class or exam. To be honest there are TONS of youtube videos. Who knows if they are tons. And the best way to learn the guts of the technology is to watch the online videos from various conferences, do what they say, watch again, repeat. Then take exams. There is no point paying $10,000 for certification when you can study, take the exam, fail it, study again, pass it when the exams are generally about $200 a piece (sometimes multiple exams are required). This is not a problem if you REALLY want to take the time to learn it.

Relatively cheap exams.

• MySQL Certification
• From mysql, they aren't bad. https://www.mysql.com/certification/
•  http://www.mariadbcertification.org/index-3.html
• No idea that is going on here. 
• Python Certification
• Linux Certification
• lpi.org
• Hadoop
•  http://cloudera.com/content/cloudera/en/training/certification/ccah.html
• Redis: 
• NOT CHEAP. You have to take the course too. http://pivotal.io/training 
• Certificates of completing a class below. 
• mongo
• https://university.mongodb.com/exams/C100DBA/about
• bastards. They don't allow the exam on Linux. I have a laptop which I have to leave Windows for this purpose. Stupid. Oh well. I hope they make it doable on Linux. I had a mac laptop, but not now. Darn. Registered for the april 28th one. Time to study and stuff.
• TODO: finsh sing up and schedule exam for may 
•  Puppet 
• Pearson Vue http://www.pearsonvue.com/puppetlabs/
• TODO: Take scheduled exam
• PostregSQL
• TODO: You can take this cert. 
• http://www.enterprisedb.com/products-services-training/postgresql-certification

For free or cheap Classes / Tests

•  http://bigdatauniversity.com/
• Hadoop: http://bigdatauniversity.com/bdu-wp/bdu-course/hadoop-fundamentals-i-version-3/
• There are lots of hadoop training courses here. 
• Apparently not all are free, I think these are:  http://bigdatauniversity.com/wpcourses/?cat=124
•  Map R --- hadoop replacement
• https://www.mapr.com/services/mapr-academy/big-data-hadoop-online-training
• Hadoop
• Free tests for 180 days
•  https://university.cloudera.com/certification/CCAH-practice-tests
• A lot that are free, and cheap ones
• https://www.udemy.com/courses/search/?q=hadoop&view=grid 
• I am just going to list the ones I did or doing , but there are lots. This list will grow as I get done with one I start another.
• DONE Big Data and Hadoop Essentials (free)
• DONE Basic overview of Big Data Hadoop 
• This one was hard to follow.
•  Hadoop Administration - Hands on ($19 after coupon code from the free class or its $200 -- not worth it for $200).
• Linux
• LPI has links to free exams and online tutorials for some of their exams. I recommend everyone get at least LPI level 1. 
•  
•  Python --- todo 
• MongoDB -- from the mongodb website
•  M101P: MongoDB for Developers
•  M102: MongoDB for DBAs
•  M202: MongoDB Advanced Deployment and Operations
•  UD032: Data Wrangling with MongoDB (Udacity)
•  https://www.udacity.com/course/
• There are others, but I am focusing on DBA stuff.  
•  
• Postgresql
•  http://www.enterprisedb.com/store/products/dba-training/01t50000001Nyi7AAC
• http://www.moderncourse.com/certification.php?cid=6
• http://www.enterprisedb.com/products-services-training/postgresql-certification

• Redis
• Certificate of completion. better than nothing:
• DONE https://www.udemy.com/learn-redis-step-by-step/ 
• DONE https://www.udemy.com/learn-redis/ --- best of 3 classes
• DONE https://www.udemy.com/rapid-redis/
•  Pivotal --- VERY GOOD
• DONE https://pivotallms.biglms.com/application-framework?q=5310bc82ecdedfdbd5ac1370

TODO: Salt, apache, find some.



Tutorials

• Hadoop
• Youtube:  https://www.youtube.com/watch?v=1jMR4cHBwZE&noredirect=1
• Probably other youtube stuff.
• MySQL: Conference links HERE. 
• Python: Conference links HERE.
• Linux: Conference links HERE. 
• redis: Conference links HERE. 
•  

Initialization and List of things to do

Original start date: 3-31-2015 - and I will be updating this frequently. When I get finished projects, they will be linked here I hope. 

I started this page for two reasons:

1. I want to investigate to do a bunch of technologies including advancing my DAD project. 
2. Prove to a person with a crocodile smile and to help with with middle management to get results done because middle management sometimes care more about image and perceived results than true results (luckily I have only come across this a few times --- I tend to work well with tech managers who don't have Napoleon complexes or political agendas):
1. grafana and graphite are much more powerful for automation than manually using grafana. Both are good, but manual is time consuming. 
2. Percona Cluster on Ubuntu 14.4 worked the first time compared to all the problems on Ubuntu 12.4. 
3. Installing grafana and graphite on ubuntu 14.4 was easy out of the box. I heard easy ways to install it on 12.4, but the way I did it from the instructions I found was hard.

Stuff started and done in 3-2015 for AWS project. Some stuff took time because I had to wait on stuff.

• Setup 3 ubuntu servers in AWS. There was actually a lot of work to get this right. 
• Reserved in instances. 
• Setup security. Ports between members in group.
• nagios and plugins
• pt-schema and other percona tools
• Mongo setup with 2 repica sets, config servers and mongos
• Apache with virtual domains
• FreeDNS for AWS server hosting apache.
• freeDNS, use their correct servers. 
• tripwire and fail2ban
• setup percona cluster with rsync
• Not enough memory for innodbackup -- will do at home.  
• Setup 3 laptops at home with LinuxMint.  

Stuff to do:

• Setup 4th laptop so I have 3 working laptops and they should be able to run my install scripts for the AWS project easily. 
• Start AWS project with install steps and scripts. 
• Start ramsey project. 
• Python
•  Redo the Python to Python standards.
• unit testing 
• osticket
• git
• 2 centos machines, maybe 3. 
• DAD
• MongoDB 3.0 and MMS and ops manager, salt for automation, graphite and grafana
• use yaml format
• Make modules semi-indpendent so you can load just parts. 
• into git
• get free space. Its gpl2 or gpl3, haven't decided. 
• backup script
• Backup binlogs
• Backup cluster
• Backup mysql 
• mysqldump
• snapshot via lvm
• inndb backup
• cluster backup
• Archive
• Into MyISAM merge tables. 
• Into Innodb tables
• Correct replication checks
• Slow log checks
• Slow log analysis feeder and grabber on server side
• feeding graphite
• local monitor/collector software on each server
• Include slow logs
• Include os status
• Include ability to throttle queries
• Include for nagios checks. A summary check and then details if something is wrong.  
• include any database
• integration with other monitoring software
• mysql
• mongo
•  Projections (supposed DBA didn't believe in these, lol)
• cpu
• memory
• diskspace
• load
• History
• Slow log reports
• General log reports
• server comparison 
• manual with one more servers
• Assign a server to a server. 
• Note allowed differences
• List which variables to look at. 
• schema comparison
• Assign server to a server to compare to in an admin page, by service_id with no drop downs. Print service_id next to name. 
• Admin page to add servers. 
• Add Mongo, Redis, and others to main page. 
• Add ability to:
• use short names
• red/yellow to highlight potential problems
• use replication names
• ignore querying data from mysql server for diskspace, etc.  
• admin page to easily add services
• Group page ( I think its there)
•  Shard analysis or servers group analysis
• Problem Page
• Replication
• settings
• Mysql
• OS
• slow logs
• diskspace
• backups
• others -- you know what I mean 
• my.cnf check
• Add mysql or mongo or other db as datastore. 
• In theory, database doesn't have to be networked. 
• Just need to store int, varchar, string, blob, dates (which could be integers).
• Script to setup dad tables and interface for it. Mylsq, Mongo, sqllite, postgresql, and others should work. 
• Put redis in front of it. 
• mySQL
• Replication
• MariaDB MaxScale
• puppet
• Takinbg exam at end of month. 
• Integrate it to manage the other servers.  
• Integrate when changes are made, or when noop is done it sends a report back to a server.  
• Setup the server so it checks out the commands and executed them. Masterless. Then the master's only duty is the puppet config on the remote host.  
• Perhaps have a database and store variables per machine so that when scripts run, it doesn't have to rely on the hostname to figure things out. You could have a database and web server in the same setting. The database would hold the information. Perhaps have it hold information which contain lists, etc.  
• Can you put variables in facter from the database?
• hadoop
• cassandra
• graphite
• grafana
• Manual graphs
• Automatic graphs (maybe graphite only)
• anemometer replacement since its broken (in a scalability sense) and in Python. I already wrote this years ago in a cruder form. 
• Data leaks
• Schema not optimized
• Written in PHP --- very simple and I already wrote this in 2006 in python, and then again in 2013. But not the pretty interface. That will be a learning experience to write that in python. 
• use yaml for everything, except config files. 
• basic config files use the same one for mysql parsing. 
• rainguage replacement since its broken (in a scalability sense) and in Python and I wrote very similar thing in 2006. 
• schema not optimized
• dangerous settings
• Needs to auto scale back
• needs default settings to not be dangerous
• Dangerous settings only turned on when asked
• Not written in Python. I already wrote this in Python years ago in 2006. 
• https login with cookie (if needed)
• MHA
• MySQL replication
• Percona Cluster
• Make scripts for each other them.
• Look for other MHA replacements
• Redis
• DONE 1st class : udemy
• DONE 2nd class : udemy
• DONE 3rd class : udemy -- do only the "redis" class
• DONE: pivotal cert on redis
• Map queries and schema to redis automatically, if the schema/query satisfy properties. 
• Validates tables and keeps it recorded somewhere.
• Semi complex queries to be mapped, which include conditions in the where clause that are not primary ---- if where conditions are at least indexed. 
• Make independent python module in dad.   
• Slow queries
• Get RedisLabs account
• integrate it with DAD 
•  hyperloglog, pub/sub, scripting
• Memacache
• Compare and run test to compare to redis
• VoltDB 
• Mongo script testing and work.
• Script to test connections when adding/removing node
• When replica set goes out.  Should be a positive result where it is detected.
• mongos goes out.  Shoud fail.
•  This should not fail (some did before 3.0):
• Adding a node
• removing a node
• Removing a replica set (that is drained)
• Adding a replica set
• Script should connect to everything in the replica set and do reads and writes. 
• Setup Mongo with redis for reads and writes. Possible? 
• Ops Manager, graphite, grfana, and DAD
• Chroot --- previous howtos would help here.
• Apache
• Mongo
• MySQL -- might be more of a pain than its worth.
• VoltDB
• Hadoop
• Cassandra
• Redis
• SSH login  
• postgresql
• and monitoring tool
• sqllite
• and mnitoring tool
• Packet sniff
• to download all mysql commands
• to download all mongo commands
• other databases or stuff, apache? 
• Article on AWS
• Get servers
• Setup DNS
• Point to DNS servers
• Setup virtual domain
• Setup blog
• TODO: Setup ticketing system behind SSL
• TODO: Setup redis, DAD, mongo, mysql 
• AWS
• Buy cheap database RDS for 3 years. MySQL preferred.  
• Setup Systems at home and AWS
• Setup salt server on AWS. 
•  If possible, have it configure  home server as well or setup Salt at home.  
• Setup DNS at home and work, controlled by salt.
• Install the following by salt:
• All services must installed in the following way:
• Configuration for topology defined in SALT somehow. 
• Detect if already running. 
• If running:
• Try to reconnect. 
• Spit out error messages if couldn't connect. 
• If not running, detect can initialize
•  If we can intitialize, do so. 
• Connect to group, or make it the primary if none of the others are up. 
• If we made primary, report it somewhere. 
• Or spit out error.
• Mongo 3.0
• VoltDB
• Vertica
• Redis
• Percona -- replication on default port. 
• This can be tricky if automated. 
• Percona -- Galera Cluster on custom ports. 
•