Initialization and List of things to do

Original start date: 3-31-2015 - and I will be updating this frequently. When I get finished projects, they will be linked here I hope. 

I started this page for two reasons:

1. I want to investigate to do a bunch of technologies including advancing my DAD project. 
2. Prove to a person with a crocodile smile and to help with with middle management to get results done because middle management sometimes care more about image and perceived results than true results (luckily I have only come across this a few times --- I tend to work well with tech managers who don't have Napoleon complexes or political agendas):
1. grafana and graphite are much more powerful for automation than manually using grafana. Both are good, but manual is time consuming. 
2. Percona Cluster on Ubuntu 14.4 worked the first time compared to all the problems on Ubuntu 12.4. 
3. Installing grafana and graphite on ubuntu 14.4 was easy out of the box. I heard easy ways to install it on 12.4, but the way I did it from the instructions I found was hard.

Stuff started and done in 3-2015 for AWS project. Some stuff took time because I had to wait on stuff.

• Setup 3 ubuntu servers in AWS. There was actually a lot of work to get this right. 
• Reserved in instances. 
• Setup security. Ports between members in group.
• nagios and plugins
• pt-schema and other percona tools
• Mongo setup with 2 repica sets, config servers and mongos
• Apache with virtual domains
• FreeDNS for AWS server hosting apache.
• freeDNS, use their correct servers. 
• tripwire and fail2ban
• setup percona cluster with rsync
• Not enough memory for innodbackup -- will do at home.  
• Setup 3 laptops at home with LinuxMint.  

Stuff to do:

• Setup 4th laptop so I have 3 working laptops and they should be able to run my install scripts for the AWS project easily. 
• Start AWS project with install steps and scripts. 
• Start ramsey project. 
• Python
•  Redo the Python to Python standards.
• unit testing 
• osticket
• git
• 2 centos machines, maybe 3. 
• DAD
• MongoDB 3.0 and MMS and ops manager, salt for automation, graphite and grafana
• use yaml format
• Make modules semi-indpendent so you can load just parts. 
• into git
• get free space. Its gpl2 or gpl3, haven't decided. 
• backup script
• Backup binlogs
• Backup cluster
• Backup mysql 
• mysqldump
• snapshot via lvm
• inndb backup
• cluster backup
• Archive
• Into MyISAM merge tables. 
• Into Innodb tables
• Correct replication checks
• Slow log checks
• Slow log analysis feeder and grabber on server side
• feeding graphite
• local monitor/collector software on each server
• Include slow logs
• Include os status
• Include ability to throttle queries
• Include for nagios checks. A summary check and then details if something is wrong.  
• include any database
• integration with other monitoring software
• mysql
• mongo
•  Projections (supposed DBA didn't believe in these, lol)
• cpu
• memory
• diskspace
• load
• History
• Slow log reports
• General log reports
• server comparison 
• manual with one more servers
• Assign a server to a server. 
• Note allowed differences
• List which variables to look at. 
• schema comparison
• Assign server to a server to compare to in an admin page, by service_id with no drop downs. Print service_id next to name. 
• Admin page to add servers. 
• Add Mongo, Redis, and others to main page. 
• Add ability to:
• use short names
• red/yellow to highlight potential problems
• use replication names
• ignore querying data from mysql server for diskspace, etc.  
• admin page to easily add services
• Group page ( I think its there)
•  Shard analysis or servers group analysis
• Problem Page
• Replication
• settings
• Mysql
• OS
• slow logs
• diskspace
• backups
• others -- you know what I mean 
• my.cnf check
• Add mysql or mongo or other db as datastore. 
• In theory, database doesn't have to be networked. 
• Just need to store int, varchar, string, blob, dates (which could be integers).
• Script to setup dad tables and interface for it. Mylsq, Mongo, sqllite, postgresql, and others should work. 
• Put redis in front of it. 
• mySQL
• Replication
• MariaDB MaxScale
• puppet
• Takinbg exam at end of month. 
• Integrate it to manage the other servers.  
• Integrate when changes are made, or when noop is done it sends a report back to a server.  
• Setup the server so it checks out the commands and executed them. Masterless. Then the master's only duty is the puppet config on the remote host.  
• Perhaps have a database and store variables per machine so that when scripts run, it doesn't have to rely on the hostname to figure things out. You could have a database and web server in the same setting. The database would hold the information. Perhaps have it hold information which contain lists, etc.  
• Can you put variables in facter from the database?
• hadoop
• cassandra
• graphite
• grafana
• Manual graphs
• Automatic graphs (maybe graphite only)
• anemometer replacement since its broken (in a scalability sense) and in Python. I already wrote this years ago in a cruder form. 
• Data leaks
• Schema not optimized
• Written in PHP --- very simple and I already wrote this in 2006 in python, and then again in 2013. But not the pretty interface. That will be a learning experience to write that in python. 
• use yaml for everything, except config files. 
• basic config files use the same one for mysql parsing. 
• rainguage replacement since its broken (in a scalability sense) and in Python and I wrote very similar thing in 2006. 
• schema not optimized
• dangerous settings
• Needs to auto scale back
• needs default settings to not be dangerous
• Dangerous settings only turned on when asked
• Not written in Python. I already wrote this in Python years ago in 2006. 
• https login with cookie (if needed)
• MHA
• MySQL replication
• Percona Cluster
• Make scripts for each other them.
• Look for other MHA replacements
• Redis
• DONE 1st class : udemy
• DONE 2nd class : udemy
• DONE 3rd class : udemy -- do only the "redis" class
• DONE: pivotal cert on redis
• Map queries and schema to redis automatically, if the schema/query satisfy properties. 
• Validates tables and keeps it recorded somewhere.
• Semi complex queries to be mapped, which include conditions in the where clause that are not primary ---- if where conditions are at least indexed. 
• Make independent python module in dad.   
• Slow queries
• Get RedisLabs account
• integrate it with DAD 
•  hyperloglog, pub/sub, scripting
• Memacache
• Compare and run test to compare to redis
• VoltDB 
• Mongo script testing and work.
• Script to test connections when adding/removing node
• When replica set goes out.  Should be a positive result where it is detected.
• mongos goes out.  Shoud fail.
•  This should not fail (some did before 3.0):
• Adding a node
• removing a node
• Removing a replica set (that is drained)
• Adding a replica set
• Script should connect to everything in the replica set and do reads and writes. 
• Setup Mongo with redis for reads and writes. Possible? 
• Ops Manager, graphite, grfana, and DAD
• Chroot --- previous howtos would help here.
• Apache
• Mongo
• MySQL -- might be more of a pain than its worth.
• VoltDB
• Hadoop
• Cassandra
• Redis
• SSH login  
• postgresql
• and monitoring tool
• sqllite
• and mnitoring tool
• Packet sniff
• to download all mysql commands
• to download all mongo commands
• other databases or stuff, apache? 
• Article on AWS
• Get servers
• Setup DNS
• Point to DNS servers
• Setup virtual domain
• Setup blog
• TODO: Setup ticketing system behind SSL
• TODO: Setup redis, DAD, mongo, mysql 
• AWS
• Buy cheap database RDS for 3 years. MySQL preferred.  
• Setup Systems at home and AWS
• Setup salt server on AWS. 
•  If possible, have it configure  home server as well or setup Salt at home.  
• Setup DNS at home and work, controlled by salt.
• Install the following by salt:
• All services must installed in the following way:
• Configuration for topology defined in SALT somehow. 
• Detect if already running. 
• If running:
• Try to reconnect. 
• Spit out error messages if couldn't connect. 
• If not running, detect can initialize
•  If we can intitialize, do so. 
• Connect to group, or make it the primary if none of the others are up. 
• If we made primary, report it somewhere. 
• Or spit out error.
• Mongo 3.0
• VoltDB
• Vertica
• Redis
• Percona -- replication on default port. 
• This can be tricky if automated. 
• Percona -- Galera Cluster on custom ports. 
•