MEN Projects Blog: August 2015

Thursday, August 27, 2015

8-27-2015 Lazy and education (status: done)

I have been lazy at doing stuff because I was reading a lot. I also condenses my blogs that had no useful information. For this blog, I will have something useful for the Hip Hop Chess Federation indirectly. And I have decided to do things manually for laptops at home only, but before they get pushed to production they must be in salt or dad or both. I need to learn about these technologies. More important than salt and dad, which is a lot of work. Stuff will be setup as how I would envision salt or dad doing it ultimately.

http://www.menprojects.com/projects/hhcf/HHCF.py

Update: rather than new blog, also made a webpage to upload file and have it processed. Next, I am going to try to make a very basic Android based application written in Python.

Sunday, August 23, 2015

8-23-2015: Adding more laptops at home for 4 computer clusters (DONE)

I have 3 laptops at home. One was for my tv and one was an old crappy laptops. I don't want to use them for real work. So I got 2 more laptops and hooked them up to my monitor that has 2 hdmi ports. Now I have a cluster. Here is the steps I used to install.

Install Linux

Download the last Linux Mint and burn it.
Put cdrom in external cdrm drive first. Bios might not detect this.
Reboot into bios and make the cdrom the first bootable device.
Remove secure boot in BIOS.
Reboot
When it comes to partitions:

100 megs for EDI parition
100 meg for swap, I know its an overkill.
10000 megs for /BACKUP
rest for /

Post Install STEP 1

In the control panel, for power management, set it to do nothing when the laptop is closed and turn off any power management after a time.
Run the script I have below.
In screensaver, make bsod the default and no locking when screensaver is activated. Who cares. This is for testing.

Post Install Step 2 .

Add the two systems to DNS
Install DNS on the two systems
Exchange SSH keys.
Install SALT on those two systems.

Then for the future, I have a cluster of 3 servers for mono, vertica, voltdb, MySQL and Galera, and maybe MaxScale.

DNS entry.


ns3   IN A 192.168.1.30
mark4 IN CNAME ns3
ns4   IN A 192.168.1.179
mark5 IN CNAME ns4

Install script



echo "deb http://us.archive.ubuntu.com/ubuntu/ trusty multiverse" >> /etc/apt/sources.list
echo "deb-src http://us.archive.ubuntu.com/ubuntu/ trusty multiverse " >> /etc/apt/sources.list
echo "deb http://us.archive.ubuntu.com/ubuntu/ trusty-updates multiverse " >> /etc/apt/sources.list
echo "deb-src http://us.archive.ubuntu.com/ubuntu/ trusty-updates multiverse " >> /etc/apt/sources.list

apt-get update

   # Unfortunately, you will have to answer questions for some of these. 
apt-get -y install emacs  tripwire nmap lynx curl vnstat ttf-mscorefonts-installer
apt-get -y xscreensaver-data-extra xscreensaver-gl-extra xscreensaver-screensaver-bsod
apt-get -y  ncftp openssh-server openssh-client

cd /usr/share/applications/screensavers
find . -name '*.desktop' | xargs sed -i 's/OnlyShowIn=GNOME;/OnlyShowIn=GNOME;MATE;/'

wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
sudo sh -c 'echo "deb http://dl.google.com/linux/talkplugin/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
sudo apt-get install google-talkplugin 

  # It seems to cause problems with apt-get update
rm -f /etc/apt/sources.list.d/google.list
   
   # This might make the computer seem to stall, but just wait. 
apt-get -y install xfonts-*

apt-get update
apt-get -y upgrade

Saturday, August 8, 2015

8-8-2015 Install Mongo 3.0 from start to finish (status : just started )

The purpose of this task is to install Mongo as a complicated sharding cluster or single server (with replicas). I did Mongo 2 for over a year, and I am getting certified in Mongo 3.0 for fun, and I like the direction Mongo is going, so I think it will be here for a while.

A big issue is the initial installation, the setup, and then graphing later. I need to install graphite at least before I am done with this blog entry.

Install DAD modules, develop them.
Integrate with Salt

Make a module n the client side which

Checks if there is already a mongo installed, if there is enough diskspace, validates we can install.
Install mongos, mongo config, or mongo data. Or standalone is replica set, or standalone.

We need a config.

The module will probably use existing salt modules.
Return the data as yaml or multi-level dictionaries.

DAD can detect if this server can execute stuff on other server via salt.

Or via sudo.
Detect is client side module is installed.

Once integrated with SALT or sudo,

Choose naming convention for mongo installations because we might have more than one mongo on it. Allow crazy configurations (which can be useful or test or QA boxes).
Connect and execute detection scripts on all targeted hosts.
Then execute the install scripts.
Execute test scripts.
Run as a background job and report status.

Make a DAD interface for monitoring mongo in its basic form.

Include basic graphite stuff.

Setting up dad. The initial setup is crude. I will need to make a package that installs this. I just

 ln -s /usr/lib/python2.7/dad  /SOME/DIR/work/git/dad

instead of making a package. Then a simple dad is just "import dad" in a python script. I am going to have to make packages for the target servers.

I am going to use MariaDB, because later I want to use the MariaDB Max stuff. https://downloads.mariadb.org/mariadb/repositories/#mirror=jmu&distro=Ubuntu&distro_release=precise--ubuntu_precise&version=10.0


sudo apt-get install python-software-properties
sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xcbcb082a1bb943db
sudo add-apt-repository 'deb http://mirror.jmu.edu/pub/mariadb/repo/10.0/ubuntu precise main

sudo apt-get update
sudo apt-get install mariadb-server

# changed in etc/mysql/my.cnf the homedir to /data/mysql_00
mkdir -p /data/mysql_00
chown -R mysql /data/mysql_00
mv /var/lib/mysql/mysql /data/mysql_00/mysql 

service mysql stop
service mysql start

TODO: my.cnf settings, connection pooling, buffers, connections, etc.

Integratig with SALt.....

Friday, August 7, 2015

8-7-2015 Setting up your own staging to prod environment at home

Well, not exactly at home. I want prod to be independent of staging and professional quality. This document will be edited as time goes on.

Goal is to develop at home and push to prod and it should still work.
Perhaps I should have a staging environment? No, my stuff is not that important. Nobody is going to die. For money making website, yes. Its the same but just another step on the chain. If you don't, and you make money, you're CEO is a bit weird to say the least. I won't work there or demand it gets changed. Believe or not, many companies don't.

2nd goal is to show people that you can get stuff to work. It was so frustrating to deal with outdated versions of software that required 10 times more work or just didn't work. In the past, this was used by a jerk to corner me to make himself look better. I remember a co worker saying once that guy is past his 3 month probation, things will change. Yes, it did lol. Total liar and I hired the guy. Never trust anyone until after their probation, and even then, you never know. Protect yourself by getting ahead of the trends.

This stuff is done:

Setup domain name record with free service. Since it is free I might as well give them some kudos.

www.namecheap.com

Setup Google's applications for menprojects.com to handle email, and other stuff.

Had to point DNS records to google. Did it months ago.

Use blogger.com for blogs.
Buy 3 laptops at home for creating 3 server clusters for mysql, mongo, etc. I got some crappy ones for less than 300 brand new. 4 gig ram, 500 gig diskspace, hmdi, and standard stuff.
Buy TV with hdmi for laptop monitor. Don't bother with actual computer monitors. This TV is for computer use only.
Setup 7 AWS servers. one main and two sets of 3 clusters. Setup permissions, keys, etc.

A micro computer for 3 years was over $100. Sold. Not much diskspace or power, but that's okay.

Setup DNS and home and DNS at AWS for inside use only.
Setup GIT on local and AWS.
Install Percona Cluster (not mysql cluster) it uses galera.

At the time of the last edit, this stuff is to do first at home then on AWS:

Automate the script that creates the package for DAD. And other tools.
Install SALT and get installations scripts for all software.
Add graphing and monitoring scripts and reports.

Integrate with SALT.

Mongo 3.0 certification.
Install and automate VoltB and Vertica.
Install MariaDB with Maxscale and make it automated.
Program DAD for VoltDB, Vertica, and MariaDB with or without MaxScale.
Install Redis and VoltDB as cache layers for apache. Different scripts can point to either I guess.
R -- a replacement for SAS and built into Vertica.

Now for the technologies:

SALT : Because I a python guy and I hate writing modules in the other ones. I have been stuck waiting on people to write modules --- one who actually worked against me. I want to be able to write my modules fast.
GIT: You might say mercurial, but there is no python I need written for source control and GIT has a bigger following than mercurial will ever have. Mercurial comes in 2nd for me.
Vertica, VoltDB, and MariaDB over Hadoop and Cassandra. I know MariaDB isn't a data warehouse, unless you want it to be. Hadoop and Cassandra have the market place but I am looking 5 years down the road. Both Hadoop and Cassandra are just not philosophical there with me.
Redis and VoltDB as cache layer. Both are good.
MariaDB and MaxScale : Percona is nice. But MariaDB is more in line with my philosophy. MaxScale takes away some of the reason to use NoSQL.
Percona or MariaDB Cluster: Galera is awesome and I like it a lot. I will switch from Percona to MariaDB cluster. But kudos to Percona for pushing it. I hope Percona takes the MariaDB Maxscale and the two companies continue to push each other and be friends.
Apache has my web service choice, but others are good. This is not written in stone for me. I have always been an apache guy.
LinuxMint or the Ubuntu knockoff. RedHat is all corporate and I disagree with the fundamentals of their company. Ubuntu is my 2nd choice but I hate Unity and I hate for them jumping on the bandwagon to turn your computer into a phone. Buttheads. Sadly for AWS its ubuntu, but I use LinuxMint at home.
Python and Bash. Don't need to say anything. If I do, you're not my audience.
Graphite and Graphana. The one good thing that came out of backstabbers and foolish people I had to work with (2 bad ones out of many that were great people). Yes, those technologies for graphing are great but they never understood its full power. It is hard to deal with people who can't see the forest for the trees and are tyrants.
Nagios. I am so used to it.
R -- a replacement for SAS. I have told it is built into Vertica nicely and is pythonish. Makes it a no brainer.

I'll add more as I go on.

Sunday, August 2, 2015

8-2-2015 GIT (status: done)

After taking 2 months off for personal reasons, back in the game.

This is going to focus on GIT.

Develop a home git environment.

initialize project
Clone the project locally
Make available for remote acess
Do a test run remotely.

Replicate it to AWS via SSH,
Integrate with SALT somehow.

This is going to be an ongoing doc. Just started it.
Links: