get a backup before it's too late

During the last years we have learned about taking care of our code doing backups in zip files, then we evolved and we start using SVN to control changes and having a backup; today, we are majorly using GIT for version control and having the backups in a system providing much more functionalities around the main concept of a BACKUP.

That is really cool, if you are doing version control then you are doing something good. But, if for example you are running a website; well, you will have you a backup of your code in a repository and if you are using any Content Delivery Network (CDN) then you will have a backup of the files that could be sharing, but what happen with your database?

Well, obviously you will need a backup of your database, but usually as soon as you start growing then your databases start being bigger and bigger, really quickly you will be unable to do a dump of your DB (damn you scalability). Ok,  now I would like to make a list of some considerations prior to deploy a backup system:

1. we should have downtime zero!! basically, the system operation should not be affected for any backup.
2. Optimal use of the storage resources, if the database grows 100% weekly then you will face a issue having tons of megabytes of storage for backups. Having incremental backups sounds good.
3. Backups should be done often, it is useless to have backups every month if the business change every day or every hour.
4. Good use of the processing time, if you need to do 4 backups a day, and each backup takes 12 hours to be done then something is really wrong.
5. Having the optimal system is irrelevant if you are not able to restore from a backup
6. If you have incremental backups then you could have the option to restore to a point in time (maybe).

Taking in count those points, I would like to give you to options based on the systems that I am managing at this time.

First, I love using mongodb, the creator of mongodb have an amazing set of tools available for everyone, those tools are called MMS, 10gen started offering a backup service really good, I have tested it and I have had really good results, it is 100% recommendable.

Second, I am also a MySQL user, I have systems running in master/slave mode and I even have cyclic replication between two masters (this sounds really nice), so it is complex and it is BIG, some time ago i was testing a tool from Percona called xtrabackup, I started testing that tool under a disaster recovery scenario, after that, i stop using it; not because of the tool, it was just lazyness but it is time to start testing again. This tools is amazing, it is like the backup system of the heaven.

The price of the backups is not related to the backup itself, it is related to the price that you will pay if you do not have a backup.

I have given you many places to look and read for, if you have any other option please let me know. And enjoy having the inner peace because you have backups!

orchestration is one step for growing

When you are taking care of a platform, sometimes you need to do some tasks on a set of servers and everything will go like a charm if you have a few servers, but imagine that your platform grows from 10 servers to 200 servers, well, that could make your job a little bit awful.

Even if you have an image of the server, every time that you need a change you can create a new image but you will need to replace all the servers that are running or coordinate an update, obviously you could not do it manually, so you need any kind of orchestration. Orchestration refers to management and coordination of tasks in big systems.

At this point you could find many people suggesting tools, I would like to give an opinion that by the way is not final because I am still testing. As usual i will give you choices, so you could walk whatever YOU think is the right path.

Let's start with Puppet, this tool is awesome you can make almost a lot of everything, monitoring and orchestrating, but let's be honest, it sounds like too much, and it is.. probably, it will awesome but it will be like cooking an steak in a nuclear reactor. Anyway, you can give a chance to it.

Second tool, Chef, it is really amazing how can automate configuration, installation and keep track of configuration.. Actually, I am trying to fall in love with this but our relationship is complicated, you need to be very pragmatic, practical and organized. When you use a tool like that, you need to know when start doing in your own way, it is better to keep the recipes in the original way but any real life chef know that they can twist a recipe to make it taste like they want. 

My last option, do it your self!! For example, i have built a set of scripts in python+fabric that allow me to do many stuff against a set of servers in AWS, for example I can define common tasks, like clean of logs, and run it against all the webservers, or I can open a virtual console that give access to send custom commands to a set of servers even in different regions. Actually, i love fabric, i think that framework is AWESOME.

You need to test every option by yourself and take in count the size and characteristic of your business, there is no map of the treasure, there are clues and you have to build your own map.

Enjoy the ride!

the nightmare of shared passwords

Today, my title is clear, every time that someone joins the team of operations (call us devops, sysadmin, etc) we need to start sharing passwords for different services and in that point we start to listening the glove armed with razors scratching a pipe (just like Nightmare on Elm Street), and now I will scare you to death, imagine that after months giving passwords for services, one of the members of the team leave the company, OMG!! kill me now Mr. Krueger!!

Well, I have some Password Managers that could give us clues and help us, but keep in mind that i am not 100% comfortable with any of them, I would like something more custom adapted to my concepts of safety, simplicity and usability.

We have three choice:

1. Hosted services, and by hosted I am talking about a service that someone provides (here my spidey sense start to alert me about someone else having my passwords)
2. Standalone apps, well nobody have my passwords, I could share the password db but keeping the synchronization is really sucky
3. Mixed environment, now we are talking, this sounds more like a good choice.

Some of the tools available includes:

• Roboform
• SplashId
• CommonKey
• 1password
• KeePass or KeePassX

Some of them hosted like CommonKey, some standalone like KeepassX and some of them like 1password that can be setup to store the DB in Dropbox.

This is my wish list of functionalities:

• Privately hosted
• Clients for multiple platforms (mac, linux, windows, android, ios)
• Secure communication between clients and server
• Access control
• Easy synchronization
• Easy modification of information stored in the DB
• Secure storage of the information

At this point, my choice is a kind of hybrid solution manually built on top of Dropbox and using 1password as client, at the end the idea is to be able to revoke access or grant it as easy as using Dropbox. The bad part is that you need to pay for 1password and you depend on Dropbox.

As an extra point let me tell you that keeping the password in that way could give you an amazing option when someone leave the organization because you have a list of the password to change and a direct way to share the new ones.

I hope that this post give you more ideas than solutions and if you decide to solve this problem, count with me as developer and tester.