keeping an eye in the platform

Sometimes we feel like our platform is not doing well, maybe we see slowness in the service. Well the answer is lies in questions like:

• are you ready for growing??
• what is the performance of your platform??
• do you know where are the bottlenecks??

Well, in order to discover those answers, you must need an eye on your platform (maybe more than one), today it could be funny and simple, but especially very powerful because the amazing amount of tools available.

If you are using any cloud provider you will enjoy some basic metrics, but the power is in another place, i will give you some options and as usual you will have the task to choose whatever you like and makes you feel comfortable.

• nagios, basically a nice alerting system, mainly use snmp to get metrics from servers and show you the status per hosts/service (ok, warning, critical).
• cacti, it is just graphing, based in snmp you could monitor metrics.
• munin, helpful to collect data because it handle a custom client/server architecture (munin-client and munin-server), this is not a graphing tool, it is for monitoring.
• collectd, i am starting to feel love for this client/server option (like munin).
• graphite, nice graphing server with many frontends available.
• ganglia, data collecting system (client/server) with a really ugly frontend, but pretty fast.

This is a important point, YOU CAN COMBINE SOME OF THEM!!! Yes, you could use collectd+graphite+nagios. Anyway, as usual you should take care of scalability in two points: 1.- do not overload your platform and 2.- your monitoring platform must be ready to scale with you.

Please, monitor and enjoy understanding where are your bottlenecks and your performance opportunities. Remember that it is required collect and process data in order to have elements to analyze and make desicions

debugging is better if you have logs

If you have been getting errors in your app, and you do not know what is wrong the first thing that hit your mind is "I NEED TO DEBUG THE APP" (usually). That is relatively easy if you are developing in your local environment where only you are accessing or you are doing a standalone app with a cool IDE like eclipse, intelliJ, Visual Studio, xcode or even netbeans; but what happens if your app is a client in a client/server environment.. Ouch, you maybe will have questions like WHO, WHERE, WHEN, WHAT... if you can answer those questions, for sure you will be able to answer the really important question (WHY), because knowing why it is happening you will be able to solve the problem (code, code, code).

To this point I have not said anything new (I hope so), this is the important part:

If you keep track of everything that is happening you will be able to look for the answer to WHO, WHERE, WHEN and WHAT, and keeping that info is what we usually call LOG. Even if you are doing a mobile app, you need to log what is happening, all we know that logs consume space but you can keep just a window (last 5 hours, last day, last 2 days, etc), the size of that window is up to you.

So, now you have logs, SWEET... but wait, if you have a web platform you will have something like this:

• webserver log (apache, iis, nginx, etc)
• scripting log (ruby, php, etc)
• application log (the one that you are creating with your logical structure)

Ouch again, three sources... and I hope you have many clients connecting so you need many webservers... Oops, now i do not know how to consolidate that.. or worse, how do I read that... :(

Do not worry, this is what I really want to share with you today:

There are solutions for logging, but the one I like most is logstash, basically with logstash you can run a process in any machine that is reading whatever is being written to one or more logs, processing, parsing and sending it to another instances where it will be stored, sorted and make it SEARCHABLE, such a beautiful word, so you can get all the logs processed and available in one place, I LIKE IT!! now with this tool you could be doing a better debugging using your logs.

Basically it works like this:

1. your app/server/whatever generates logs locally (inexpensive, fast, easy)
2. you have a process reading those logs (it is called SHIPPER) and sending to the search engine, you can use regular expressions to parse each log entry in the client (AWESOME), the timestamp is courtesy of logstash
3. you can setup a web interface (two or three availables), I suggest kibana that is really good.
4. search, search, analyze, SOLVE.

It is easy, isn't it?

What now?, well watch this presentation of the creator of logstash at the puppetconf, it is for falling in love with the tool. and then start reading the docs for getting started.

A couple of suggestion:

• you can test it in your local machine without problem.
• be careful with the memory usage, you will find different shippers that could help you to use less memory reducing functionality. Keep an eye on that specially if you are using Java.
• if you are planning a live deploy, please take in count the scalability, I would like all of you having millions of users generating logs.

Two final comments, 

1. there is no tool that replace you ability to analyze and THINK, take your time to think in whatever you are watching. Making mistakes is ok, what is not ok is to make mistake because "I do not think that.." 
2. I have not tried to use any shipper in a mobile device, if you found anything please share it, because that is one of the advantages of newrelic.

Enjoy it!