debugging is better if you have logs

If you have been getting errors in your app, and you do not know what is wrong the first thing that hit your mind is "I NEED TO DEBUG THE APP" (usually). That is relatively easy if you are developing in your local environment where only you are accessing or you are doing a standalone app with a cool IDE like eclipse, intelliJ, Visual Studio, xcode or even netbeans; but what happens if your app is a client in a client/server environment.. Ouch, you maybe will have questions like WHO, WHERE, WHEN, WHAT... if you can answer those questions, for sure you will be able to answer the really important question (WHY), because knowing why it is happening you will be able to solve the problem (code, code, code).

To this point I have not said anything new (I hope so), this is the important part:

If you keep track of everything that is happening you will be able to look for the answer to WHO, WHERE, WHEN and WHAT, and keeping that info is what we usually call LOG. Even if you are doing a mobile app, you need to log what is happening, all we know that logs consume space but you can keep just a window (last 5 hours, last day, last 2 days, etc), the size of that window is up to you.

So, now you have logs, SWEET... but wait, if you have a web platform you will have something like this:

• webserver log (apache, iis, nginx, etc)
• scripting log (ruby, php, etc)
• application log (the one that you are creating with your logical structure)

Ouch again, three sources... and I hope you have many clients connecting so you need many webservers... Oops, now i do not know how to consolidate that.. or worse, how do I read that... :(

Do not worry, this is what I really want to share with you today:

There are solutions for logging, but the one I like most is logstash, basically with logstash you can run a process in any machine that is reading whatever is being written to one or more logs, processing, parsing and sending it to another instances where it will be stored, sorted and make it SEARCHABLE, such a beautiful word, so you can get all the logs processed and available in one place, I LIKE IT!! now with this tool you could be doing a better debugging using your logs.

Basically it works like this:

1. your app/server/whatever generates logs locally (inexpensive, fast, easy)
2. you have a process reading those logs (it is called SHIPPER) and sending to the search engine, you can use regular expressions to parse each log entry in the client (AWESOME), the timestamp is courtesy of logstash
3. you can setup a web interface (two or three availables), I suggest kibana that is really good.
4. search, search, analyze, SOLVE.

It is easy, isn't it?

What now?, well watch this presentation of the creator of logstash at the puppetconf, it is for falling in love with the tool. and then start reading the docs for getting started.

A couple of suggestion:

• you can test it in your local machine without problem.
• be careful with the memory usage, you will find different shippers that could help you to use less memory reducing functionality. Keep an eye on that specially if you are using Java.
• if you are planning a live deploy, please take in count the scalability, I would like all of you having millions of users generating logs.

Two final comments, 

1. there is no tool that replace you ability to analyze and THINK, take your time to think in whatever you are watching. Making mistakes is ok, what is not ok is to make mistake because "I do not think that.." 
2. I have not tried to use any shipper in a mobile device, if you found anything please share it, because that is one of the advantages of newrelic.

Enjoy it!