being "agile" does not means being "scatterbrained"

I am not (and I do not attempt to be) a Scrum Master or any kind of agile guru, for that I have friends like Jesús Enrique Méndez (http://agileinpills.wordpress.com and his initiative http://www.agileopenspace.org), Gustavo Bonalde (http://gbonalde.blogspot.com/) or Carlos Gabriel (https://twitter.com/CarlosGabriel_). Anyway I am a guy that has spent a lot of time researching, studying, watching in silence how people do the stuff and secretly making some experiments; so I have developed my own opinions.

First, I believe that we need to be really agiles in our process, any kind of waste should be minimized, so why should we keep doing things in the most costly way (time related)?

Second, we need to be productive, that means that expending a year to develop a new functionality is unacceptable, even if you are developing an Operating System the process should not be that long.

Third, your business model or your economical maturity level could give you an advantage over other companies but that does not mean that you are allowed to make mistakes and if you are a small company or startup you will need to use properly any piece of resource that you can get.

Thinking in those three ideas, I usually see how people start being "agile" by ignoring any kind of design or worst, they start being "agile" and they stop thinking in the problem or the possible solutions and they start doing "anything" to solve the problem. I do not want to talk about efficiency or any related aspect to that, but let's take a time to understand the situation, let's draw something, let's get to the reasons of our problem and then we can solve it looking an good/optimal solution. If you want to be "agile" you can start by using the best simple solution instead of the most complicated solution. At this point my two advices are:

remember that thinking and designing are not sins.

keep it simple, do not oversize the problem.

I also see how people try to reduce the development cycle by doing something and sending it to the production environment as soon as they finish, then everybody gets crazy because something is falling; well, when I was a young developer, I learned about a technique (almost magical) to minimize troubles, it is called TESTING, we can not be in a rush and forget about tests, we HAVE to test our codes, obviously something can change and something could fail, but that HAS to be the exception and not the rule. I have saw people that is unable to send a functionality to production without having tons of errors and then breaking everything. Today, we have tools to automate our test, we can use jenkins or any other but we have to use it.

clumsy is not agile.

lazy is not agile.

testing is important for everybody, and we know that things can fail but it should be the exceptions.  

you can be agile testing, that does not mean NO TESTING.

Finally, I have saw people that as soon as they start getting error they start blaming any single API, third party tools or even the astrological configuration of mercury. People, let's face it, usually when something goes wrong we are the responsible, if we work in an orderly way, if we document our code, splitting the problems in smaller problems could make us agiles replacing broken pieces, fixing bugs or improving functionalities.

Well my friends, if you are trying to become an agile developer then you have my bless, just do it, but do it in the right way, do not become a "scatterbrained" trying to speed up developments.

providers, providers, providers.. do not kill me, help me!

when you have a business, it is important to be careful with every aspect, but there are two that are really important to me:

1. People that you are hiring
2. Services providers

I use that order because i believe that it is the correct, at the end an organization (company, enterprise or however you want to call it) is a structure with people giving meaning to that structure and the most important element IS THE PEOPLE. 

the second point is related to service providers, that is because many of the business we are building today rely on services that we are hiring. History teach us that some providers could fail, for example having a blog in wordpress.com, maybe they could suffer a outage and take our blog down, that is bad but imagine having your core business app hosted in a place that goes down (in that point you could start crying). If you do not want to cry, what you will need are really reliable providers.

talking about providers, we expect help from them, having a cleaning company that leaves more trash than the one that is collected is not a good deal. Let's face it, we need more help than problems, that my friends is a good indicator of the quality of the provider (problems over help, it have to be almost zero).

Being very honest, problems happen, and we will have to deal with them, but what we do not need (want) is the same problems over and over.

I will give you two examples:

• AWS, they are the first provider of cloud computing services, they have (in my personal opinion) one of the most amazing stacks of cloud oriented services. Obviously, they will charge you for breathing if they have a chance. They have premium support that could be something really expensive, for my it could be like USD 2000 monthly (OUCH). If you are not a member of the premium group then you are screwed!! You can open a ticket and pray for a solution in less 2-3 days or post something in the AWS forum and again pray for a solution. For example, they showed me in the AWS Management Console that I have a pending "EVENT", it was an Instance that was running in a "DEGRADED HARDWARE", one of the AWS solutions is Stop/Start the instance, well, i did that... the instance was stuck in stopping state for 5 hours. Thankfully it was not a really important server, otherwise that would mean a downtime of 5 hours, obviously, i reported the issue after 5 minutes but the solution arrived a "little bit" later. 
• Time Warner Cable (TWC), do i need to say more?? Anyway, I will do it.. Imagine you have an Internet based product, so you will need a reliable connection, well, TWC offers connections of 30Mbps/3Mbps, 30Mbps/5Mbps, 50Mbps/3Mbps and 50Mbps/5Mbps (something like that). Usually, if you have the 50% of that, you are really lucky, and the trace routes could vary from 20ms per hop to 1200ms per hop in a five minutes period. By the way, if you call to support, they will give you reasons like "it is related to solar flares (really, they told me that once and my connection is coaxial)", "it could be a problem with the AMPLIFIERS (OMG, do i have to plug a headphone to the router??, where are the speakers)". Whatever they say the final sentences is the same.. "It should be fixed in a while, wait for a couple of hours", in real life, that means something like "stop bothering us because we are busy playing solitaire".

Did those examples sound like reliable providers??

I can give you some personal keys:

• try to build your apps over disposable servers, so you could get over any kind of situation like the "DEGRADED HARDWARE" case. Volatile is key!
• If you could not make it volatile, make it easy to recover, maybe with amazon AMI or any kind of machine images, backups with fast recovery and fast deploys. Chef is awesome, but having a recipe that takes 1 hour to run is a really BAD option, speed it up.
• get redundancy, do not put all the eggs in one basket. For example, if you are in AWS, use multiple regions and availability zones in each region.
• if you could combine multiple providers, DO IT!!

my app is not growing, should i become Steve Jobs?

Today, when I was having lunch with a really good friend, one of our topics of conversation was about the process of getting more users and keeping them using the app. Let's face it, the only reason we can increase the valoration of our softwares is based on how many user we have and the retention. Here, I would like to quote “The Lean Startup”, Eric Reis talks about the importance of measuring as a part of your product development cycle. Seeing how your users react to and interact with new features and ideas should influence your decisions about those features and ideas.

That is a key, we need to measure things like:

• Daily Active Users (DAU)
• Monthly Active Users (MAU)
• New Users
• Retention

Those are really important values that show you if you are on the right direction, but knowing the retention do not give you the answer to what to do, lets suppose that the users are not coming back to your website, you will see a drop in the DAU, so do you know what to do? if you want to solve the DAU drop you will need to find out how to get your users back. So this is the answer to your problem:

you need that the user to likes your app

It is kind of obvious that increasing the DAU means to keep the user coming back as much as possible, for that I would like to share with you what I think are keys:

• having an idea do not makes you an expert
• never consider yourself a regular user, come on, you got the idea.
• if you want to develop something new and you are having a gut feeling, it could be food poisoning.. you need features that are oriented to keep the users.
• investing lot of time and resources to develop what YOU believe that is ok could be really expensive, it is better to invest the resources in a smart way in whatever is aligned to the users.
• do you need A/B testing
• because you need users to come back you need to develop what users want/need/expect. At this point you need to know that you are not alone in this task, there are tools for collecting the user feedback (like opinionlab, ideascale, userecho, feedbackify, survey monkey, google forms or any custom tool developed by your self
• there are techniques like QFD, that could translate user requirements into functional requirements, PLEASE USE THOSE KIND OF TOOLS/TECHNIQUES
• usability is really important, the UX (user experience) is also important, do not forget about that!
• use your own data, direct marketing is a good option, take you data and profile your powerusers, try to get to more people like that, again.. BE SMART.

I would like finishing answering the question:

NO, YOU SHOULD NOT BECOME STEVE JOBS.

He was a visionary, one of his abilities was understanding people. You are special because of the abilities you have developed during the time, that effort that makes you shine over the crowd and everything you can do and everything you know.

keeping an eye in the platform

Sometimes we feel like our platform is not doing well, maybe we see slowness in the service. Well the answer is lies in questions like:

• are you ready for growing??
• what is the performance of your platform??
• do you know where are the bottlenecks??

Well, in order to discover those answers, you must need an eye on your platform (maybe more than one), today it could be funny and simple, but especially very powerful because the amazing amount of tools available.

If you are using any cloud provider you will enjoy some basic metrics, but the power is in another place, i will give you some options and as usual you will have the task to choose whatever you like and makes you feel comfortable.

• nagios, basically a nice alerting system, mainly use snmp to get metrics from servers and show you the status per hosts/service (ok, warning, critical).
• cacti, it is just graphing, based in snmp you could monitor metrics.
• munin, helpful to collect data because it handle a custom client/server architecture (munin-client and munin-server), this is not a graphing tool, it is for monitoring.
• collectd, i am starting to feel love for this client/server option (like munin).
• graphite, nice graphing server with many frontends available.
• ganglia, data collecting system (client/server) with a really ugly frontend, but pretty fast.

This is a important point, YOU CAN COMBINE SOME OF THEM!!! Yes, you could use collectd+graphite+nagios. Anyway, as usual you should take care of scalability in two points: 1.- do not overload your platform and 2.- your monitoring platform must be ready to scale with you.

Please, monitor and enjoy understanding where are your bottlenecks and your performance opportunities. Remember that it is required collect and process data in order to have elements to analyze and make desicions

debugging is better if you have logs

If you have been getting errors in your app, and you do not know what is wrong the first thing that hit your mind is "I NEED TO DEBUG THE APP" (usually). That is relatively easy if you are developing in your local environment where only you are accessing or you are doing a standalone app with a cool IDE like eclipse, intelliJ, Visual Studio, xcode or even netbeans; but what happens if your app is a client in a client/server environment.. Ouch, you maybe will have questions like WHO, WHERE, WHEN, WHAT... if you can answer those questions, for sure you will be able to answer the really important question (WHY), because knowing why it is happening you will be able to solve the problem (code, code, code).

To this point I have not said anything new (I hope so), this is the important part:

If you keep track of everything that is happening you will be able to look for the answer to WHO, WHERE, WHEN and WHAT, and keeping that info is what we usually call LOG. Even if you are doing a mobile app, you need to log what is happening, all we know that logs consume space but you can keep just a window (last 5 hours, last day, last 2 days, etc), the size of that window is up to you.

So, now you have logs, SWEET... but wait, if you have a web platform you will have something like this:

• webserver log (apache, iis, nginx, etc)
• scripting log (ruby, php, etc)
• application log (the one that you are creating with your logical structure)

Ouch again, three sources... and I hope you have many clients connecting so you need many webservers... Oops, now i do not know how to consolidate that.. or worse, how do I read that... :(

Do not worry, this is what I really want to share with you today:

There are solutions for logging, but the one I like most is logstash, basically with logstash you can run a process in any machine that is reading whatever is being written to one or more logs, processing, parsing and sending it to another instances where it will be stored, sorted and make it SEARCHABLE, such a beautiful word, so you can get all the logs processed and available in one place, I LIKE IT!! now with this tool you could be doing a better debugging using your logs.

Basically it works like this:

1. your app/server/whatever generates logs locally (inexpensive, fast, easy)
2. you have a process reading those logs (it is called SHIPPER) and sending to the search engine, you can use regular expressions to parse each log entry in the client (AWESOME), the timestamp is courtesy of logstash
3. you can setup a web interface (two or three availables), I suggest kibana that is really good.
4. search, search, analyze, SOLVE.

It is easy, isn't it?

What now?, well watch this presentation of the creator of logstash at the puppetconf, it is for falling in love with the tool. and then start reading the docs for getting started.

A couple of suggestion:

• you can test it in your local machine without problem.
• be careful with the memory usage, you will find different shippers that could help you to use less memory reducing functionality. Keep an eye on that specially if you are using Java.
• if you are planning a live deploy, please take in count the scalability, I would like all of you having millions of users generating logs.

Two final comments, 

1. there is no tool that replace you ability to analyze and THINK, take your time to think in whatever you are watching. Making mistakes is ok, what is not ok is to make mistake because "I do not think that.." 
2. I have not tried to use any shipper in a mobile device, if you found anything please share it, because that is one of the advantages of newrelic.

Enjoy it!