Don’t just pay lip service to running experiments

Startups are experiment machines. In the early days you’re testing hypotheses, testing market demand, testing pricing, everything is an experiment. The problem is that most startups don’t approach or think about the experiment they are running with any degree of rigor.

Founders tend to think a failed experiment is one that haven’t given the desired result. This is wrong. Any experiment that gives an accurate result, regardless of whether we like it or not, is a successful experiment. The only failed experiment is one that hasn’t given an accurate result.

Rigor is key to running a successful experiment. Lack of rigor leads to a host of problems, such as running an experiment for too long because we don’t like the answer, silently changing the goal posts or searching for data to fit our idea rather than letting the data shape our idea.

Introducing a Base Level of Rigor

Write down the hypothesis you are trying to prove and the constraints of the experiment (how much time, effort and cash are you willing to spend proving it?).

What results will indicate that your hypothesis was proven and what results will show that it was disproven?

If the experiment produces a result that lies between the two, then it’s a failed experiment that either needs to be extended or abandoned.

Example: Experiment to test market demand using Google Adwords

Hypothesis: There is a market demand for my product that can be reached through Google Adwords

Constraints:  €500 and two weeks

Proven: 20+ sign ups

Disproven: Fewer than 5 sign ups

If the experiment results in between 5 and 20 sign ups then we don’t have a result. We can extend the experiment or decide that it’s no longer worth it.

Startups need to be fluid and it is expected that you will need to change your criteria during or even after an experiment, however if you ensure those criteria have been written down then at least when you change them you will do it knowingly. Not doing so guarantees over/under investing in experiments, sand shifting and unclear communication with stakeholders,  and it opens you up to a whole host of cognitive bias.

Don’t just pay lip service to running experiments – actually run experiments. WRITE IT DOWN!

Stop looking at your analytics all the time

Whatever gives you that dopamine hit first thing in the morning: Google Analytics, Adwords, Mixpanel, bank balance, overnight orders, Stripe, stock price, Salesforce ….  it’s time to stop. You know you’re exhibiting compulsive behaviour, you know there’s no business reason you need to check your key metrics 20 times a day and certainly you don’t need to hit refresh to see if things changed in the last 30 seconds.

All you are doing is searching for that next high. Like a self-destructive addict, if the first metric is good you go onto the next, knowing if you keep going you will eventually find the inevitable – a metric that’s going the wrong way. If you are tracking 15 metrics it is statistically improbable that they can all be positive.

Once the negative metric has been spotted, it’s impossible not to keep looking. Logically you know that it’s not statistically significant and you can’t judge your site or product’s performance hour to hour, but you do it anyway, living the emotional rollercoaster of highs and lows dictated by the shape of the graph.

I was this soldier. I’d check 20-30 metrics across 5 different systems within 60 seconds of my eyes opening in the morning, 7 days a week. Occasionally I’d wake in the middle of the night to get my fix. Some days I’d check hundreds of times.

It wasn’t like it was even my job to stay on top of them. I had good people looking after all aspects of the business. But I felt like it was my job. I felt like I had to be on top of every aspect of the business and I boneheadedly took pride in being more up to date than anyone.

Self-realization dawned early on a Tuesday morning. I’d woken at about 5am and, as was my habit, checked revenue, site availability & traffic. Something was very clearly wrong. Even though the site was normally quiet at this time of the day, revenue was way out of line. I got out of bed. I was worried and stressed and I picked up my phone to call my CTO. It was then I realized that the best course of action was to do nothing and wait till the office opened and fresh, well rested engineers looked into the problem. My revelation was that information is valueless unless you are prepared to act on it.

I created a new rule for myself – only look at metrics when I was prepared to act on them. For example unless looking at the bank transfers that had arrived overnight would lead me to make a different decision then I wouldn’t look.

So I created a schedule for myself. Firstly, no looking at analytics in the AM unless they were needed for a specific purpose such as a meeting.  This allowed me to me to be proactive in the morning without having the day blown off course.

Schedule

  • Everyday 1.30pm: Bank Balance
  • Every Friday: 1.30pm Analytics, Revenue and Sales Pipeline  (Alright I’ll admit it became daily on the last week of the month. I didn’t say I was totally cured).
  • First day of the month. Adwords, Engineering tickets, site performance and everything else

Did I stick to it? Mostly. Every now and then I’d still crack but I easily cut out 95% of my habit.  I was happier and able to focus on the longer term and didn’t waste time stressing over irrelevant data.  This resulted in more thoughtful business decisions, better time with my family, and better time with me.

Next time you reach for your crutch of choice, ask yourself what decision you are going to make differently. If you are not sure and the answer is that you ‘just need to know’ then stop. You are not making anything better by looking and you are making life a lot worse for you and everyone around you.