Don’t just pay lip service to running experiments

Startups are experiment machines. In the early days you’re testing hypotheses, testing market demand, testing pricing, everything is an experiment. The problem is that most startups don’t approach or think about the experiment they are running with any degree of rigor.

Founders tend to think a failed experiment is one that haven’t given the desired result. This is wrong. Any experiment that gives an accurate result, regardless of whether we like it or not, is a successful experiment. The only failed experiment is one that hasn’t given an accurate result.

Rigor is key to running a successful experiment. Lack of rigor leads to a host of problems, such as running an experiment for too long because we don’t like the answer, silently changing the goal posts or searching for data to fit our idea rather than letting the data shape our idea.

Introducing a Base Level of Rigor

Write down the hypothesis you are trying to prove and the constraints of the experiment (how much time, effort and cash are you willing to spend proving it?).

What results will indicate that your hypothesis was proven and what results will show that it was disproven?

If the experiment produces a result that lies between the two, then it’s a failed experiment that either needs to be extended or abandoned.

Example: Experiment to test market demand using Google Adwords

Hypothesis: There is a market demand for my product that can be reached through Google Adwords

Constraints:  €500 and two weeks

Proven: 20+ sign ups

Disproven: Fewer than 5 sign ups

If the experiment results in between 5 and 20 sign ups then we don’t have a result. We can extend the experiment or decide that it’s no longer worth it.

Startups need to be fluid and it is expected that you will need to change your criteria during or even after an experiment, however if you ensure those criteria have been written down then at least when you change them you will do it knowingly. Not doing so guarantees over/under investing in experiments, sand shifting and unclear communication with stakeholders,  and it opens you up to a whole host of cognitive bias.

Don’t just pay lip service to running experiments – actually run experiments. WRITE IT DOWN!

Stop looking at your analytics all the time

Whatever gives you that dopamine hit first thing in the morning: Google Analytics, Adwords, Mixpanel, bank balance, overnight orders, Stripe, stock price, Salesforce ….  it’s time to stop. You know you’re exhibiting compulsive behaviour, you know there’s no business reason you need to check your key metrics 20 times a day and certainly you don’t need to hit refresh to see if things changed in the last 30 seconds.

All you are doing is searching for that next high. Like a self-destructive addict, if the first metric is good you go onto the next, knowing if you keep going you will eventually find the inevitable – a metric that’s going the wrong way. If you are tracking 15 metrics it is statistically improbable that they can all be positive.

Once the negative metric has been spotted, it’s impossible not to keep looking. Logically you know that it’s not statistically significant and you can’t judge your site or product’s performance hour to hour, but you do it anyway, living the emotional rollercoaster of highs and lows dictated by the shape of the graph.

I was this soldier. I’d check 20-30 metrics across 5 different systems within 60 seconds of my eyes opening in the morning, 7 days a week. Occasionally I’d wake in the middle of the night to get my fix. Some days I’d check hundreds of times.

It wasn’t like it was even my job to stay on top of them. I had good people looking after all aspects of the business. But I felt like it was my job. I felt like I had to be on top of every aspect of the business and I boneheadedly took pride in being more up to date than anyone.

Self-realization dawned early on a Tuesday morning. I’d woken at about 5am and, as was my habit, checked revenue, site availability & traffic. Something was very clearly wrong. Even though the site was normally quiet at this time of the day, revenue was way out of line. I got out of bed. I was worried and stressed and I picked up my phone to call my CTO. It was then I realized that the best course of action was to do nothing and wait till the office opened and fresh, well rested engineers looked into the problem. My revelation was that information is valueless unless you are prepared to act on it.

I created a new rule for myself – only look at metrics when I was prepared to act on them. For example unless looking at the bank transfers that had arrived overnight would lead me to make a different decision then I wouldn’t look.

So I created a schedule for myself. Firstly, no looking at analytics in the AM unless they were needed for a specific purpose such as a meeting.  This allowed me to me to be proactive in the morning without having the day blown off course.

Schedule

  • Everyday 1.30pm: Bank Balance
  • Every Friday: 1.30pm Analytics, Revenue and Sales Pipeline  (Alright I’ll admit it became daily on the last week of the month. I didn’t say I was totally cured).
  • First day of the month. Adwords, Engineering tickets, site performance and everything else

Did I stick to it? Mostly. Every now and then I’d still crack but I easily cut out 95% of my habit.  I was happier and able to focus on the longer term and didn’t waste time stressing over irrelevant data.  This resulted in more thoughtful business decisions, better time with my family, and better time with me.

Next time you reach for your crutch of choice, ask yourself what decision you are going to make differently. If you are not sure and the answer is that you ‘just need to know’ then stop. You are not making anything better by looking and you are making life a lot worse for you and everyone around you.


If you are going to sell then commit 100% – don’t waste your time “cleaning your apartment”

Adding the job of sales to the CEO or founder’s role is common for startups that are exiting the gates of customer discovery. Now they’ll sell the product for 50% of their time and continue with their other responsibilities for the remainder of the time.  

This almost never works. If you are going sell, then sell with 100% of your time and effort and find someone else to deal with your other responsibilities. Anything less than this requires a mythical level of willpower and dooms most people to failure.

It’s like that time you were meant to be studying for exams and you found yourself cleaning your bedroom, then doing the dishes. Studying was hard and uncomfortable and you’d search for any reason to avoid starting. You’d search for any activity that you could define as ‘productive’.  You wouldn’t play video games, or go to the pub. You’d clean your bedroom because you could fool yourself into thinking you were still being productive and therefore feel like less of a waster.

It’s the same thing with sales. Sales is uncomfortable and full of rejection. If the “salesperson” can do anything else and still feel like they are being productive then they will do that instead of selling. This is particularly the case with non commissioned salespeople such as founders.

Almost no one wants to pick up the phone and start to navigate a large organization, dealing with rejection every step of the way.  Good sales people do it because there is no other way to achieve their goals. As soon as you give them multiple goals then you’ve given them a way out.

Remove all other activities and responsibilities from your salesperson – even if that’s you.  Give them exactly one way in which they can feel good about themselves. Selling, not cleaning their bedroom.

Sales – it’s about Getting to No

Trying to sell something?

Well the answer’s no. Get used to it.

Stop looking for ‘yes’. It’s never yes. It’s always no.  If it  was yes then they’d already be your customer.

Your goal should be discover why it is no. Once you know why, you can work to overcome the objection. Maybe it’s price (it’s never price), credibility, risk, timing, authority, budget. You are dead in the water if you don’t know why.

If you go in looking for yes, then the best case is you find out that it’s not a ‘yes’. The worst case is you don’t even get that and the prospect stays in your pipeline like a ship becalmed. Your job is to force the issue, to question, to probe and to discover why they won’t do business with you NOW

Why I Hate A/B Tests

I hate A/B tests that are recommended to startups when they don’t have the volume to run them. I hate when they are used to avoid being courageous. I hate the false lessons they teach. I hate the illusion of certainty they give. In short I hate nearly everything anyone says about them and nearly every application of them.  So yes, I hate A/B tests.

They Provide No Insights and Generate False Narratives

A well executed A/B test will tell you which side performed better but it won’t tell you why. At the end of the day you may know that “Sign Up” works better than “Create Account”. Is it because the word “Account” has a very specific meaning for your target audience? Did “Sign Up” simply fit the button better? There are any number of possible explanations and you have no idea what the real answer is.  

The worst thing is that humans need narratives and when presented with a fact that is unsupported by a narrative we invent one. That invented narrative will spread further and be remembered longer than the original ‘fact’. But you have no idea if that narrative is actually true – someone just made it up. Try it yourself, go along to Optimizely and try not to create a narrative in your own mind.

They require a LOT of traffic

Split tests require a lot more traffic than most people who are running them have. If you have a page converting at 5% then on average you need about 30,000 visitors to that page to run an A/B test with 95% confidence. With the random walk inherent in your results you might sometimes need triple that. How long is that test going to take to run?  Are you happy standing still during that time?

Even then, the test is going to be wrong 5% of the time – statistically significant does not equal true.

The moment anything changes it invalidates your result

An A/B test requires both sides of the test to be identical except for the item being tested.  So what happens when something changes after the test? A soon as anything changes then the test needs to be rerun because the change invalidates the test. NO-ONE does this.

  • If you change a word on the page, a colour, a button, then the test is invalid
  • If your audience changes then your test is invalid
  • If the damn time of the year changes then your test is invalid
  • Even if significant time passes, your test is invalid as consumers tastes and views evolve over time

Most things aren’t worth split testing

Most of the time the cost of testing is going to far outweigh the potential benefit of the change. Your page probably has thousands of variables. The vast majority aren’t going to be worth the cost of testing.  Don’t expect minor changes to have major results.

On what basis are you choosing your test and what is your rationale?  Fear of being wrong, fear of being caught out, fear of being seen as rash or just because you think you should? Stop covering your ass and have the courage to plough forward and trust your own judgement.

P.S. A/B tests have their place and that place is where you are making fundamental changes that could radically impact performance and where you have large scale.

Adwords is like poker: If you can’t see the sucker, you’re it

When it comes to effective advertising methods Google Adwords rules the roost. You get to display your advert at exactly the right time to exactly the right prospect – the moment your potential customer is looking for it. This is astonishingly powerful and will generate over $100 billion in revenue for Google in 2018. However, all that revenue comes at the expense of the advertiser.  Most categories are now so competitive, making the cost of advertising very high, and when the costs are so high, advertisers need to live and breathe the figures in order to turn a profit.

In a competitive space Google will eventually take all the profit. Imagine:

  • There are four identical competitors
  • Each sell a digital book for $10 which they have written with no associated cost of sales
  • 10 clicks from Google Adwords are required to generate a sale
  • There are only 3 advertising slots available from Google.

How much will each advertiser bid per click?

Game theory tells us they will bid up to $0.99 cents. Any further and they won’t turn a profit. Any less and the other three competitors will out compete for the three slots and revenue goes to zero. So in for $10 sales, Google will take $9.96 – all but $0.04. In effect, Google takes all the money!

But it’s much worse than that. In the above example we limited the number of competitors to 4 and assumed they were all logical actors with complete knowledge and skills. Unfortunately this is rarely the case.

Most competitive markets have a few ill-informed actors. These actors will frequently outbid all the competition in order to maximise their traffic and resulting revenue. Imagine the above scenario with a couple of these ill-informed actors. Each bid $1.05 a click, resulting in Google taking $10.50 for each $10 sale. Google takes more than all the money!

But won’t the ill-informed actor eventually exhaust their budget, realise the error of their ways and either amend their bidding strategy or exit the market? Well yes, but there is an ample supply of ill-information actors ready to take their place.

So if you’re newish to advertising on Google, take a good look around at what your competitors are doing and if you can’t spot the one who is over bidding then chances are it’s you.   

Feedback Easily Given is Nearly Worthless

Ever received a cold call? Ever said the first thing that came into your mind to get rid of the person? Now imagine that salesperson meticulously compiling that feedback into a report. 42% of people are too busy, 30% already have a solution, 15% are driving and about to go into a tunnel and 13% of people are rude. Further imagine a company actually making business decisions on this data, confident in the knowledge that they are doing so on the basis of real market knowledge.

Ridiculous isn’t it? Yet we all do this.

No one likes to give difficult feedback. That’s why employee reviews are hard. It’s the same for customers. They don’t want to tell you that your baby is ugly – so they make something up – just so you go away.

You should always view feedback through the lens of how difficult it was to give.  If the feedback was easy to give you should regard it warily and probe for more. Conversely if the feedback was difficult you should really value it. Someone just went through an emotionally difficult time to give it to you. It has real value – do something with it.

When a prospect decides not to proceed with you, they will typically respond to your request for feedback. However, for the most part they just want to get rid of you and will tell you whatever is easy and end the conversation, not the real reason.

Be suspicious of:

  • You were too expensive
  • We decided not proceed with any vendor
  • Our budget got pulled
  • Corporate rules only allow us to do business with companies in business for over 5 years

Your job is to delve deeper and force feedback that’s difficult to give – that’s where the value is.  

Feedback Gold

  • We don’t think you can do the job
  • Your product sucks
  • Your lack of sales process worried us
  • We think you are going to go bust
  • Our engineering team hates your engineering team

Parking One Side of a Marketplace

Building a marketplace is hard. It’s difficult because you’ve got two different customers: buyers and sellers who are typically two different people (exceptions include classified ads and dating apps).  Two different customers implies:

  • Two different value propositions
  • Two different routes to market
  • Two different products
  • Two different support structures
  • Two different EVERYTHING!

Marketplaces are two businesses running simultaneously and both have to succeed for the overall operation to prosper.  This results in marketplace failure rates that are much worse than a normal startup, however they have the benefit of being able to scale wildly if they are successful.

I’ve dealt with a lot of marketplaces over the last year. Everything from a private tutoring market in the Middle East, rental accommodation in Ireland to dog boarding in Europe. After ten years in WhatClinic, people think I’m some sort of expert. I don’t have any secret sauce, however I do know a lot of the pitfalls.

The first pitfall is trying to build both sides of the marketplace before you have sufficient resources / team. Even in a successful marketplace with teams supporting both sides of the business, it’s hard for the CEO to wear two hats; it’s nearly impossible when resources are very limited. Focus your resources on one side of the market first.

Recognise who your primary customer is. In general, this is the side that has fewer problems finding the other party . This is the side of the market that you have to crack. It’s where the real risk is in the business – if you can crack this side of the market, the other side is simple in comparison.  

So for WhatClinic the primary customer was the consumer not the clinic, for Google Ad Network it’s the advertiser not the partner website, for Amazon it’s the consumer not the seller. Uber, HostelWord, Hotels.com & Opentable all built supply first, parking the consumer side until they were ready [Note some edits made here from original posting thanks to comments from Ronan Percival)

  • If Amazon has has loads of consumers it is safe to assume they’ll be able to sign up vendors
  • If Google Ad network has loads of advertisers then partner websites will flock to them.
  • Uber reckoned that if they sort out supply that demand would exist

But it’s a chicken/egg (horse/cart) problem. It’s difficult to build one side without the other because no value is being delivered. What’s the answer? Try and park away the second side of the marketplace by directly addressing the primary side’s value proposition without building a market on the other side. 

Can you park away one side of the market?

  1. Offer the service directly yourself. So if you are an online babysitting marketplace you can directly hire a few babysitters in your first city. Once you have proven the value proposition for the parent and have more demand than supply, you can switch your attention to building the other side of the market, targeting the sitters.  Amazon was an online store before it became a marketplace.
  2. Seed the market. Get one prestigious name on the secondary market that will drive large volumes from the primary market. For example if you are a jobs board then a job for Ferrari or Dolce & Gabbana can drive tens of thousands of CVs.
  3. Build the supply without the relationships. For example if you are a real estate marketplace you can simply list properties without developing the paying relationships with the real estate agents. Once you have the primary audience you can start to build the relationships
  4. Build the primary audience for a directly related informational need. If you are an investment real estate marketplace you could build your primary customer audience by publishing research information before building the market.

My final thought is try and recognise when the marketplace exists because of a customer need for a market or because of internal business requirements. Uber’s riders & drivers don’t want a market (the rider just want a ride and the driver a fare) but Uber does. They don’t want the hassle and legal liability with directly employing drivers. However the consumer clearly wants a marketplace for homes to buy as they want to compare lots of options.  Having clarity on exactly who the marketplace serves should give you insight on how to get it started.

Investor & Founder Alignment is a Myth – Maintaining the Fantasy is Important to Both

It’s a common refrain among professional investors that founder and investor interests need to be aligned. The theory is sound – as long as the investor’s interests are also in the founder’s interests then the investor is protected by the self-interest of the founder. The problem is that this theory ignores reality. An investor’s success is only partly dependant on the success of the company whereas the founder’s success is solely dependant on it.

An investor’s success depends on the blended return of a number of investments. While they are generally over exposed to a market segment, they have their risk diversified within that segment. The founder’s return, on the other hand, can be thought of as binary. They either lose or they win, and there isn’t much of a middle ground.

I’ve seen this innate misalignment between investor and founder cause problems several times. Generally it only happens at the extremes – when the company is going very well or very badly. These are the moments where alignment matters most, as misalignment can come at a huge cost to the founder or investor.

Where the company is doing very well it often makes sense for an investor to increase the company’s chances of failure but also increase the potential returns. For an investor managing a portfolio this is a sensible action. As long as they increase the potential return by more than the chance of failure then the value of their portfolio improves. For the founder, with a binary outcome, this makes little sense – they should be more focused on minimising the chance of failure.

Example: Company ABC is worth €20M and the investor and founder own 50% each. The founder has no other notable assets whereas the investor has 30 identical investments (obviously a very contrived example). It is in the investor’s interest to try and 10X the company’s value even if there is an 80% chance of failure. Ideally the investor would do the same with every company in their portfolio.  However, for the founders, introducing an 80% risk of failure into their only asset of value is clearly a foolish thing to do.

This is a real problem. Investors have blocked the sale of companies that would have made their founders very happy because they thought a better sale was possible, and founders have pushed through sales that aren’t in investors’ best interests. Investors have encouraged and promoted courses of action to ‘pour fuel on the fire’ of promising companies even when doing so increased the chances of bankruptcy.  

When companies are doing extremely poorly and the only way forward is for the founder to continue spend more of their time in the venture, it’s in the investor’s interest for the founder to continue. The investor doesn’t suffer the cost of the founder’s ongoing investment, but gets to keep their investment ‘live’. Meanwhile the founder suffers the full brunt of the cost of continuing to work at the enterprise. From the founder’s perspective, a dispassionate analysis frequently suggests that shuttering the company would be the best course of action. Once again, different investment exposure results in misaligned interests.

It’s nice to think we stand together and together we fall/succeed but both investors and founders should be working on a different set of calculations and assumptions and recognising that they have different interests.  

Even though investor and founder interests aren’t aligned, it is still in everyone’s interest to pretend that they are. Founders want potential investors to think their interests are aligned so that the investor will put money in the company. Investors want founders to think they’re aligned so they can convince founders to take a course of action that is not in the founder interest.  Normally everyone maintains this fantasy for as long as possible but when things go very well or when the shit hits the fan, make sure you know what your interests are and don’t rely on the other party.   

Your Company isn’t ‘Alive’ and it can’t ‘Die’ – But You Can

I recently advised a young entrepreneur to shut down his two year old business. This was despite being able to raise enough money to ‘survive’.  His company was generating revenue but was in a very small sector and the available financing wasn’t going to be enough to pivot his way out of it. It was only going to lead the entrepreneur back to the same situation in 6 – 9 months, except this time he’d be that much older.  It was the very essence of surviving for survival’s sake.

What struck me the most about our conversation was our use of language and how it didn’t serve us well. We talked about the company as if it were a living, breathing entity. We used words like ‘alive’, ‘survive’, ‘keep alive’, ‘time to live’ and ‘die’. This use of emotional language automatically put us in place where talking dispassionately about the business was difficult and if ever there was a time where dispassionate analysis was needed it was then.

I particularly remember the entrepreneur’s response to me telling him that the small amount of money he could raise wouldn’t get him anywhere except back to the same place he was. “Well it’s better than being dead” was his response.  

This anthropomorphizing of his company was automatically leading him to the wrong conclusion and if we continued to use language that suggested his company was ‘alive’ it was going to be close to impossible for him to accept that his company should be shut down. As humans, we find the ‘death’ of something to be tragic. However shutting a company that’s not going anywhere isn’t tragic – the founder continuing to expend more of his limited human capacity on it is.   

A business isn’t alive – it’s not even a natural thing, like rock or stone. It’s an artificial construct and it can’t be alive, dead, surviving or any of the other things that are the primacy of living things.  The only thing that’s alive about a company is the entrepreneur and the people working there. Use of language that suggests otherwise doesn’t serve founders.

It may serve investors but more on that later.