Ouch! The cost of badly designed tests

pharmacy tablets

I was talking to a friend the other day about her love of alternative medicines and her belief in their ability to heal all manner of ills. We’ll call my friend Hanna.

Hanna had recently had a terrible toothache that had grown in intensity during a long gruelling day at work. By the time she got home she’d had enough. She went straight to her cupboard full of expensive tablets, powders and pills that she’d bought from her trusted supplier goodhealthdoesntcomecheap.com After a day of agony and determined to get a good nights sleep Hanna took a cocktail of her healthy magic bullets and retired for the night.

The terrible pain soon went away and it wasn’t long before Hanna drifted off into a deep and peaceful sleep.

The next morning when she came downstairs the kitchen table was covered with the detritus of her home made cure. As she gathered up the boxes, bottles and jars she totted up the cost of her natural remedy and realised that her cure for toothache had not come cheap.

One of these remedies had definitely done the trick……the trouble was she had no way of knowing which one it was.

When we have a problem that we want to solve it is often a good idea to try out a number of possible solutions at the same time because we don’t know if any of them will work. But we need to be careful if we are tempted to try them all out in the same place……..on the same problem……just to make sure we get the outcome we were after. Because even if we do achieve the result we hoped for we run the risk that like my friend Hanna we will be left not knowing which of our new services or interventions was the one that had the desired effect.

So with social innovation as it is with natural remedies. It is very easy to spend a lot of money trying all sorts of new ideas……and even to achieve the effect we hoped for……whilst learning nothing at all about which of our ideas actually made a difference.

Taking a little time to plan how and where to test our idea; to think about what information we might need to collect in order to be sure that our idea has worked; to decide how long we’ll need to try it in order to reach a meaningful conclusion about its effect; is all time well spent.

Hanna certainly got rid of her nagging toothache but she had spent over £350 on expensive natural remedies in the process – without learning anything about which one had actually got rid of the pain…….ouch!


4 responses to “Ouch! The cost of badly designed tests

  1. interesting points John
    I think the need to remedy a problem is one thing often born out of pain urgency or discomfort.
    However that’s different to an organisation showing willingness (and a degree of bravery) to try new things so you can be a social innovator -influencer such as @bromfordlab.
    Accepting that sometimes things fail and yes its critical to know when to pull the plug or not to let your passion for an idea take over ( yep that’s me sometimes lol ) taken from @paultaylor Blog ” 10 things we learned from launching an innovation lab”


  2. Interesting post John, thank you.
    The ‘throw everything at the wall and see what sticks’ approach is something that happens quite a lot, particularly when there is an urgent situation.
    This is a very effective illustration you have used.
    As you rightly point out, if you can’t tell which one out of many approaches worked, how can you learn anything to apply in future situations.
    An approach that is a huge waste of money, all for the sake of thinking about evaluation as part of the experiment.
    I like the new blog format by the way, very clear and good to read.

    Liked by 1 person

  3. Thanks Chris.
    I think many of the organisations that want to innovate, try new things, get better at what they do, etc. also tend to be stuffed full of people who are activists…..the sort of people who can’t be doing with sitting around talking about random controlled trials or measures of success…….they’ve had an idea and now they want to go out and try it! Trouble is it leads to loads of wasted effort, time and cash.
    We want lots of perfect failures not imperfect successes.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s