Here's the Ultimate Impact Measurement Tool

I am very proud today to unveil on behalf of NCVO the ultimate impact evaluation tool. After fifteen years of work on impact evaluation, NCVO is launching this tool today – and making it available for free to the entire sector.

What should the ultimate impact evaluation tool look like?

Research and reflection has shown that there are four characteristics that an “ultimate impact evaluation tool” would need to satisfy:
1. It must be able to capture evidence in a huge range of formats, not just one form or type of evidence
2. It must be cost-effective, preferably free, and easily accessible
3. It must be easy to use and to implement
4. It must be powerful and sophisticated, able to analyse a range of evidence and form credible robust judgements about what works and what doesn’t, while minimising bias.

What are the benefits of the ultimate impact evaluation tool?

This tool enables charities, voluntary organisations, social enterprises and community groups:
• To understand what actual difference they are making as a result of their work
• To make more of a difference by innovating and improving, creating greater impact for the same resource
• To justify and show off to the world what they are achieving, helping to bring in people, ideas and money
• To share what works – and what doesn’t – with other projects and organisations increasing overall collective sector-level impact, improving the world faster

So what is this tool, and how can we get hold of it?

Well, believe it or not your organisation already has this tool. In fact, it almost certainly contains a number of them. By using two or three or four of these tools simultaneously, like a supercomputer you can come up with the most robust, credible, powerful, and insightful evidence imaginable.

Here’s the tool:

Yes, the ultimate impact evaluation tool is the human brain, and our associated abilities to perceive, to discuss, to reason, to feel and to learn.

This is not a New Year’s joke. This is an important point.

The ultimate impact evaluation tool is the human brain. Every paper-based or online tool out there, every survey, every clever dial or star, every randomised control trial, every focus group, is an attempt to get close to the interpretative capability of the human brain.  And many of these formal evaluation tools are ersatz versions of the brain, poor approximations of the real thing, chalk instead of bread flour, fizzy lager instead of real beer.

The key is putting to good use the brains in your organisation. In a moment I’ll explain how. First, here’s the theoretical background. If you want to get straight to the practice, feel free to skip this section.

The Power of Tacit Knowledge

I want to reiterate at this stage that while the title of this post may be playful, the argument is deadly serious. Our organisations are stuffed full of tacit knowledge – that deep understanding we have but find difficult to explain to others or codify – and our failure to use it, and our clumsy attempts to substitute it with forms and questionnaires and the like, constitutes a massive waste of knowledge and resource.

Let me briefly explain tacit knowledge, in case it’s a new phrase to you. The classic example of tacit knowledge is ‘riding a bike’.  Very few cyclists are aware that when their bike leans unintentionally fractionally to the left, to keep their balance they actually, actively steer left.  And few know that when they want to turn right, around a corner, they actually initially steer slightly to the left, which starts the bike falling to the left, creating some leeway for when they actively steer right, which avoids over-steering to the right.

Our bodies know all this inside out, but if you try to tell this to a four year old so they can learn to balance and steer, you’d only confuse them and create more scraped knees than necessary. Indeed, if you tell an everyday cyclist this stuff, they can start thinking too much about whether to steer left or right, and it’s been evidenced that trying to put the conscious mind in control of cycling is a dangerous move! Leave it to the part of our brain that knows that it’s doing.

Many writers about ‘knowledge management’ in organisations imply that tacit knowledge is bad, because managers can’t access it. They argue that it needs to be turned into explicit knowledge, needs to be codified and form-alised.  My view is that tacit knowledge is good. It is insightful, consistent, relevant, useful.  Rather than attempts to codify and convert it, we should simply be attempting to use it.

For those interested in tacit knowledge, here are two books and an article that I’d particularly recommend:

• Polanyi, Michael (1966) The Tacit Dimension, Doubleday & Co
• Sanders, A F (1988) Michael Polanyi's post critical epistemology, a reconstruction of some aspects of 'tacit knowing', Amsterdam: Rodopi
• Tsoukas, H (2003) ‘Do we really understand tacit knowledge?’ in The Blackwell handbook of organizational learning and knowledge management, Easterby-Smith and Lyles (eds), 411-427. Cambridge, MA: Blackwell Publishing

How to … use brains as the ultimate impact evaluation tool

Let’s use the example of a free ‘health transport’ service for elderly people. The core service involves volunteer drivers using their own cars to take older people, who are unable to drive and struggle with public transport, to and from hospital and doctors’ appointments.
Let’s assume you’re the manager of this service and you want to understand its impact. How do you find that out?

1. Trust Your Volunteers

Well, ask and trust your volunteer drivers. Day in, day out they take in a vast range of information through their senses – seeing smiles, hearing “thank you”s and hearing how people are, physically feeling the frailty of a man as they help him into the seat, sensing tension or relief or other emotions, and also hearing complaints about the service or perceiving dissatisfaction in body language.  Their brains will subconsciously process this information and be able to give a balanced sense of the impact of the service.

You can unlock this tacit knowledge by simply talking to your drivers face-to-face, in an unstructured or semi-structured interview, asking some open questions such as “Do you think the elderly people value our service?” “What do you think they value most about it?”

Note the crucial phrase “do you think” in both these questions. This gives them the licence to give you their interpretation, and also makes them feel their view is valued.  The question “What do elderly people value about our service?” is far more intimidating – it makes it sounds as though there is a right answer and the first response is likely to be “I don’t know, ask them.”

2. The Evaluation Burden

Of course, a major advantage of asking your volunteers is that it removes the burden from your beneficiaries. A growing and worrying trend is for charities to put all the time and effort costs of monitoring onto the people they are meant to be helping.

Obviously, the burden in this example is falling on your volunteer drivers instead, and that may also be problematic.  The key is only to ask them when you genuinely need new information and learning.  If you know all this already, why are you wasting anyone’s time getting more evidence?

In particular, unless you have a real need for permanent ongoing evidence (and why would you?) it is almost certainly enough to ask them occasionally, say once a year, or once every three years if there are no changes to the service. Or ask just a few them.

One of the most common mistakes in impact assessment is to collect the same data week after week after week, year after year, which all essentially tells you the same thing, costing you lots of extra money for no new learning. Sometimes we think we’re doing this because our funders insist. But do they, really? Or would they, in fact, find an occasional study quite acceptable?

In most cases, it’s absolutely fine to demonstrate the value of a service once, until either the service or the context changes. Indeed, ask your drivers to let you know if they sense anything happening that may require the service to adapt, eg, a noticeable increase in the number of people with dementia who are not ready to be collected at the agreed time.

3. Brains and Bias

How do we deal with potential bias from the drivers, who probably have an interest in promoting the value of the service?

Firstly, trust them. Assume they are honest and self-reflective and able to give a balanced view.

Secondly, talk to more than one and compare the results. Use your own brain to form a view of the consensus.

Thirdly, openly ask about any negative impact and the limits to impact, such as “Do you ever sense that any elderly people are unhappy with the service?” and “Is there anything else you think we should be doing for them?”

Finally, trust your own perception to pick up on anything that sounds over-the-top or is being hidden.

4. Going up a level

So your volunteer drivers can go a long way towards helping you assess the impact of the service. But how can you decide whether it’s a better value-for-money service than an alternative?

Again, trust in brains, this time your own.  As long as you are reflective, open-minded, self-critical and expose yourself to alternative ideas, you are likely to come up with a robust, credible, insightful answer. Be aware of your own potential biases, and ask the views of others.
For example, let’s say you’re faced with a decision about whether to:

• continue with your health transport service as it is
• tweak it a little (eg, move to multiple passengers per trip)
• overhaul it completely (eg, move to paid drivers only and exclude certain beneficiary types)
• stop it and replace it with a completely different service.

This is a management decision. Impact thinking is critical to it. Impact and outputs and cost evidence will all help. But ultimately evidence will rarely ‘speak’ without contradiction. Evidence can contribute to a decision, but is no substitute for it.

Which highlights another common mistake: the belief that the more evidence we collect, the easier our decision will be. The trick is to get enough evidence to tell you things you really didn’t know - and really need to know - to help you make the decisions; and then to know when evidence is no longer the key to the decision and can in fact hinder, by causing procrastination and wasting time and money.

Use your brain, take the decision mindful of the evidence, driven by the desire to make the most impact.

Anti-tool?

I am not suggesting that the use of forms, surveys, dials, etc is necessarily wrong and that we should stop using them to evaluate impact. But I do feel that sometimes these tools are poor substitutes for our powerful, perceiving and interpreting brains. And I certainly think we should be using common sense ways of getting evidence, based on the perceptual and interpretative abilities of the brains at our disposal.

Final thoughts: Brains and decision-making

I think our sector would benefit from more conversations about the relationship between evidence and decision-making, and about decision-making and impact. I’d love to hear from anyone with a view on that, or anything else related to impact or this post.