Measuring Impact - A Guide To Resources
A guide to resources for impact measurement by voluntary organisations (based on research conducted in 2003).
Find out more about how to manage and improve your impact.
Contents
1. Introduction
2. Why measure?
4. Obstacles to impact measurement
5. Methodologies, toolkits and overviews
6. Problems with existing approaches and the need for further research
7. What next?
8. Resources
1. Introduction
With an annual income of around £16 billion, the voluntary sector is increasingly perceived as 'big business', and arguably it is. But there is a key difference between the voluntary and private sectors: in the absence of share prices or profit figures, how do we know what we are achieving? In other words, how do we know what
our impact is?
This is a difficult question for voluntary organisations. A wealth of literature and research over the last 20 years has gone some way towards providing some of the answers, but there is still a long way to go. The diversity of the sector means that there can be no 'one size fits all' approach for measuring impact. And, although a range of tools is available for organisations who want to measure some aspect of their impact, many of these are resource intensive and difficult to implement. So, as more organisations are beginning to grapple with the idea of impact, what is the current state
of play?
This guide aims to offer some direction to those with an interest in impact and its measurement. For those still not sure, it firstly suggests what the drivers for impact measurement are. It then brings together and defines the main terms currently in use, including outputs, outcomes
and impact. It goes on to discuss some of the problems and difficulties involved in measuring impact. The main body of the review then looks at some of the available tools and approaches for those interested in delving a little deeper. There is a strong emphasis on tools developed in the UK, and tailored specifically to the voluntary sector, although where relevant, reference is made to overseas work. The guide ends with a short discussion of how the field may be advancing, and a directory of resources.
The guide is a useful starting point for voluntary organisations who want or need to demonstrate their impact. It forms part of a larger Community Fund project that includes testing a range of approaches to
measurement, but more importantly it contributes to a growing range of resources that should enable organisations, and ultimately the sector, to effectively demonstrate their impact.
2. Why measure?
Motivation for measuring the voluntary sector's impact can come from a number of sources. By combining new methodologies with lessons learned from the public and private sectors, voluntary organisations have been measuring various aspects of their impact for years.
However, while few would question the benefits of measuring an organisation's impact, the Government's recent emphasis on 'what works' and the voluntary sector's involvement in the delivery of public services, means that the sector has more reason than ever to demonstrate its worth and this
has lead to added interest in impact measurement.
Government initiatives and public service delivery
Government initiatives including Best Value, and the recent Treasury review of the voluntary sector's role in the delivery of public services, mean that voluntary organisations are keener than ever to demonstrate their ability to deliver public services.
Voluntary organisations are competing not only with each other for the public's donations, but with other public, voluntary and private sector organisations for public service delivery contracts. They need to be able to demonstrate their added value; that is, any particular qualities or strengths that the voluntary sector has in comparison with the other sectors. These might include the ability to engage the community, to contribute to social capital and to access hard-to-reach groups. It is generally accepted that with these distinct qualities, the sector needs a distinct type of performance measurement. The danger is that if the voluntary sector is judged on the same criteria as the other sectors, it will not be demonstrating the characteristics that make it unique. Also, unlike the private and public sectors, the voluntary sector lacks any other means of measuring its performance, like share prices, profits and election results, hence the need for some kind of performance measurement system tailored to the sector (Kendall and Knapp, 1999).
Additional pressure to demonstrate impact comes from a recent report by the Prime Minister's Strategy Unit (Private Action, Public Benefit) which made a number of recommendations to update the legal and regulatory framework for charities and voluntary organisations. The report sets out a range of measures to encourage charities to be more open about their objectives and performance. For example, it recommends that charities be made to demonstrate 'public benefit' and that they should adopt higher standards of information provision and transparency.
Funders
Additional external pressure to measure may come from
non-government funders. While some funders may ask for evidence that a funded organisation or programme has met its outcomes, some will require proof of its past performance as a prerequisite to awarding funding. Or, a funding organisation may itself want to carry out an evaluation of its funded projects in order to establish the impact of that funding.
Donors
Motivation for measuring impact may also come from donors. With an increasing number of organisations to choose from, donors are becoming more discerning in their choice of who to give to. Not only will an organisation want to demonstrate to its existing donors that it is using funds in an efficient and effective way, but it will also want to appeal to potential donors. Furthermore, the public not only funds voluntary organisations directly through donations, but indirectly through taxes, and has an additional stake in the sector as users and beneficiaries: all reasons why voluntary organisations are keen to be publicly accountable.
Internal motivation
Equally, motivation to measure impact may come from within an organisation. It may want to collect information on its own performance in order to help it to improve efficiency and effectiveness. This information may be financial, like expenditure, or it may be outcome information based on organisational objectives.
There are also benefits to an organisation arising from the actual process of impact measurement. The process of defining aims and objectives is in itself useful for encouraging staff to think about and clarify their organisation's goals. Also, measuring and demonstrating impact can help to motivate staff by providing them with concrete evidence that their work is making a difference.
Soft outcomes
Finally, there has been a lot of discussion in recent years about 'soft outcomes': intermediate outcomes that are achieved along the route to attaining the ultimate goal of an activity or organisation.
The final goal, or 'hard outcome' might be for example, getting unemployed people into work. While this might be the ultimate goal, it is not necessarily a feasible one in all cases and may not be achievable in the short-term. There may be a whole range of reasons why a person is unemployed: they may lack interview skills, motivation, punctuality or specific job skills, or they may have a mental or physical illness. There are therefore many separate areas to address and the process of achieving the end result - employment - may not be straightforward. It is possible to think of this progression towards a final goal as a journey, and even if the end result is not achieved, some progress will almost certainly have been made towards this goal. The amount that has been achieved in the process is sometimes referred to as 'distance travelled' and usually consists of 'soft outcomes'. These tend to be fairly intangible in nature and may be seen as stepping-stones, or necessary achievements en route to the final, hard outcome. In the case of getting an unemployed person into work, they may include increased confidence or a change in attitude.
The danger is that funders can be primarily concerned with the final, hard outcome, and will see anything short of this as a failure. Organisations therefore want to be able to demonstrate that something worthwhile has been
achieved - that some distance has been travelled.
3. What is impact?
The previous section looked at some of the motivations for measuring an organisation's impact. We turn now to the definition of impact and how it relates to the more widely used concepts of outputs and outcomes.
There is a whole range of terms that occurs frequently throughout the literature with varying degrees of consistency. Broadly speaking, they concern the kinds of things that can be measured (inputs, outputs, outcomes and impact), and the processes and practicalities involved in measurement (benchmarks, targets, indicators and evaluation). There is much wider agreement over some terms than others.
There is little disagreement over the more tangible concepts like inputs and outputs. Inputs are the resources that contribute to a programme or activity, including income, staff, volunteers and equipment. Activities are what an organisation does with its inputs in order to achieve its mission. These could be, for example, training, counselling or guidance. Outputs are countable units, and are the direct products of a programme or organisation's activities. They could be classes taught, training courses run or people attending workshops. In themselves they are not the objectives of the organisation but are a means to an end.
Outcomes are the benefits or changes for participants, or intended beneficiaries. They are more difficult to measure than outputs, as they will often be less tangible and less countable. They may occur during participation in an activity or sometime afterwards. Outcomes are usually planned and are set out in an organisation's objectives. Outcomes may be causally and linearly related; that is, one outcome leads to another, which leads to another and so on, forming a linear sequence of if-then relationships. Outcomes may be defined as hard or soft. A hard outcome is an organisation's final or core goal, like getting unemployed people into work. Soft outcomes are things achieved en route to this final hard outcome, like a change in attitude or increased punctuality. Soft outcomes are more likely to be intangible and therefore difficult to measure, and although attitudes to soft outcomes are slowly changing, they tend to be of less interest to funders.
We now get into more difficult territory. Impact is a widely used but rarely defined term in evaluation literature. Everyone wants to know how to measure their organisation's impact but without knowing quite what they mean by the term. Blankenburg (1995) describes it as:
"...long-term and sustainable changes introduced by a given intervention in the lives of beneficiaries. Impact can be related either to the specific objectives of an intervention or to unanticipated changes caused by an intervention; such unanticipated changes may also occur in the lives of people not belonging to the beneficiary group. Impact can be either positive or negative."
A simpler definition might describe impact as being any consequence or effect of an activity, and it is this definition that we will use throughout this guide:
Impact is any change resulting from an activity, project, or organisation. It includes intended as well as unintended effects, negative as well as positive, and long-term as well as short-term.
This then is an all-encompassing definition that includes outputs and outcomes as well as unintended effects and both short-term and long-term effects, be they positive or negative, regardless of the recipient which might be organisations, individuals or the environment. Outcomes and outputs, along with expenditure and employment are therefore components of the much wider, all-encompassing definition of impact. This is the definition that will be used in this guide.
The breadth of this definition means that some sub-division is necessary to make it more manageable. This can be done in whatever way makes most sense to the organisation in question. Impact may be categorised according to the level at which it is being measured: for example, sector, organisation, programme or activity level. Similarly it might make more sense to divide impact into broad types, like environmental, political, social and economic. Other subcategories might concern the types of beneficiaries, such as communities, local businesses or the environment. Or, it may make more sense to categorise impact as being intended or unintended. Most organisations embarking on some sort of impact measurement will be content not to measure the full spectrum of their impact. It is important therefore, for an organisation to be selective and realistic about the types of impact that it wants to measure, and find an approach that meets its particular needs.
| Term | Definition |
|---|---|
| Inputs | Resources that contribute to a programme or activity, including income, staff, volunteers and equipment |
| Activities | What an organisation does with its inputs in order to achieve its mission. They could be training, counselling or advice |
| Outputs | Countable units; they are the direct products of a programme or organisation's activities.They could be classes taught, training courses delivered or people attending workshops. In themselves they are not the objectives of the organisation. |
| Outcomes | The benefits or changes for intended beneficiaries.They tend to be less tangible and therefore less countable than outputs. Outcomes are usually planned and are therefore set out in an organisation's objectives. Outcomes may be causally and linearly related; that is, one outcome leads to another, which leads to another and so on, forming a linear sequence of if-then relationships. |
| Impact | All changes resulting from an activity, project, or organisation. It includes intended as well as unintended effects, negative as well as positive, and long-term as well as short-term. |
There are also a number of terms that relate to the process of measurement, the definitions of which are fairly consistent throughout the literature. Benchmarks are data that act as a baseline and are used for before-and-after comparison. A target, standard or objective is a particular level of outcome that an organisation or activity aims to achieve. Indicators are specific data that can be measured to determine whether an organisation has met a particular outcome. Evaluation is a general term for the process of determining what has been achieved during or after a particular activity.
4. Obstacles to impact measurement
Impact measurement is rarely straightforward. There are numerous challenges to be faced by any organisation measuring its impact, and as a rule the more intangible or long-term the factors being measured, the more difficult the process. Furthermore, even if no problems are encountered along the way, there are limits to the usefulness of different types of impact measurement. This section explores some of these problems and limitations, helping organisations to be realistic about what can be achieved, and to be prepared for the possible pitfalls.
Attributing cause
Identifying a causal relationship between a particular activity and a particular outcome is often difficult and sometimes impossible. The activity, intervention or programme being assessed may be only one of many factors that have affected the beneficiary during the time period in question, so attributing a particular change to the programme may not be possible. An organisation may wish to limit their measurement to outcomes that can be directly attributed to a particular activity.
Establishing causality proves particularly troublesome when measuring long-term impact. Not all changes occur immediately or even soon after the activity or programme is complete. In fact, some changes may not be seen until several years after the end of the intervention. Not only will an organisation be keen to see results soon after a particular project or activity is complete, but the implications for measuring changes several years down the line are enormous. The reason is that several years' worth of other factors will have had an effect, and ascertaining exactly how much of the impact can be attributed to the original activity is likely to be impossible. The if-then chain of causally related outcomes mentioned in the previous section is a good example. Each outcome along the chain is not the only factor contributing to the next. At each stage along the journey, a host of external factors will have an impact on subsequent outcomes, making it difficult to establish to what extent each outcome can be credited to the original activity.
Prevention activities
Organisations whose main purpose is the prevention of something, like teenage pregnancies or drug use, may find it hard to demonstrate positive impacts, partly because success for them is something not happening, and also because it is often difficult to identify exactly who the beneficiaries are. For example, a particular drug use prevention campaign may result in five percent fewer people using illegal drugs in a particular year. However, identifying those people who would have taken drugs if it were not for the programme is likely to prove impossible.
Baseline data
Some types of measurement require baseline data representing the position before the programme or activity begins, which can be used for comparison after the event. This assumes that you know what the types of outcomes will be in advance in order to be able to collect the relevant baseline data. While this is a common way of measuring impact, it is somewhat restrictive as it limits the measurement process to outcomes that were anticipated beforehand. It can result in a rather blinkered view of impact, and makes it unlikely that any unanticipated impacts will be identified during the process.
Type of data
Traditionally, there has been a general preoccupation in impact measurement with quantitative data, and an unfamiliarity with or wariness of the use of qualitative data. While many aspects of impact may be measured perfectly well using quantitative data, more difficult-to-measure aspects of impact may require qualitative techniques. For example while it may be possible to measure unanticipated impacts using quantitative data, at the very least they will need to be identified using qualitative approaches, like interviews or focus groups.
Of course, there are problems with the use of qualitative approaches, like the fact that they rely quite heavily on people's perceptions. This links to the problem of attributing cause to effect, mentioned earlier. Unless a respondent is able to accurately say "I feel 32% happier than I did last year, as a result of this activity" then it is never possible to know exactly which causes are linked to which effects. And, it may be that a person's perception of an activity's impact will change from one day to the next, depending on how positive they are feeling on that particular day. This is particularly likely to be the case where the very nature of the activity means that beneficiaries are under mental strain or in a delicate state of mind. These could be people with drug addictions or mental health problems, or people in a highly emotional state, like victims of violent crime.
"It's hard to say I feel more self-confident and it's 70 per cent due to planting trees with Groundwork and 30 per cent due to a Women's Institute course I went on." (Prove it! NEF)
Impact of the measurement process
Another problem is the impact of the measurement process itself. In many cases this may be an advantage: the process of measuring impact may make staff think about their objectives and activities in new ways. However there will be occasions when any effect resulting from the intervention of an assessor will not be desirable as it will affect the extent to which impact can be attributed to the activity or programme. Similarly, there may be ethical issues to take into account when assessing the impact of an activity that deals with sensitive issues or vulnerable clients.
Attitudes to measurement
Attitudes to measurement often prove to be a barrier, and these can include a mistrust of measurement in general. Managers may perceive it as a waste of resources that should be spent on delivering results rather than measuring them. They may see impact measurement simply as more unnecessary paperwork, taking up time that could be better used elsewhere. Members of staff may feel they are being examined or the quality of their work scrutinised. Funders may feel unhappy about a proportion of their grant being used for measurement, and in turn, organisations bidding for funding may not feel able to justify including impact measurement in their funding application.
If the evaluation is being carried out externally, say by a funder, this may have a detrimental effect. An external assessment may be seen as a test. The assessor may be regarded with suspicion as someone who is checking whether things have been done correctly, possibly with a view to cutting funding, and this will not encourage people to be entirely truthful. Evaluation is too often seen as a negative process, associated with cuts in funding or closure of projects. Another possible attitude-related barrier is the feeling held by some that impact measurement in the voluntary sector simply isn't a good idea: that to measure a voluntary organisation's performance is to "look a gift horse in the mouth" (Davey, 2001).
Diversity of the sector
The heterogeneity of the voluntary sector poses a number of problems for impact measurement. No single methodology is applicable to all voluntary organisations. Clearly there is the need for a wide range of methodologies tailored to the requirements of different types of organisation, depending on their type, size, activities, objectives, the aspects of their impact that they want to measure and their reasons for measuring their impact. Furthermore, the diversity of the sector means that developing a method of measuring the impact of the sector as a whole is likely to be impossible.
Cost
Perhaps the most fundamental of all the objections to impact measurement is that of resources. Measurement of any type requires resources, no matter how much an organisation manages to integrate the measurement process into its everyday routine. Some performance measurement systems have been developed especially for small organisations (for example, PQASSO and Quality First). While not measuring impact as a whole, they at least provide small organisations with a means to demonstrate and improve some aspects of their impact.
5. Methodologies, toolkits and overviews
Very few tools, methodologies or guides claim to address the broad concept of impact. Most focus on outcomes, while some pay particular attention to characteristics that are unique to the voluntary sector. Others look at organisations' internal processes, or at relationships with their stakeholders.
However, the broad, all-encompassing definition of impact described earlier means that all these approaches can be said to address some aspect of impact, whether they use the term or not. While it would be ideal to have a tool that allows an organisation to measure every aspect of its impact, this is unrealistic. What we find instead is a range of tools that have been developed to suit different types of organisation, and fulfill different purposes. As a result, the range of methodologies and tools for measuring impact in the voluntary sector reflects the sector's diversity.
One area of the voluntary sector where a significant amount of impact measurement has been done is regeneration. Although the range of regeneration activities carried out by the voluntary sector is huge, there are particular features that are common to most regeneration. These include community involvement and a contribution to social capital. A number of methodologies for measuring the impact of regeneration organisations have emerged which pay special attention to capturing these particular qualities. Examples include Achieving Better Community Development by the Scottish Community Development Centre.
Similarly, some methodologies have been developed by funders, who want to measure the impact of the projects they fund. These include the Millennium Commission's study of the social impact of their Millennium Award scheme and Lloyds TSB Foundation's First Steps in Impact Assessment. The latter has been developed with a view to helping other organisations to measure their impact, by testing the feasibility of using a questionnaire as a tool for assessing impact.
Common themes occur throughout some of the methodologies. Perhaps the most persistent is the use of community involvement in impact measurement. Another is the idea that the voluntary sector differs fundamentally from the public and private sectors, and that any methodology for measuring its impact should reflect and capture this difference. Also, it seems that there is general agreement that not all that much is known about the impact of the sector. This is being rectified, at least in part, by a number of recent studies which have looked at the range of methodologies being used to measure impact in the voluntary sector (Independent Sector's Measures Project for example) or to establish the impact of the voluntary sector in a number of countries (The Nonprofit Sector: For What and For Whom, Salamon et al, 2000).
Another common feature is the development of toolkits, methodologies or guidelines for inhouse use, but with the recognition that they may be of interest to a wider audience. Examples include Connexions' A little Book of Evaluation, NEF's Prove it!, and Lloyds TSB Foundation's First Steps in Impact Assessment.
This section looks at a selection of the methodologies and guides that have been developed for the measurement of impact, as well as one or two studies that tell us something about the use of various approaches in the sector as a whole. There is a strong emphasis on the UK, and while not exclusively voluntary sector, there is a strong voluntary sector bias.
Details of all the guides, toolkits and studies mentioned here can be found in the resources section at the end of this guide.
Guides
There are a number of guides to help organisations that want to measure their impact. They tend not to offer prescriptive methodologies, but advise on the approaches that can be adopted and define key terms. Alcohol Concern has published two editions of its A DIY guide to implementing outcome monitoring. The guide provides step-by-step advice on various aspects of outcome measurement including selecting an appropriate monitoring system and analysing and presenting the data. The guide is clear and easy to read and is a good starting point for organisations embarking on impact measurement for the first time. This approach however, is focused specifically on outcomes, and the identification of outcomes prior to measuring makes it unlikely that any unanticipated impact will be captured.
Another useful guidebook is Connexions' A Little Book of Evaluation. Connexions is the Government's support service for people aged 13 to 19. It provides advice and guidance as well as access to development opportunities and is delivered through local partnerships consisting of public, private and voluntary sector organisations. The booklet was originally developed to offer guidance to Connexions managers in Lincolnshire, although it was recognised that it would have relevance to a much wider audience, including organisations not involved with Connexions.
The booklet does not offer a definitive methodology. Instead, it offers guidelines for evaluation, taking the reader through the various stages of the process. It is based on the idea that evaluation can help us to ascertain "what works best" and that it can result in an evidence base with which to inform future policy and practice. The booklet discusses the main stages involved in the evaluation process, and the methods that can be adopted. These include different ways of sampling and of collecting data, including questionnaires, focus groups and interviews, and ways of reporting the information collected. The booklet presents the range of techniques that can be adopted at each stage in the evaluation process. It is a useful tool for any organisation starting to think about impact measurement, although organisations would need to look elsewhere for a methodology.
Other guides include NEF's Prove it! and CES's How well are we doing?.
Measurement by funders
Organisations who fund projects in the voluntary sector always want to know how well they are doing. Are they meeting targets? Are they efficient? Are they having any effect?
The projects they fund will usually be in line with the funding organisation's own aims and values, meaning that they will have a particular interest in their impact. Measuring impact will also help funding organisations to know which kind of projects to fund in future. Funders lend a different perspective to impact measurement. They may have different priorities from the organisations they fund, and their approaches to impact measurement will differ accordingly. While most funders insist on certain amount of monitoring throughout the duration of the project, some have developed innovative in-house methodologies for measuring the impact of their funded projects.
Lloyds TSB Foundation funds a range of projects focusing on their three areas of special interest: family support, challenging disadvantage and discrimination, and promoting effectiveness in the voluntary sector. It carried out some research to establish whether the work that they fund has an impact, and to test the use of questionnaires as a tool for the measurement of impact. The hope is that this will contribute to the development of a method for assessing impact for use by the Foundation as well as being helpful to other voluntary organisations.
Similarly, the Millennium Commission carried out a social impact study of its Millennium Awards programme. The programme awards small grants of roughly £2500 to individuals or small groups to carry out a range of community projects. This research differed fundamentally from the Lloyds TSB Foundation work as it was not intended to result in an approach for use by other organisations.
Other funders who have commissioned impact measurement research include Barclays Bank. Barclays Sitesavers is a scheme funded by Barclays and run by Groundwork UK which involves local people in the transformation of derelict land into recreation areas. Barclays and Groundwork commissioned the New Economics Foundation (NEF) to carry out some research to develop a means of measuring impact, that involved local people and that could be used by Barclays Sitesavers projects. NEF recognised that the audience for such a methodology would be wider than just the Sitesavers projects, and so has published its guidelines in the form of a handbook, for use by any community or regeneration organisation that might find it useful (Prove it!: measuring the effect of neighbourhood renewal on local people). It is not a methodology as such, in that it is not prescriptive. Instead, it flags up the issues that need to be addressed when carrying out an evaluation, and offers advice, but ultimately leaves it up to the reader to decide which indicators to use, how to select a sample and carry out the survey, and how to present the results.
Identifying and measuring the voluntary
sector's 'special features'
There is recognition by most voluntary sector researchers that the sector is fundamentally different from the public and private sectors and that it has a number of unique qualities. There is recognition too, that these qualities will not be captured using evaluation techniques designed for the public and private sector and that special tools and indicators are needed.
In Measuring the Outcomes of Voluntary Organisation Activities, Kendall and Knapp (1999) point out that as well as not having indicators like share price by which to measure their performance, voluntary organisations are also not necessarily scrutinised by a governing or regulatory body. Kendall and Knapp recognise the need for performance measurement in voluntary organisations, but argue that
they are fundamentally different from the public and private sectors, and therefore require a different approach to performance measurement.
They go on to consider the widely accepted criteria, the four Es (economy, effectiveness, efficiency and equity), which have been used extensively for the evaluation of public sector organisations.
Kendall and Knapp accept that the four Es can be applied to the voluntary sector and that they "...are likely to be necessary but not sufficient for assessing voluntary organisation performance". To use only these criteria would fail to do justice to a whole range of characteristics that are unique to the sector. These include its role in campaigning and advocacy, its contribution to social capital, its innovation, and the provision of goods and services that would not otherwise be provided. To take account of these characteristics, Kendall and Knapp propose the use of four more criteria or 'domains' (choice, participation, advocacy and innovation) in addition to the already established economy, effectiveness, efficiency and equity. For each of their domains, as well as the original four, they suggest a number of indicators, but point out that not all domains and indicators will be relevant to all organisations, and that some will be more important than others.
Kendall and Knapp's model for outcome measurement is purely theoretical. However, it was put to the test by DHSS Voluntary Activity Unit who tested their indicator set on a number of organisations. Various conclusions were drawn. It was thought that some of the terminology used in the indicators was too strongly based on economic theory and needed to be simplified if they were to be fully understood. Also, it was thought that some indicators relating to final outcomes would be very difficult to measure, particularly where the outcome is something intangible, like a change in attitude for example.
Kendall and Knapp's work has also been incorporated into Evaluating Community-Based and Voluntary Activity in Northern Ireland, by Community Evaluation Northern Ireland. It developed a framework based on different levels of social capital, which could then be used to select appropriate indicators from various approaches, including Kendall and Knapp's model.
Other research to consider the special features of the sector includes The Nonprofit Sector: For What and For Whom? (Salamon et al, 2000) and Evaluating Community-Based and Voluntary Activity in Northern Ireland, by CENI, mentioned above.
Bell (2000), in his paper Identifying just what is special about the social economy talks about how the social economy (including the voluntary sector) adds value to the services it delivers by, among other things, contributing to social capital, and making use of community resources. He points out that the social economy lacks the necessary tools to be able to demonstrate this 'added value'.
"Whilst evaluation of the outcomes and achievements of regeneration and social programmes as a whole is well developed, it currently offers few practical tools for assessment of the social economy's full contribution (particularly the added value it is thought to bring)."
(Bell, 2000)
Overviews
There are a number of studies which look at the range of impact measurement techniques that have been used by and are available to voluntary organisations.
Independent Sector's Measures Project does just this. It carried out a survey of the types of outcome measurement used by nonprofit organisations in the US. The work was a response to the general lack of knowledge on the state of outcome measurement in the nonprofit sector, despite the fact that in recent years the sector has been under increasing pressure to demonstrate and report on its outcomes. The results are based on interviews with 36 organisations, selected from the respondents of Independent Sector's 1998 Measures Survey. Organisations were not selected randomly: they were representative of those that were involved in relatively advanced outcome measurement. Organisations were questioned on their outcome measurement procedures, indicators, data collection, and reporting of results, as well as the cost of outcome measurement. The resulting report is aimed at managers of nonprofit organisations who are interested in measuring their outcomes. It provides examples of procedures for outcome measurement, as well as guidelines and recommendations for good practice, based on the information collected from the organisations.
Other overviews look at the range of tools and resources available to the sector for measuring its impact. Bozzo and Hall's A Review of Evaluation Resources for Nonprofit Organizations is a good example.
Demonstrating the impact of the sector as a whole
Very little research has been done on measuring the sector's collective impact. The Nonprofit Sector: For What and For Whom? (Salamon et al, 2000) is an attempt to establish the impact of the sector in a number of countries and to highlight the ways in which it differs from the public and private sectors. Like the Independent Sector research, this paper was a response to the gap in the sector's knowledge about its own impact, and presented results of a comparative study that looked at the impact of nonprofit organisations in a range of countries. It identifies a number of features of impact measurement. These include the fact that impact is something more than just outputs, that the process of measuring impact must be systematic, and that it must not be blind to the possible negative impacts of the activity in question. The study aims to assess the impact of nonprofit organisations in almost 40 countries.
Quality standards
A particular breed of assessment tool called 'quality standards' has developed in recent years. They concentrate not on the broader aspects of impact but on internal processes within organisations. The focus is on how these processes can be improved to allow smoother running of an organisation, or to increase its efficiency and effectiveness.
Some quality standards have been developed specially for the voluntary sector, while some are tailored to organisations of a particular size. PQASSO (Practical Quality Assurance System for Small Organisations) for example, is designed specifically for organisations with fewer than 12 paid staff. Some voluntary organisations develop their own quality standards, or adapt an existing off-the-shelf system, tailoring it to their specific needs.
The motivation for adopting a quality standard has typically come from within an organisation, although more recently there has been additional pressure from some funders. Furthermore, the prestige associated with some quality standards provides additional motivation. Quality standards provide a means by which organisations can measure certain aspects of their impact, and the way in which they relate to internal processes within the organisation. There are too many quality standards to describe in detail, but the following table shows some of the main ones and their particular strengths.
| Measure of success | EF Excellence Model | Investors In People | ISO 9000 | PQASSO |
|---|---|---|---|---|
| Overall Organisation improvement | 5 | 4 | 2 | 5 |
| Confidence that users' requirements are being met | 4 | 2 | 3 | 4 |
| Motivation of our people | 4 | 5 | 1 | 4 |
| Efficient use of our resources | 4 | 3 | 5 | 4 |
| Meeting the demands of our funders | 3 | 2 | 3 | 3 |
| Greater accountability | 4 | 3 | 2 | 4 |
| Check by an independent, external person | 3 | 5 | 5 | 1 |
| Formal accreditation to a standard, with a 'badge' | 2 | 5 | 5 | 1 |
| Comparison with other organisations | 4 | 1 | 1 | 2 |
| Help in dealing with change | 5 | 4 | 2 | 3 |
| Clear diagnosis of how good we are | 5 | 2 | 1 | 5 |
Evaluation of regeneration and community development
Regeneration and community development is a common theme throughout the impact measurement literature.
Prove it! developed by NEF on behalf of Barclays Sitesavers and Groundwork UK is a good example. It was developed in response to the fact that while some quantitative data were collected on Barclays Sitesavers projects, it was felt that they did not show the full range of social, economic and environmental benefits. The guide is a nonprescriptive one, which leads the reader through the various issues relating to evaluation. It is aimed at organisations that have not done any impact measurement before. While it was originally developed as a methodology for regeneration programmes, the principles can be applied to any type of community programme.
A different kind of approach was developed by the Scottish Community Development Centre (SCDC), a partnership between the Community Development Foundation and the University of Glasgow. They developed a framework for planning and evaluating community development activities called Achieving Better Community Development (ABCD). It is not a methodology, in that it is not prescriptive regarding the specific measurement techniques that should be adopted. Rather, it is an evaluation plan, based on the principles of community development, where evaluation is seen as an integral part of the process of community development, and community involvement as a vital part of any evaluation process.
The fundamental principle is that any evaluation of community development should be carried out using the same principles that govern community development itself. That is, that communities should be involved at every stage of the process, including planning and implementation, as well as discussing ideas for the future.
The report suggests two models for thinking about community development, and these form the basis for their framework for implementing evaluation. The first is the 'cycle of change' model, which shows inputs feeding into processes, which result in outputs that contribute to outcomes (inputs - processes - outputs - outcomes). The second is the 'pyramid of outcomes', which consists of a four-tier hierarchy of outcomes. They range from the four fundamental themes of community development including personal empowerment and participation, through to the higher level, more general outcomes of a livable, equitable and sustainable community.
Another approach developed by Community Evaluation Northern Ireland (CENI) resulted from a project commissioned by the Voluntary Activity Unit. The motivation for the project was the feeling in the voluntary sector that most existing models and indicators for assessing impact are difficult to apply in practice and do not do justice to the diverse or 'multi-layered' nature of the sector's activities. Take for example the established system for measuring the performance of publicly funded organisations, whereby performance is judged on four criteria: economy, efficiency, effectiveness and equity (the four Es). This, it is felt, cannot do justice to the social outcomes of organisations' activities. This is because voluntary organisations do more than just deliver services; they empower communities and involve them in their own development, thus contributing to social capital. So, the sector cannot allow itself to be judged by the same criteria as the public sector, and needs an evaluation system that recognises and captures its unique qualities. Any evaluation system designed for the voluntary sector should have a strong emphasis on these wider outcomes.
The resulting report (Evaluating Community-Based and Voluntary Activity in Northern Ireland) looks at the Kendall and Knapp research on indicator sets that go beyond the four Es, as well as the Scottish Community Development Centre's Achieving Better Community Development (both are reviewed above). These two studies, says the report, provide a comprehensive range of indicators for evaluation in the voluntary and community sectors respectively. The aim of CENI's work was not to develop new indicators, but to develop a unified framework that incorporates some of the indicators suggested by both these studies, using social capital as a theme.
Another approach was adopted by Highlands and Islands Enterprise in their Community Confidence research. The aim of the project was to measure the level of community confidence in highland and island communities. While not a measure of an organisation's impact, it includes indicators for people's perceptions of and confidence in their community that could form the basis of a community regeneration project's impact measurement system.
Loughborough University's Water, Engineering and Development Centre has produced a guide to measuring performance in the small projects that form the basis of development and regeneration in low-income communities in South Asia. The guide (Performance monitoring of micro-contracts) presents tested performance indicators and guidelines on how to use them. The idea is to provide a tool by which small projects can measure some of their socio-economic impacts, in addition to standard measures like cost and quality.
Stakeholder and systems approaches
A systems approach to voluntary sector impact assessment is not a methodological tool, so much as a conceptual tool. It is primarily a way of organising information as well as informing the process of gathering it.
As a conceptual tool, systems approaches bring a number of advantages to impact assessment, including the idea that the "the whole is more than the sum of its parts", and an emphasis on the relationships between a system's components. It is a "no organisation is an island" approach that recognises that any organisation is part of a complex network, or system of relationships, and that its impact cannot be properly measured without taking account of this network.
Social auditing is a stakeholder based approach to evaluation and was developed by the New Economics Foundation (NEF). Like systems approaches, it recognises the fact that organisations do not work in isolation, and that there is a whole range of stakeholders that should be consulted to achieve a broad picture of the organisation's impact.
It was developed in response to the fact that while most organisations regularly report on their financial performance, there is no legal obligation for them to report on their social performance. It recognises the fact that most organisations have at least some non-commercial objectives, and that are a number of reasons why they might want to measure how well they are delivering them. It is not aimed specifically at voluntary organisations. However, it offers a particular advantage for voluntary sector organisations, as it provides a way of demonstrating to funders that they are achieving non-financial outcomes which are none-the-less relevant to the organisation's objectives and should be taken into account.
Social auditing is essentially a stakeholder centred approach to social performance measurement, which takes into account all those who are involved with, or are affected by, the organisation. A social audit is not a one-off measurement or evaluation. It is an ongoing process that draws on information from a range of sources. Not all data need to be collected especially for the social audit. Much of the information is likely to be collected anyway as part of an organisation's financial and personnel records. The process should be repeated every year, so that a picture of the organisation's social performance over a number of years can be built, up allowing comparisons over time to be made. Involving stakeholders in agreeing indicators and completing questionnaires means that not only can the social audit process itself contribute to increased community involvement and social capital, but stakeholders will feel some 'ownership' of the audit process and are more likely to take it seriously and have an interest in the results. The interactive, 'all hands on deck' nature of the social audit is also likely to help focus employees' and other stakeholders' ideas about what the objectives and aims of the organisation are.
Social auditing addresses a number of the issues surrounding impact measurement that were raised earlier. The stakeholder approach means that a range of perceptions is taken into account, avoiding too narrow a view of the organisation's impact. The use of qualitative data addresses the criticism that impact measurement is often too focused on quantitative data, and allows for better measurement of intangible and unanticipated impacts.
Like the work of Kendall and Knapp, social auditing recognises the fact that financial measures are not enough to provide a complete picture of an organisation's performance. There are other factors to take into account, and while Kendall and Knapp concentrate on features that are unique to the voluntary sector, social auditing is concerned more with the noncommercial, social or ethical aspects of an organisation's performance.
6. Problems with existing approaches and the need for further research
Many of the limitations and criticisms associated with impact measurement in the voluntary sector are addressed by some of the toolkits and methodologies discussed in the previous section. In particular, those developed in recent years have tried to embrace a broader view of impact, moving away from simple outputs and taking into account unanticipated, long-term and intangible effects. No approach, however, can address all the limitations, so it is impossible for an organisation to measure its impact without being open to some degree of criticism.
Collective impact
Adequate measures of collective impact still do not exist. While some approaches advocate the measurement of a cluster of projects at once, this is only likely to work for very small projects, perhaps coordinated or funded by the same organisation.
No approach for measuring the collective impact of larger organisations exists, and there may be an increasing demand for an approach of this kind given the recent increase in partnership working. The demand for such a methodology might come from funders, local authorities, or regional development agencies who might be interested in the collective impact of say, a regeneration partnership. It is doubtful however, whether the individual organisations that make up the partnership will be happy to take part in a collective impact assessment without really just being interested in their own impact, or how much of the collective impact they are responsible for. They might argue that an impact assessment can be of no help to an individual organisation if they are unable to establish how much of the total impact they contributed, to allow them to amend their organisational procedures and objectives accordingly. This is particularly likely to be the case in a mixed sector group, where voluntary organisations may be keen to distinguish themselves from the public and private sectors.
Qualitative methods
Qualitative approaches, as a rule, do not lend themselves to quantitative statements like '45% of beneficiaries thought that...'. This is due in part to the small number of respondents usually involved in qualitative research, as well as the loosely structured and open ended style of questioning often adopted. Some of the approaches discussed above however, get round this by adopting a semi-qualitative, structured style of interview, and a large enough sample to be able to treat the resulting data in a quantitative way (the Millennium Commission social impact measurement research for example). Such approaches however, make it less likely that the interviewer will stumble across new, unanticipated impacts during the interview.
Unanticipated impact
The 'blinkered' view of impact discussed earlier (blindly focusing on intended outcomes) has been addressed by a number of approaches. For example, social auditing gives a more rounded view of impact, by taking into account the wide group of stakeholders that affect or are affected by the organisation. However, selecting indicators at the beginning of the process means that again, there is little scope for encountering unexpected impacts throughout the measurement process.
Systems approaches also address this problem of the blinkered view of impact. By encouraging an organisation to think about how it fits into a complex network of organisations and relationships it is more likely to see how broad and far reaching its impact can be, rather than just concentrating on its intended beneficiaries.
Cost
Impact measurement remains an expensive business, although some approaches designed especially for small organisations are sensitive to this. As a rule, qualitative approaches are more costly than quantitative ones, and so, if small organisations with few resources are to be lured away from the straightforward counting of outputs towards measuring impacts, they have to be presented with an approach which is financially feasible. Some quality standards like Quality First and PQASSO have been developed specifically for small organisations with very few paid staff. Social auditing relies fairly heavily on information that is already collected for one reason or another, supplementing it with additional information where necessary. The fact that it is an ongoing process means that it should be able to be incorporated into the day-to-day running of an organisation and not feel like a burden.
The cost issue may become less of a problem, as organisations become more prepared to include impact measurement in their bids for funding, and funders become more willing to pay for it.
7. What next?
Impact measurement is no longer in its infancy. The range of approaches is impressive, and most are tailored to particular types of organisation, meaning that there is an appropriate method available for almost every voluntary organisation that wants to measure its impact. There are gaps though, and the previous section highlighted a number of these.
With recent interest in the sector's added value, there is likely to be an increased demand for tools to measure collective impact. In an ideal world, we would be able to measure the impact of the whole sector, but in the absence of a methodology capable of such a feat, we will have to settle for something much less ambitious.
Attention has turned in recent years to the idea of 'soft outcomes' or 'distance travelled', with the recognition that while final or hard outcomes may not always be achieved, some distance may have been travelled towards the target destination, and particular skills attained along the way. This recognition has resulted in guidance for the measurement of soft outcomes and we can probably expect further developments in this area in the future. This is particularly likely to be the case with the sector becoming more involved in the delivery of public services, as it will find itself increasingly in need of a means to demonstrate progress towards desired outcomes.
The sector is by no means lacking in tools for demonstrating impact. Organisations can pick and choose from a wide range which focus on various aspects of impact. The fact that no single tool can measure the full spectrum of impact means that organisations have to be quite specific about what they want to measure and what resources they can allocate to the process. And no organisation need single-handedly assess all the available tools and approaches in order to hit upon one which is appropriate for them - there are a number of overviews which summarise the available tools making it easier for organisations to choose appropriately. The following section lists some of the main resources available to organisations who wish to measure their impact.
8. Resources
This section lists some of the main resources for organisations who are interested in measuring some aspect of their impact. Some are relevant to a particular type of organisation such as funders or regeneration organisations. Others provide a particular kind of guidance, like step-by-step instructions or an overview of tools and resources.
Guides
- British Quality Foundation (1999) Assessing for
Excellence: a practical guide for
self-assessment. British Quality Foundation. - Burns, S. (2000) A DIY guide to implementing outcome
monitoring. Alcohol Concern www.alcoholconcern.org.uk - Charities Evaluation Services, How well are we doing?
CES. www.ces-vol.org.uk - Connexions (2000) A Little Book of Evaluation.
Connexions/DFES Publications www.connexions.gov.uk - Quality Standards Task Group (QSTG)
Self-Assessment Workbook - Measuring
Success. NCVO
Measurement by funders
- Annabel Jackson Associates (2001)
Social Impact Study of Millennium Awards - Carrington, D. (2002) The Investor Approach:
A Way Forward for the Community Fund?
Community Fund
www.davidcarrington.net/consultancy - Lloyds TSB Foundation for England and Wales (2001)
First Steps in Impact Assessment.
Lloyds TSB Foundation - QSTG (2001) Quality Matters - Funders' Attitudes
to Quality. NCVO - Walker, P., Lewis, J., Lingayah, S., and Sommer, F.
(2000) Prove it: measuring the effect of
neighbourhood renewal on local people.
NEF Publications
Overviews
- Bozzo, S. L. and Hall, M. H. (1999) A Review of
Evaluation Resources for Nonprofit Organizations.
Canadian Centre for Philanthropy - Fine, A., Thayer, C. and Coghlan, A. (1998)
Program Evaluation Practice in the Nonprofit
Sector. Innovation Network, Inc. - Morley, E., Vinson, E. and Hatry, H. (2001)
Outcome Measurement in Nonprofit Organizations:
Current Practices and Recommendations.
Independent Sector - Murray, V. The State of Evaluation Tools
and Systems for Nonprofit Organizations. - Palmer, P. and Randall, A. (2001) Financial
Management in the Voluntary Sector.
Routledge - QSTG, In Pursuit of Excellence: Measuring
Quality in the Voluntary Sector. NCVO
Quality standards
- Astbury, R. Quality Assurance in
the Voluntary Sector. CES - British Quality Foundation (1999)
Assessing for Excellence: a practical
guide for self-assessment. British Quality
Foundation. - Pearce, J., Raynard, P. and Zadek, S.,
Social Auditing for Small Organisations
- A workbook for trainers and
practitioners. NEF Publications
www.neweconomics.org - QSTG (1999) Approaching Quality
- a guide to the choices you could make.
NCVO - QSTG In Pursuit of Excellence:
Measuring Quality in the Voluntary Sector.
NCVO - QSTG (2001) Quality Matters -
Funders' Attitudes to Quality.
NCVO - QSTG (2001) Self-Assessment
Workbook - Measuring Success.
NCVO
Evaluation of regeneration and community development
- Highlands and Islands Enterprise/Systems Three
(1999) Highland and Islands Community Confidence.
www.hie.co.uk - Morrissey, M. and McGinn, P. (Community Evaluation
Northern Ireland) (2001) Evaluating Community-Based
and Voluntary Activity in Northern Ireland.
The Voluntary Activity Unit, DSD - Scottish Community Development Centre. Achieving
Better Community Development. www.scdc.org.uk - Sohail, M. and Cotton, A. (2000) Performance
Monitoring of Micro-Contracts for the Procurement
of Urban Infrastructure. Loughborough University - Voluntary Activity Unit/Northern Ireland Statistics
and Research Unit/PriceWaterhouseCoopers (1999)
Measuring the outcomes of voluntary sector activity - Walker, P., Lewis, J., Lingayah, S., and Sommer, F.
(2000) Prove it: measuring the effect of neighbourhood
renewal on local people.
NEF Publications www.neweconomics.org
Stakeholder and systems approaches
- Checkland, P. (1990) Soft Systems Methodology
in Action. Wiley Publishers - Pearce, J., Raynard, P. and Zadek, S., Social Auditing
for Small Organisations - A workbook for trainers
and practitioners.
NEF Publications www.neweconomics.org
International
- Arveson, P. (1999) Translating Performance Metrics
from the Private to the Public Sector. - Bozzo, S. L. and Hall, M. H. (1999) A Review of
Evaluation Resources for Nonprofit Organizations.
Canadian Centre for Philanthropy - Morley, E., Vinson, E. and Hatry, H. (2001)
Outcome Measurement in Nonprofit Organizations:
Current Practices and Recommendations.
Independent Sector www.independentsector.org - Murray, V. The State of Evaluation Tools
and Systems for Nonprofit Organizations. - Salamon, Hems, and Chinnock (2000)
The Nonprofit Sector: For What and for Whom?
Johns Hopkins University. www.jhu.edu
General
- Adams, J. (2001) NGOs and Impact Assessment.
INTRAC. www.intrac.org - Bell, J. (2000) The challenge of acceptance:
identifying just what is special about
the social economy. - Blankenburg, F. (1995) Methods of Impact
Assessment Research Programme: Resource pack
and discussion. The Hague: Oxfam UK/I and Novib - Charities Evaluation Service (2000):
Discussion paper 7. CES - Davey, B. (2001) The Measurement of Performance
in the Community and Voluntary Sector. - Dewson, S., Eccles, J., Tackey, N. D. and Jackson, A.
(2000), Guide to Measuring Soft Outcomes and Distance
Travelled. Institute for Employment Studies, Brighton.
www.employment-studies.co.uk - Herman, R. and Heimovics, R. Researching non-profit
organisational effectiveness: whose criteria? A rejoinder
to Osborne and Tricker. Voluntas, 6:1, 93-99 - Kendall, J. and Knapp, M. (1999) Measuring the
performance of voluntary organisation activities.
Belfast Voluntary Activity Unit - Osborne, S. P. and Tricker, M. Researching non-profit
organisational effectiveness: A comment on Herman
and Heimovics. Voluntas, 6:1, 85-92 - Outcome Measurement Resource Network.
Measuring program outcomes: A Practical Approach. - Palmer, P. and Randall, A. (2002) Financial Management
in the Voluntary Sector: New challenges.
Routledge Performance Assessment Resource Centre.
www.parcinfo.org - Plantz, M., Taylor Greenway, M. and Hendricks,
M.Outcome Measurement:
Showing Results in the Nonprofit Sector. - www.nhi.org/online/issues/sf119.html
Find out more about how to manage and improve your impact.
