Guest post: How do you sample 170,000 charities?

We're starting to think about our data on charities and the next Almanac, and we've been working with colleagues at the Third Sector Research Centre (TSRC) to think about how we can make sure our estimates of the income and spending of charities are as accurate as possible - and more importantly that we know how accurate our estimates are.

This guest blog is written by David Clifford from TSRC at the University of Southampton, and in it he outlines how you create a sample of 170,000 organisations, and the decisions needed along the way. An alternative suggested title for this blog was "Sampley the Best".

Along with the blog below, we've also shared (using Google Docs) a description of the issues, and a spreadsheet of the figures used to calculate the sampling.

Collecting data can be expensive - so to makes sense to think of ways to keep the costs down.

That's where the principles of survey statistics come in.  A well designed sample can give an accurate estimate of what's happening in the population as a whole. 

And surveys are the foundation of much of social research, across the world.  To cite just one important example, Demographic and Health Surveys provide reliable figures relating to population and health issues in developing countries. 

We've been thinking of ways to design our own survey to collect detailed financial information for charities - income and expenditure streams, assets etc. All charities on the Charity Commission register above a certain income threshold have to submit annual accounts.  We want to make conclusions about this whole set of charities, but we don't have the money to look at every single set of accounts!

So a group of us at NCVO and the Third Sector Research Centre have been thinking how best to design a survey, where we'll look only at a sample of these accounts.

We want to make sure that the answers that we get from our sample are a good guide to what's happening across the population of charities as a whole (in technical terms, we want 'unbiased estimates').

How do we do this?

1. An important principle is that the sample should be randomly selected from the population of charities.  On average (considering a hypothetical scenario, where we take a series of random samples) the estimate from a random sample will be 'unbiased'.  This relates to something called the Central Limit Theorem, which statisticians get very excited about (!).
2. We've also built in some added 'insurance'.  We know that on average a random sample will give unbiased estimates. But we're only doing it once.   So we make sure that our sample is representative of the population in certain ways by 'stratifying' our sample - in other words, dividing our sample into certain groups and then taking a random sample within each of these groups.  That way we're ensuring that we get a representative spread of these groups in our sample.  So we've 'stratified' by size and by region - to ensure that we're including charities of all sizes and right across the country in our sample.

We also want to make sure that we're minimising something called 'sampling variability' - which refers to the tendency that different samples, even if they are random, will give slightly different answers.

How do we do this?

The slightly tricky thing is that we're interested in estimating different things.  We're interested in estimating totals (total voluntary income across the charitable sector), and we're also interested in the typical charity (the 'median' or average voluntary income for a charity).  To minimise sampling variability for totals, we would design our sample in such a way that we oversample the big charities heavily (then 'weight' our totals at the end to ensure our estimates our 'unbiased'.)  To minimise sampling variability for the 'median', we wouldn't oversample the bigger ones.

So what we've decided on is a comprimise between these two different scenarios.  We will oversample the big charities to ensure totals are well estimated, but will also ensure we have enough charities of all sizes in our sample, so that we get good estimates of income and expenditure for the typical charity.

Hope this has been a helpful insight into some of the things we've been thinking about.  I think well-designed surveys are an amazing tool for answering important questions about what's happening in the world around us.  They're also a practical tool, since they're much cheaper than looking at the whole population.