How not to prove that a charity works
Anecdotal evidence
Stories are interesting and motivating, so what's wrong with them? Unfortunately, there are many problems with using them as evidence. For example, how representative is one story of all the people served by a given charity? Is it typical or is it rare?
Also, would the improvements described have happened even without the charity? Would people approved for microloans have run a successful business regardless? Are the addicts who are motivated enough to go to regular counseling the sort of people who are on the road to sobriety anyways?
To solve this problem, you need to go out on the ground and observe. However, you cannot just ask a few questions, take a few photos, and then leave; you need to observe strategically. This is where science comes in.
Take microloans for example. To make sure they are successful, you must make sure you observe many people who received them, not just a few. If you only have enough money to fund 100 people, you should find 200 qualified applicants and run a lottery to see who gets a loan and who doesn't. This way, any differences between the two groups are solely because of the loan rather than because only people who are good with money are accepted.
Next, you follow up with both groups and see what the difference between them is. The surprising result is that a lot of charities don't pass this test. It turns out that the impact they thought they were having was caused by something else.
If you’re wondering whether microfinance actually works, check out this wonderful article summing it up by GiveWell, a brilliant team of charity scientists.
Also, would the improvements described have happened even without the charity? Would people approved for microloans have run a successful business regardless? Are the addicts who are motivated enough to go to regular counseling the sort of people who are on the road to sobriety anyways?
To solve this problem, you need to go out on the ground and observe. However, you cannot just ask a few questions, take a few photos, and then leave; you need to observe strategically. This is where science comes in.
Take microloans for example. To make sure they are successful, you must make sure you observe many people who received them, not just a few. If you only have enough money to fund 100 people, you should find 200 qualified applicants and run a lottery to see who gets a loan and who doesn't. This way, any differences between the two groups are solely because of the loan rather than because only people who are good with money are accepted.
Next, you follow up with both groups and see what the difference between them is. The surprising result is that a lot of charities don't pass this test. It turns out that the impact they thought they were having was caused by something else.
If you’re wondering whether microfinance actually works, check out this wonderful article summing it up by GiveWell, a brilliant team of charity scientists.
Thinking that it "just makes sense"
Studies have shown that we cannot even predict what will make ourselves happy. (2) What makes us think that while sitting in our armchairs we can know what will make somebody on the other side of the city happy? Or on the other side of the world?
The world is complicated. We still don't know everything that's in a carrot. Every time we think we know how big the universe is, we find out that it's even bigger. Every day, there is so much content created on the internet that even if you were immortal you would never be able to catch up. Because of the world's complexity, even the best-laid plans almost never work in practice the way they do in theory.
We cannot simply think that "what makes sense" will follow through in the real world. We need to get off of our couches, test our theories, and use the powerful tools of statistics and logic to see what works. Alternatively, we can let other people do that and read about it from our comfy couches, like you're doing right now.
The world is complicated. We still don't know everything that's in a carrot. Every time we think we know how big the universe is, we find out that it's even bigger. Every day, there is so much content created on the internet that even if you were immortal you would never be able to catch up. Because of the world's complexity, even the best-laid plans almost never work in practice the way they do in theory.
We cannot simply think that "what makes sense" will follow through in the real world. We need to get off of our couches, test our theories, and use the powerful tools of statistics and logic to see what works. Alternatively, we can let other people do that and read about it from our comfy couches, like you're doing right now.
Studies done by the charity itself
Would you trust a study saying that wearing glasses makes you smarter if it was funded by a corporation that made them? Of course not. The same goes with charities.
Now, it's not that charities are tinkering with their data (although it's not out of the question that some do), but rather that people are biased. People see what they want to see, and there are a lot of judgment calls in a study. Some example are which data points to include, how to word a question, which metrics to report on, and which studies to publicize.
Now, it's not that charities are tinkering with their data (although it's not out of the question that some do), but rather that people are biased. People see what they want to see, and there are a lot of judgment calls in a study. Some example are which data points to include, how to word a question, which metrics to report on, and which studies to publicize.
Another reason not to trust a charity's self analysis is that charity workers have their livelihoods and self-worth on the line. It only makes sense that they are biased towards seeing the good in their organization.
This doesn't mean that all studies done by charities themselves are bad; it instead means that you should look more carefully at the methodology of the study and insist that other studies be done my independent researchers.
For example, The Humane League ran a study on pro-vegetarianism leafleting (an activity they do) that looked promising, so we at Charity Science ran a similar study to see whether the results could be replicated. Unfortunately, we did not find the evidence as compelling as they did. Read more about our results here.
This doesn't mean that all studies done by charities themselves are bad; it instead means that you should look more carefully at the methodology of the study and insist that other studies be done my independent researchers.
For example, The Humane League ran a study on pro-vegetarianism leafleting (an activity they do) that looked promising, so we at Charity Science ran a similar study to see whether the results could be replicated. Unfortunately, we did not find the evidence as compelling as they did. Read more about our results here.
How to prove a charity
So that's the way not to prove a charity. If you wanted to prove a charity instead, ideally, a study should:
- Be peer reviewed
- Be done by researchers who are not paid by the charity
- Contain statistically significant data
- Have a control group
- Have a decent effect size
- Measure the right things (e.g. measuring increased long term income, not website traffic.)
- Have a randomly chosen sample
- Use objective measures. (E.g. medical records rather than self-reported health assessments.)
- Be on the specific intervention in the particular country it was run
- Be numerous, with several independent studies rather than just one. One scientist might have a bias, but many scientists all having the same bias is much less likely
Learn more about scientific charities
- See the most evidence-based and cost effective activities
- Learn more about the process of identifying a good charity
- Watch a TED talk on the power of charity science by Esther Duflo, a Charity Scientist Super Hero from IPA
- Stay up to date on events near you and Charity Science news by subscribing to our newsletter