The most common mistake charities make is measuring the wrong thing

One problem we have seen with many organizations is that they often measure intermediate goals without making sure these intermediate goals lead to the real results they want.  For example, if a fundraising organization claimed that we were successful at raising money for top charities because they had high website traffic in a particular month you would not trust their fundraising. This is because while website traffic is clearly a positive thing and might contribute to more people donating to effective charities, it is not necessarily connected to real good happening in the world. 

We think there are a few reasons charities fall into this trap, and wanted to suggest a few ways we can make sure we really measure what matters and maximize that instead of less useful outcomes. It is generally good to share more information rather than less. But to spend time focusing on, or directing attention towards less useful metrics can make the impact of an organization less clear, or even harmful.


Why might charities focus on the wrong metrics?

1) It is easier 
Some metrics are way easier to measure than others. Website traffic is a good example of a very easy to measure, positive metric, so it gets cited a lot as a criteria for success, even when it may not be connected to the charity's real goals. 

2) It looks more impressive
Reporting on several metrics looks more impressive because you are showing more data even if the data is not reflective of the good your charity is doing. 

Additionally the more different metrics you have the more you can clearly display the ones that are going well and play down the ones that are not going as well. This kind of practice makes your organization more appealing to donors and members, even if it is ultimately an illusion.

3) Charities are unsure what the important metrics are
This is true particularly with younger charities, as well as charities with less of a clear focus. When the goals are not clear, this will often result in reporting on several unhelpful metrics or missing very important ones.


Ways to avoid this mistake

1) Think of the number one most important metric
This is hugely important as it makes clear what your organization is really aiming to do, and how you will measure it. Letting the public know your most important metric also allows them to focus on what really matters. While we at the Charity Science publish lots of data, our most important metric is money moved to evidence-based charities Without a good amount of money moved, it does not really matter if our website gets tons of traffic, or if we put a lot of hours into our week to week work.

2) Be sure of the connection between your metric and to real good happening in the world
Even with a straightforward metric like money moved you have to make sure that it really translates into good getting done. With money moved you would primarily need to look at the charities you are moving money to, and make sure they will accomplish good with extra donation. For example, we only move money to charities that have rigorous research showing that they increase happiness or decrease suffering. For a more disconnected metric like website traffic you would have to make sure website traffic really correlates with what you really want to achieve. For example, we would have to have some evidence on how much money goes to effective charities from people seeing our website to see this as a very valuable metric. 

3) Think if it would be possible to cheat this metric
Is it possible to be “game” this metric, thus making it less valuable? For example, if I wanted Charity Science to gain a bunch of website traffic, it would be quite easy for me to invest in non targeted online ads or just directly buy “views” to my website. Although this would boost my website traffic its very unlikely to cause any real good on the metric I really care about. Money moved is a much harder number to cheat in this way (although it is still possibly by not taking into account counterfactuals).

4) Watch out for counterfactuals
An easy mistake to make is to measure metrics that have many possible causes. For example, if we used the total amount of money one of our top charities received as a metric of how much good we were doing this would be massively confounded by the many other ways our top charities receive donations. Given that many organizations are working towards the same goals, it is necessary to be able to isolate the impact your organization is having when compared to the wider movement.

Examples of good and bad metrics

To have a good metric, one must have a clear causality chain between your metric and good happening.

Consider us for example:
We give grants of unconditional cash transfers (UCTs) to the global poor → the global poor spend the money on what they desperately need → They are happier because they could afford a basic necessity or invest in their future → Good is achieved. We’re pretty confident that each link on this causal chain because there are multiple studies supporting each link. 

But now imagine that we argued for our own impact based on web traffic instead:
Charity Science gets website traffic → People are then more interested in donating to UCTs → They go to the UCT website to learn more → More money is donated → The UCT are spent on basic necessities or investments → People are happier → Good is achieved

Not only is this chain longer, but there is also a huge problem in the assumption that website traffic results in more donations to UCTs. While we can easily track website visits, it’s very difficult to track how many of these visits translates into more donations, and it’s easy for the metric to get cheated by getting large amounts of "lower quality" traffic. For example if we started writing blog posts about rationality or self-improvement to draw in a larger crowd. Generally the more steps you have the more confidence you need to have in each of the steps working. 

Examples of good metrics that are more clearly connected to doing good:
- Counterfactual money moved to proven effective charities 
- Reduced rate of illness- Number of people made happier
- Number of animal lives saved

Examples of less good metrics:
- Membership
- Website traffic
- Pledges
- Staff satisfaction
- Organizational growth
- More hours put into project

It’s worth putting in some time to set up solid tracking systems to measure your most important metrics. This makes organizational self improvement and having an impact much more likely, and more quantifiable when it does happen. Making these metrics public also ensures more accountability towards the most important metrics.