Summary

A convincing empirical study in animal rights needs the following:

1. Randomization of the control and experimental group.
   
2. Public pre-commitment to a strong analysis plan.
   
3. Transparency with original data.
   
4. A focus on metrics that matter.
   
5. An adequate sample size to detect a very small effect.
   
6. Be able to account for selection and social desirability bias

At Charity Science we’re super excited to hear that over the next 3 years there’ll be $1 million dollars committed to empirical studies within animal rights. This combined with the Open Philanthropy’s announcement that they’ve hired a Factory Farming Program Officer means that the empirical studies of animal rights issues are going to be receiving more money than ever before. This could actually fill the gap in the animal rights movement and take some of their interventions from “my gut feeling is that this works” to “I would bet my house that this works.” But before we proclaim that the end of speciesism is nigh, it’s worth acknowledging that past empirical studies within animal rights have been far from perfect. That’s why in this post we are outlining a checklist of things to do when doing a study in animal rights.

First let’s look at a 2012 study by the Humane League which focused on the effects leafleting had on diet choice. The Humane League is one of ACE’s Top Rated Charities and the Humane League Labs has led some pioneering research, but this study has some significant methodological flaws. That’s because to be able to truly tell the effect of an intervention it has to be compared to a control group - and not just any control group either - it has to be a randomized control group. This means that statistically significant differences between the control and the experimental group can be attributed to being caused by the intervention. In this study, participants were compared to the “average American meat-eater”. But what if the college students involved eat less meat to begin with and were in fact far from the average? That’s why the first item on our list is a randomized control group.

When analyzing the differences between the randomized control and the experimental group, pre-committing to what you’re analysing and how you’ll analyse it is a must. Otherwise humans have an incredible ability to ignore what they actually see in favor of (a) what they expect to see and (b) what they want to see.




For more on what should be included in a pre-analysis plan please see this. Without this pre-commitment studies are more likely to find a false positive.

By torturing data you can get it to say anything you want. If you don’t believe me then check out this paper providing a more concrete example. A pre-analysis plan helps make the widespread problem of p-hacking for statistically significant results much more difficult. Which may help to avoid issues like the possible problems with ACE’s Leafleting outreach study in 2013. I hope that you see why item 2 on the checklist is a detailed, specific and public pre-commitment analysis plan.

To further guard against questionable statistical analysis of the data, the raw data of the study should be made public. That way many eyes can look the data up and down. I really feel like there is no reason not to do this. We’re after what actually works, aren’t we? So the third item is be transparent and let the raw data loose upon the masses.  

The fourth item worth mentioning is to focus on metrics that matter. If we’re trying to find out how to help animals the most important metric is animal product consumption and after that perceptions of animals and perceptions of reducing animal product consumption. In fact the latter intermediate metrics can be quite helpful because it’s likely that a larger effect will happen which means we can be more confident detecting an effect there. Lastly, we want similar outcome metrics that can be used across studies to make the comparisons between different interventions very easy to do.

To be able to detect very small differences in these metrics between groups you will need a decent sample size. That might mean thousands or even tens of thousands of people in the study so that miniscule changes in the most important variables can be detected. For instance, with the classic probability level = 0.05 and desired power level = 0.8 and estimating that the proportion of vegetarians at follow up in the control group is 3% and 4% in the experiment group then the sample size will have to be 11,000 people in total. And that’s assuming that you managed to follow up with every single participant! So number 5 on our list is: Have a sample size big enough to give the statistical power to detect a very small effect on metrics. Before going on I should note that researchers should be careful of spillover effects which could contaminate the control group and make it much harder to detect the effect of the intervention. For instance, people that convert to veganism in the experiment group may easily influence and contaminate people that they know in the control group into changing their diet. This can be prevented by having several schools participated instead of several classes. That way the control and experiment group will almost no interaction.

Another thing to be very aware of are two strong biases that constantly come up in these studies. The first is a bias in who participates in the study and the second is a bias in the answers of those who do participate in the study.

The first bias is a selection bias and it comes in the selection of those who participate and the selection of those participants who the researchers follow up with. Basically if people know that a survey is about topic ‘x’, those who are unfamiliar with or uncomfortable with topic ‘x’ may not want to be included in the survey. Or if people know that someone is going to offer them a leaflet on something that they don’t care about, then they’re less likely to take that leaflet. 

The selection bias rears its ugly head in the follow up as well. If only 20% of participants are followed up with, they could be very unrepresentative of the entire sample. Often the low proportion of participants who are followed up with in these types of studies makes them particularly problematic. A great study would aim to have high response rates, ideally in the 80% range.


The second bias is a social desirability bias. This occurs when respondents clue in that a particular response is in some way preferred, or more socially desirable. Surprise, surprise - after finding this out they are more likely to give that response. A clear example of this would be if the surveyor was known to hand out flyers or was wearing a vegan t-shirt well giving out the survey.

Thus point 6 is: beware selection and social desirability bias. 







Just to recap, the keys to doing a good study in animal rights are:

1. Randomization of the control and experimental group.
2. Public pre-commitment to an analysis plan.
3. Transparency with original data.
4. A focus on metrics that matter.
5. An adequate sample size to detect a very small effect.
6. Be able to account for selection and social desirability bias

If these are followed than we’ll see the true beauty of empirical studies, and their results could take us from relying on personal experience, rules of thumb and intuition to having informed and updated opinions based upon the relevant data. As such, RCT studies are the most powerful approach that the Animal Rights movement can use to determine the means by which they can achieve their ends. For that reason, the animal rights movement can only benefit by informing their intervention selection with properly done RCTs. I can only drool at what the possible results could be. 

See full board meeting minutes here 

This is a survey of over a thousand fundraisers, and they disagree on virtually everything. It hovers around 50-50 on practically everything. Even the most agreed upon strategy, events, 1 in 5 fundraisers think isn’t effective. So if experts can’t be followed, then maybe we could follow the science? 

There’s Very Little Rigorous Science or Data Out There

Almost all of the science out there is observational. For example, the best stuff we could find about the ratios of money spent to money raised was based off of surveying organizations who participated in an expensive benchmarking program. This could have had a huge selection effect. What if only the large successful organizations participated in this program? Maybe they get better ratios than charities just starting out. 

This was good observational data though, relatively speaking. Most of the time it’s data like the graph above, where it’s just based on the opinions of fundraisers, saying stuff like, “I think this is effective”. Clearly not the most rigorous methodology. 

External Validity is Elusive

But even if there was rigorous methodology, that’s no guarantee of external validity. We are dealing with a crowd that is very different from the average population in a lot of ways. Take for example the famous study showing that people gave more if they saw a cute picture of a girl, rather than being told statistics. This is quite true of the average population for sure, but what about skeptics and intellectuals? There’s a good case to be made that they don’t want anecdotes but want the hard data. That’s what we’re all about after all. 

There was another rather rigorously run study on the effects of giving more options or less. Lots of studies have found that analysis paralysis seizes people if they’re presented with too many options. It’s better to have one call to action, rather than to give them some options. But The Life You Can Save just found that, when applied to donors, it’s better to give more options rather than less. So maybe that applies to choosing toothpastes, but doesn’t apply to charities? Or maybe it depends on the population? Or maybe it should be lots of options for small donations, but fewer for bigger ones, because then it becomes stressful? External validity is hard to find in the social sciences and fundraising doesn’t have enough people studying it to come to any sort of consensus. 

The Numbers Can Be Misleading

When you find numbers, it can be very exciting until you find out that the numbers only work under certain circumstances. Take for example legacy fundraising (asking people to put your charity in their will). It has the remarkable fundraising ratios of 20:1 or higher, whereas direct mail (sending letters to people about your charity) has much lower ratios, in the range of 1.2:1 . However, you can’t just jump right into legacy fundraising. It would quite presumptuous and untactful to ask somebody to put you in their will after only talking to them once. They have to be long-term supporters who really love your cause and your organization. That takes years of them knowing you and trusting you. Basically, you have to use things like direct mail first to have that donor base of people who might consider putting you in their will. 

Experts Don’t Know Why They’re Doing What They’re Doing

When we’ve asked people why they’re doing X fundraising strategy instead of Y, they have given us a puzzled look. They’d then say they didn’t know, it was hard to compare, or because that’s what their boss had told them to do. They didn’t seem to really know why. Why is that? Why don’t people know? That leads me to -

Almost Nobody Keeps Proper Track on Fundraising Metrics

Part of this is because they’re not doing what they should be doing, and part of it is because it’s really, really hard. Maybe even impossible. Fundraising is a lot like marketing, and marketing is very hard to measure. Say you have an ad on a bus about your charity. How do you measure the impact of that? You could add a dropdown menu during the donation phase and ask how they heard about it, but what if they heard about it from two different sources? What if they don’t remember? What if they’ve been reading your blog diligently, then it’s the ad that finally got them to donate? Counterfactuals muddy the waters even further. 

Over Half the Advice Out There is Empty Unsubstantiated Buzzwords

A huge percentage of the fundraising gurus out there say a whole lot of nothing. When we’ve interviewed consultants, they say a lot of stuff, but when we ask them “So, what exactly would I have to do today to enact this plan?” they something along the lines of, “Well, first you plan, then you do the plan” or some such empty sentence. Their advice is vague and unactionable. If you then ask why we should do their vague plans, they don’t have any well thought out reasoning. 

Why is it This Way?

We don’t know for sure why this is the case, but we can venture some guesses. One reason might be that it’s because fundraising is a social endeavor, and psychology has just barely become a proper science, and that’s debatable in the scientific community. (This is coming from people who have studied psychology too!) Psychology, being a new field, is particularly full of confusion and contradictory knowledge. Additionally, humans are the most complex thing studied scientifically, compared to chemistry or physics, so it might take a lot longer to make any progress. It could be because it’s not popular enough and doesn’t have enough money behind it to get real research. It could also be because of a myriad of other reasons that I am not aware of. 

The Light at the End of the Tunnel

It’s not all bad. We have found it comparatively easy to get relatively good information on some facets of fundraising, such as:

• Donor retention 
  
• How to do a specific thing well (e.g. how to improve social media). Just not how to compare between methods. 
  
• What not to do. For example, a small nonprofits should not do direct mail. (But everyone disagrees on what we should do)

The Solution(?)

It would appear that we cannot simply analyze the all available data already out there, then reliably pick the optimal path. We can do some minor research using the best information available, but ultimately we will just have to take best guesses and rapidly update depending on how things are going. We have tried the other way, and it led to a lot of analysis with very little or no progress. There’s just too little rigorous information out there to make a good decision that way. It’s better to simply try things after a shallow amount of research and do our best to see how well they go.

Of course, this has its own problems. For example, how long should we run the experiment? We don’t want to run the risk of staying on a project for too long when we’re wasting our time. On the other hand, we don’t want to give up too soon on a project that would have worked if we stuck with it. If Michael Jordan had given up after he got rejected from college basketball, that would have been a tragedy. (Well, a first world tragedy). Additionally, as with most skills, you aren’t amazing at it right away. There’s a learning curve and we don’t want to limit ourselves to only things that we are good at right away. 

We don’t have an answer to this question yet, but we are working on it. This still seems to be like the best solution given the information sparse environment we are in. 

See full board meeting minutes here.

See full board meeting minutes here