How to do Studies that will switch EAs from Poverty to Animal Rights

Summary

A convincing empirical study in animal rights needs the following:

1. Randomization of the control and experimental group.
   
2. Public pre-commitment to a strong analysis plan.
   
3. Transparency with original data.
   
4. A focus on metrics that matter.
   
5. An adequate sample size to detect a very small effect.
   
6. Be able to account for selection and social desirability bias

At Charity Science we’re super excited to hear that over the next 3 years there’ll be $1 million dollars committed to empirical studies within animal rights. This combined with the Open Philanthropy’s announcement that they’ve hired a Factory Farming Program Officer means that the empirical studies of animal rights issues are going to be receiving more money than ever before. This could actually fill the gap in the animal rights movement and take some of their interventions from “my gut feeling is that this works” to “I would bet my house that this works.” But before we proclaim that the end of speciesism is nigh, it’s worth acknowledging that past empirical studies within animal rights have been far from perfect. That’s why in this post we are outlining a checklist of things to do when doing a study in animal rights.

First let’s look at a 2012 study by the Humane League which focused on the effects leafleting had on diet choice. The Humane League is one of ACE’s Top Rated Charities and the Humane League Labs has led some pioneering research, but this study has some significant methodological flaws. That’s because to be able to truly tell the effect of an intervention it has to be compared to a control group - and not just any control group either - it has to be a randomized control group. This means that statistically significant differences between the control and the experimental group can be attributed to being caused by the intervention. In this study, participants were compared to the “average American meat-eater”. But what if the college students involved eat less meat to begin with and were in fact far from the average? That’s why the first item on our list is a randomized control group.

When analyzing the differences between the randomized control and the experimental group, pre-committing to what you’re analysing and how you’ll analyse it is a must. Otherwise humans have an incredible ability to ignore what they actually see in favor of (a) what they expect to see and (b) what they want to see.




For more on what should be included in a pre-analysis plan please see this. Without this pre-commitment studies are more likely to find a false positive.

By torturing data you can get it to say anything you want. If you don’t believe me then check out this paper providing a more concrete example. A pre-analysis plan helps make the widespread problem of p-hacking for statistically significant results much more difficult. Which may help to avoid issues like the possible problems with ACE’s Leafleting outreach study in 2013. I hope that you see why item 2 on the checklist is a detailed, specific and public pre-commitment analysis plan.

To further guard against questionable statistical analysis of the data, the raw data of the study should be made public. That way many eyes can look the data up and down. I really feel like there is no reason not to do this. We’re after what actually works, aren’t we? So the third item is be transparent and let the raw data loose upon the masses.  

The fourth item worth mentioning is to focus on metrics that matter. If we’re trying to find out how to help animals the most important metric is animal product consumption and after that perceptions of animals and perceptions of reducing animal product consumption. In fact the latter intermediate metrics can be quite helpful because it’s likely that a larger effect will happen which means we can be more confident detecting an effect there. Lastly, we want similar outcome metrics that can be used across studies to make the comparisons between different interventions very easy to do.

To be able to detect very small differences in these metrics between groups you will need a decent sample size. That might mean thousands or even tens of thousands of people in the study so that miniscule changes in the most important variables can be detected. For instance, with the classic probability level = 0.05 and desired power level = 0.8 and estimating that the proportion of vegetarians at follow up in the control group is 3% and 4% in the experiment group then the sample size will have to be 11,000 people in total. And that’s assuming that you managed to follow up with every single participant! So number 5 on our list is: Have a sample size big enough to give the statistical power to detect a very small effect on metrics. Before going on I should note that researchers should be careful of spillover effects which could contaminate the control group and make it much harder to detect the effect of the intervention. For instance, people that convert to veganism in the experiment group may easily influence and contaminate people that they know in the control group into changing their diet. This can be prevented by having several schools participated instead of several classes. That way the control and experiment group will almost no interaction.

Another thing to be very aware of are two strong biases that constantly come up in these studies. The first is a bias in who participates in the study and the second is a bias in the answers of those who do participate in the study.

The first bias is a selection bias and it comes in the selection of those who participate and the selection of those participants who the researchers follow up with. Basically if people know that a survey is about topic ‘x’, those who are unfamiliar with or uncomfortable with topic ‘x’ may not want to be included in the survey. Or if people know that someone is going to offer them a leaflet on something that they don’t care about, then they’re less likely to take that leaflet. 

---

The selection bias rears its ugly head in the follow up as well. If only 20% of participants are followed up with, they could be very unrepresentative of the entire sample. Often the low proportion of participants who are followed up with in these types of studies makes them particularly problematic. A great study would aim to have high response rates, ideally in the 80% range.


The second bias is a social desirability bias. This occurs when respondents clue in that a particular response is in some way preferred, or more socially desirable. Surprise, surprise - after finding this out they are more likely to give that response. A clear example of this would be if the surveyor was known to hand out flyers or was wearing a vegan t-shirt well giving out the survey.

Thus point 6 is: beware selection and social desirability bias. 







Just to recap, the keys to doing a good study in animal rights are:

1. Randomization of the control and experimental group.
2. Public pre-commitment to an analysis plan.
3. Transparency with original data.
4. A focus on metrics that matter.
5. An adequate sample size to detect a very small effect.
6. Be able to account for selection and social desirability bias

If these are followed than we’ll see the true beauty of empirical studies, and their results could take us from relying on personal experience, rules of thumb and intuition to having informed and updated opinions based upon the relevant data. As such, RCT studies are the most powerful approach that the Animal Rights movement can use to determine the means by which they can achieve their ends. For that reason, the animal rights movement can only benefit by informing their intervention selection with properly done RCTs. I can only drool at what the possible results could be. 

---