How to make a survey say what you want

I just finished reading this article about a survey of people and their feelings about animal testing of cosmetics.  Unsurprisingly, a large majority of people oppose the testing of cosmetics on animals.  But what this article really demonstrates more than people’s opinions is a way in which you can structure your survey so that you get results that you want. Here is what they did.

Step 1 - Poison the well

They use the classic rhetorical device of giving some information that will outrage or shock just about anyone.

“Information from numerous animal protection organizations indicates that millions of rabbits, guinea pigs, and mice are killed every year to test cosmetics. The most common testing involves applying cosmetics to animals’ eyes and shaved skin to look for harmful effects. Testing can last days to weeks, at which time the animals are killed.

While the information they provide isn’t necessarily false it primes the respondent to be more apt to answer in a negative way towards any question about the subject of animal testing.  If the people who were responsible for the study really wanted to know what respondents thought, they would not have included this information.

Step 2 - Ask leading questions

The next step to getting a survey to come out the way you want it to is to ask questions which lead most reasonable people to the answer that you want.  You phrase the questions in such a way that it is actually uncomfortable to answer in ways that you don’t want them to answer.  For example, here are some of their questions…

2. For the next few statements please tell me if you strongly agree, somewhat agree, somewhat disagree or strongly disagree with each.

• A. Testing cosmetics or personal care products on animals is inhumane or unethical

Imagine how the responses would be different if they phrased the question like this…

• A. Selling untested cosmetics or personal care products is inhumane or unethical

Both of these questions are biased and lead the respondent to answer the way you want.  A proper survey would avoid leading questions.

Step 3 - Publicize the results highlighting what you want

The final step is to write a press release or blog post that highlights only what you want to get out of the study.  So their headline becomes “New Survey Shows 72 percent of Americans Oppose Testing Cosmetics Products on Animals” instead of “More Than a Third of all Americans Expect Companies to Test Cosmetics on Animals.”  Both of these headlines are true based on the data but they have markedly different effects on the reader.

What’s the point?

The biggest issue I have with this survey is that it does not meet the objectives of what the authors were trying to do.  They state their objective is to

“…gain insight into the public’s current knowledge of and opinions on animal testing for cosmetics–and to find out how many Americans are inclined to purchase cruelty-free cosmetics…”

But they didn’t find that information out at all.  Instead they found out something about the effectiveness of manipulating people with leading questions and extraneous “facts.”  This is too bad because I really do wonder what the general American population thinks about animal testing and cosmetics.  The best I could find is the Pew Research report that suggests 50% of Americans disagree with animal testing.