How to Interpret Empirical Studies: Four Lessons from Political Polling

Political polls provide a good setting for a discussion of empirical research. They seem simple and are often in the headlines so we’re familiar with them. Also, we don’t always have an accessible compendium of all the studies on the same topic, but it’s pretty easy to find polls in a presidential race during the same time period.  So polls are a convenient example of empirical research.

You might expect that all the polls would agree if they’re conducted by reputable professionals.  But actually, that isn’t true — and the reasons why it’s false are instructive, not just about polls but about other kinds of empirical studies.

Consider the following table of results from political polls four years ago:

Most of these polls are well-known enough and  reliable enough that they are frequently cited  by political observers and the press.  Yet the results are all over the place, ranging from a nine-point lead for Obama to a two-point loss.

What can we learning about this, in terms of empirical studies on topics such as the economic impact of environmental regulation or the  link between a pollutant and illness?

Here are four main implications that are relevant to  environmental studies:

1. Any one study has limited value in the absence of confirming evidence.  Keep an open mind and try to consider the weight of the evidence, not just a single study that may have caught your attention.  This can be hard to do when a study confirms your expectations!
2. The source of a study matters.  Among the pollsters in the table, some are especially respected or have a really good track record for accurately predicting results.  Others not quite so much.  Also, Rasmussen has a “house effect” favoring Republicans — that is, they tend to come up with more favorable GOP numbers than other polls. (Other pollsters have the opposite house effects, but I’m not aware of any in this particular table.) That’s significant in interpreting their findings.
3. Pay attention to confidence intervals and margins of error.  For instance, Rasmussen shows a tie, with a 2 point margin of error.  So even if the poll is perfect, either candidate could be up 2 or down 2 from the 48%.  Thus, it’s possible that the race is 50-46% either way and that one candidate has a significant lead that the poll couldn’t reliably detect.  Knowing the margin of error can also be important in interpreting a study, too.
4. Consider models embedded in the study.  Notice that most of the polls are about likely voters (LV).  Each pollster needs some model to decide what voters are likely to vote, and the models aren’t necessarily the same.  So what looks like raw, unvarnished data actually has some assumptions about voting turnout built in.  This can happen with studies, too.

In short:

Do pay attention to empirical studies — how else can we find out about the world?

Don’t latch onto the latest study regardless of source, or assume that there’s no noise factor in the results or that the study is free of assumptions.