Survey error is one of the most challenging topics for marketing researchers. This is true because many of the errors cannot be measured statistically, and because many of them are the result of our inherent biases and predispositions. Sampling error is one of the most widely cited sources of survey error, probably because it cannot be estimated quantitatively. Non-response error is another frequently discussed topic. There are additional types of error that are less commonly found in reports, including universe definition error, sample screening error, and transition error. (If you are not familiar with these, click here for clarification.) All of these can severely impact the accuracy of your survey results and may lead to invalid conclusions. But survey error doesn’t only apply to marketing research reports.

As the old saying goes, “There are three kinds of lies: lies, damn lies, and statistics.” With the upcoming 2020 presidential election, it is very tempting to add a fourth kind of lie to the list: polls. Americans are awash in poll results. Polls, because they are developed and reported in an adversarial environment, are likely candidates for survey error and misleading or conflicting results. As marketers, we may be more aware of using statistics and understanding their vulnerabilities, but many people are not. Therefore, it is an excellent time to remind everyone how to read polls. This blog highlights some of the most critical errors that are rampant in polling data.

The Purpose

First, let’s talk about the purpose of polls. Why does anyone invest in polling data? Polling data exists for two reasons. First, to persuade voters. Whether the sponsor (pollster, candidate, PAC, etc.) is trying to convince someone to stay with their current candidate or to change their mind, polls can be persuasive and useful. Second, sponsors may be trying to raise funds (which, admittedly, is another type of persuasion). The candidate, political party, or PAC may be fundraising for their cause. So, right in the purpose of the poll, you may have a source of bias. To understand the value of the data, you need to understand who is conducting the poll, for whom, and what their track record is in conducting reliable polls that match the eventual election result.

The Timing

Second, let’s talk about timing. Poll results are more accurate when they are done closer to the actual election. So polling data collected in January 2020 is less accurate than data collected in late October 2020. Check to see when the data was collected before you decide how to interpret poll results.

The Participants

Third, who was sampled? Did the sample consist of registered voters, voters who participated in the last presidential election, voters who participated in the last local election, or anyone over 18? Was it a state, regional, or national sample? What was the representation of Republicans versus Democrats versus Independents? How was the resulting data weighted, if at all? How did the sample and the results compare to the population of interest? Understanding the composition of the sample and the participants can help you understand potential bias and temper your reaction to the results.

The Questions

Fourth, how were the questions asked in the poll? We all know that question wording can influence how people respond to survey questions. But question order and format are also at issue. As reported by the Brookings Institute, “Even when people have strong views, a single polling question rarely captures those views well. Human beings are complicated, and so are their opinions.” Understanding question wording, order and format can help you understand why the results of different polls vary so widely.

The Errors

Fifth, let’s talk about what we might call “human error.” People are fickle and change their minds. Until they are in the voting booth (or post their mail-in ballot), it is impossible to know how many of them will vote, particularly in very close elections. As Samuel Popkin writes in The Reasoning Voter, this uncertainty arises because “Ambivalence is simply an immutable fact of life.” That does not mean there is anything wrong with the public. Voters, being human, may be more involved with some issues and less engaged with others. And most voters are not continuously, deeply involved in public affairs. It just means as you look at polling data, keep in mind that it is a picture of a point in time, it is not reality.

We are not picking on polls because marketing research is better. As we have already stated, these (and more!) sources of bias and error impact all surveys. However, during elections – and especially presidential elections – when so many resources are being poured into polling and those poll results then impact much of the information we consume which may impact how we cast our votes, it is crucial to put polling data in perspective.

Polls provide valuable information. However, just as we would not make a financial investment decision based on one data point, neither should we rely blindly on polls for all of our political decisions.

Wondering what other types of survey error may be sabotaging your results? Download the definitive Survey Error sourcebook here!