Documenting Bad Respondents to Improve Sample Quality

February 24, 2020

It happens in nearly every research project: some of the respondents just aren’t delivering deep, rich, well-thought-out responses. No matter what type of sample you use – panel, river, even customer lists – you run the chance of getting a certain number of poor respondents. Whether they made it through the screener and avoided your quality traps, aren’t paying attention or just aren’t giving thoughtful answers in your survey, you don’t want to include their data in your project – or pay for it! So, you ask your sample partner to remove them from the sample and to replace them with better-qualified respondents.

Sample providers and other research partners want to deliver good sample filled with attentive, qualified respondents. They replace the respondents, and you get better data and better insights.

But what happens to those bad respondents? It all depends on the sample provider. They might remain in the panel, with the potential of delivering poor quality responses on the next project, and the one after that, and so on. Wouldn’t it be better for everyone if researchers could identify and completely remove the bad respondents from their sample source by having more information on why the respondent was a poor complete?

Yes, of course! If sample providers fill every complete from their databases, they would be able to track which respondents were being removed most often, why, and eliminate them. That is not as easy to do these days as most samples require partnering between sample providers due to incidence rates continuing to drop on most surveys.

Technological advances allow corporate researchers to complete the “easier”, shorter, gen pop surveys themselves with DIY tools at times, which means that at times more difficult projects (low-incidence samples, longer surveys) are the ones going to research partners. Because of this, sample providers may end up partnering with other sample providers to fill those difficult quotas. Which is just fine to do as blending sources helps allieviate biases in the sample. Even if you think you are purchasing sample from one source that source may be getting assistance from their partners who they have vetted and trust because of the reasons above unless you state “proprietary sample only” to your provider.

However, partnering between panels creates complexity in communications about the reasons why a respondent is or is not acceptable to the client. This means that everyone – clients, partners, and providers – has to do a better job of communicating about these low-quality respondents to help make the sample universe a better place for all.

What’s the Solution? Timely Communications

Providers need to let clients know what will be needed to identify a poor-quality respondent and remove them from the sample, the bill, and, eventually, the panel. Ideally, it is most helpful if this communication happens early on in the project. Clients, on the other hand, need to document and communicate why poor-quality respondents are tagged as “poor,” with enough basic information that the panel providers can decide if the panel member should be permanently removed from the panel to help prevent this from happening again in the future.

Clients begin their analysis by examining their data, and in that process they identify bad respondents. At this point, if clients could simply document the reasons for respondents being unacceptable, that information could help sample providers. If sample providers and clients work together to communicate more detail about why respondents are unacceptable, panel providers can improve their panels by quickly dealing with bad respondents.

The Win-Win

We all have the same goal: good data from high-quality respondents. Providers want to improve their panel databases and deliver high-quality respondents. Most have no problem removing, replacing and not charging for bad respondents. Clients can help this by providing the basic information so providers can improve their performance. We can all be part of the solution!

Symmetric is committed to delivering high-quality samples. We even have an in-house department whose sole job is to review the survey data in detail and read and review all open-ends to take that burden off our clients. We want to be your partner in quality research. Contact us today!