No Tricks, Just Treats

October 1, 2021 | Becky Crane

 

It's that time of year again: time to dust off the costumes, carve the pumpkins, and select the candy of your choice. As we gear up for the one night a year when costumed children shout, "Trick or treat!" as loudly as possible, we hope you will also take a few minutes to stop and reflect on the sample plans you have chosen for your research projects. Why is that? Because the fielding process can often get a little "batty."

As researchers, we worry about questionnaires and obsess over data analysis, but the sampling plan often does not get the same attention. Most of the time, everything works out okay. However, when it doesn't, researchers pay the price with poor data quality, unusable results, and sometimes a higher price tag than expected. We don't want you to be haunted by poor or misguided sampling decisions, so remember to put as much thought into your sample-plan decisions as the rest of the project. If you don't, you might just be walking into a spider's web of disappointment!

The Costly Nat Rep Zombie

Sometimes we can be so fixated on the "brains" of a nationally representative sample that projects will proceed with that plan even when it's not the best or most appropriate option. The story of The Costly Nat Rep Zombie is an example of how to avoid hair-raising sample decisions.

In this case, our client was unsure of the demographics of their target audience. We planned to screen a nationally representative sample and match all incoming responses to U.S. Census data at the client's request. After reviewing the study objectives in more detail, we realized they were also interested in a few customer subsets. Therefore, we needed to set quotas by a variety of brand categories to get sufficient base sizes for target brands.

Our concern with their initial plan was that the client would not end up with enough respondents in all of their customer subsets, as some were very low incidence. Additionally, the amount of sample that we would have to filter through the study to meet the survey quota requirements and remain nationally representative was gruesome, not to mention costly. We recommended using a two-step sampling process where we first collected the natural demographics on their customer base, setting screener quotas that matched the Census. Then, after we achieved a representative group, we switched to a targeted-brand approach, utilizing previously collected information about the respondents.

Upon reviewing the results of the soft-launch data, the vast majority of the contacts in the nationally representative sample were people who either didn't use the specific brand category or had not used it recently enough. As a result, we were churning through a cauldron of sample without generating the data the client ultimately needed. The client agreed to proceed with the two-step approach, using Census screener quotas first and then following with category quotas, saving money and gaining better data.

The next time you're determined to run a nationally representative sample, we invite you to carve out a plan that addresses all of your data needs, so the price tag does not spook you!

The Invisible Respondent

You won't be frightened by the data you've collected if you think first about the target audience from many different perspectives.

In our second example, a transportation agency commissioned a study that involved interviewing residents of a northeastern state. Their ultimate goal was to reach those interested in using a new commuting option. As we examined the screener with them, we learned that in the northeast, commuters could pass through three different states on their way to work! This meant that by targeting residents of only one state, we'd be missing feedback from those who lived elsewhere.

After discussing this reality with the client, we all wanted to ensure the sample frame would represent both resident and nonresident customers. We thought about people from several states who were using the highways or other transportation options in the area. We concluded that casting a wider sample net (knowing it might come with a lower incidence and higher costs) would be more effective for gathering the opinions of all potential users of the new transportation concept. We agreed on a min-and-max quota structure for state residents and for nonresidents, ensuring access to transportation users who lived in-state as well as those who were just passing through.

The online quantitative research was followed by a few in-person focus groups, where a subset of those who completed the online survey had an opportunity to express their views on the upcoming transportation project in more detail. The broader scope meant that the groups had to take place in a variety of locations, but, in the end, the data we collected was what the client needed.

While we may not see some potential respondents at first glance, that invisible group may be crucial in the research, and we should include them in the sample plan. It's up to everyone involved in the research (client, researchers, and sample providers) to understand the project's objectives and ask the hard questions that lead to the right sample solution. Not asking those questions can lead to chilling results!

Do you get goosebumps when purchasing sample? Talk to the sampling experts at Symmetric today! At Symmetric, we take the time to understand even the scariest project objectives and help our clients avoid monster mistakes.

For more information or to get a quote, please call 1-817-649-5243 or email info@symmetricsampling.com. 

Becky Crane

Chief Client Success Officer of Symmetric