It’s that time of year again: time to dust off the costumes, carve the pumpkins, and select the candy of your choice. As we gear up for the one night a year when costumed children shout, “Trick or treat!” as loudly as possible, we hope you will also take a few minutes to stop and reflect on the sample plans that you have chosen for your research projects. Why is that? Because we know the fielding process can often get a little “batty.”
As researchers, we worry about questionnaires and obsess over data analysis, but too often the sample goes unnoticed. Most of the time, everything works out okay. However, when it doesn’t, researchers pay with poor data quality, unusable results, and sometimes a higher price tag than expected. We don’t want you to be haunted by poor or misguided sampling decisions, so remember to put as much care into your sample-plan decisions as the rest of the project. If you don’t, you might just be walking into a spider web of disappointment!
The Costly Nat Rep Zombie
Sometimes we can be so fixated on the “brains” of a nationally representative sample that projects will proceed with that plan even when it’s not the best option. The story of The Costly Nat Rep Zombie is an example of how to avoid hair-raising sample decisions.
In this case, our client was unsure of the demographics of their target audience. At the client’s request, we planned to screen a nationally representative sample and match all incoming sample to U.S. Census data. After reviewing the study objectives in more detail, we realized they were also interested in a few customer subsets. Therefore, we needed to set quotas by a variety of brand categories to get sufficient base sizes for target brands.
Our concern with their initial plan was that the client would not end up with enough respondents in all of their customer subsets, as some were very low incidence. Additionally, the amount of sample that would have to filter through the study to meet the survey quota requirements and remain nationally representative was gruesome, not to mention costly. We recommended that they use a two-step sampling process where we first collected the natural demographics on their customer base, setting screener quotas that matched the Census. Then, after we achieved a representative group, we switched to a targeted-brand approach, utilizing information about the respondents that had been collected previously.
Upon reviewing the results of the soft-launch data, the vast majority of the contacts in the nationally representative sample were people who either didn’t use the specific brand category or had not used it recently enough. As a result, we were churning through a cauldron of sample without generating the data the client ultimately needed. The client agreed to proceed with the two-step approach, using Census screener quotas first and then following with category quotas, saving money and gaining better data.
The next time you’re determined to run a nationally representative sample, we invite you to carve out a plan that addresses all of your data needs so you’re not spooked by the price tag!
The Invisible Respondent
You won’t be frightened by the data you’ve collected if you think first about the target audience from many different perspectives.
In our second example, a transportation agency commissioned a study that involved interviewing residents of a northeastern state. Their ultimate goal was to reach those interested in using a new commuting option. As we examined the screener with them, we learned that in the northeast, commuters could pass through three different states on their way to work! This meant that by targeting residents of only one state, we’d be missing feedback from those who lived elsewhere.
After discussing this reality with the client, we all wanted to ensure the sample frame would represent both resident and nonresident customers. We thought about people from several states who were using the highways or other transportation options in the area. We concluded that casting a wider sample net (knowing it might come with a lower incidence and higher costs) would be more effective for gathering the opinions of all potential users of the new transportation concept. We agreed on a min-and-max quota structure for state residents and for nonresidents, ensuring access to transportation users who lived in-state as well as those who were just passing through.
The online quantitative research was followed by a few in-person focus groups, where a subset of those who completed the online survey had an opportunity to express their views on the upcoming transportation project in more detail. The broader scope meant that the groups had to take place in a variety of locations, but, in the end, the data we collected was what the client needed.
So, this is just a reminder that while we may not see some potential respondents at first glance, that unseen group may be a crucial voice in the research and should be considered in the sample plan. It's up to everyone involved in the research (client, researchers, and sample providers) to both understand the objectives of the project and to ask the hard questions that lead to the right sample solution. Not asking those questions can lead to chilling results!
Do you get goose bumps when purchasing sample? Talk to the sampling experts at Symmetric today! At Symmetric, we take the time to understand even the scariest project objectives and help our clients to avoid monster mistakes.
For more information or to get a quote, please call 1-817-649-5243 or email [email protected]. 600 Avenue H East ● Arlington, TX 76011, USA ● www.symmetricsampling.com
VP of Symmetric, a Decision Analyst company