A very learned marketing professor once told me that there is no need to use anything other than online surveys for marketing research. Well, learned professor, we beg to differ. Survey methodology should be chosen based on the specific requirements of each survey project, the available sampling frame, the timeline, and the budget. When you look at all of those variables, you realize there is more to life than online surveys.
To choose the right survey methodology for your project, we propose evaluating each methodology on three categories of factors:
- First are the Task Factors. These are the specific requirements of your project that different methodologies can meet to varying degrees. Think about the diversity of questions the methodology can accommodate. Not all questions can be asked effectively in all settings, and not all tools are flexible enough to ask all types of questions. Some methodologies are better or worse for different kinds of questions (think grid questions in mobile surveys). Consider whether you will need respondents to evaluate stimuli, and what form that stimuli might take. It is very difficult (or impossible) to present graphic, text, or video stimuli to phone interview respondents, so that would be a major drawback to choosing that methodology. The quantity of data needed is also a concern (especially when combined with budgetary limitations). Sometimes, it is just not feasible to conduct as many in-home interviews as you need for a given project. Finally, response rate, which is a concern in all methodologies, should be evaluated against your intended analysis.
- Second are the Situational Factors. Does the chosen methodology give you enough control over the data collection environment to accommodate your project? Doing an in-home interview with a busy mom with three kids running around may not be optimal for her to focus on the task. Additionally, do you have sufficient control of your field data-collection force? It is critical that each survey is administered consistently to control bias. You should also consider the potential for interviewers to inject their own bias into data collection as well. Finally, which methodology allows you to bring the project in on time and on budget? Social media surveys are notoriously fast and inexpensive, but are these the right respondents and the right question formats for your project?
- Third, are the Respondent Factors. Do the respondents feel comfortable with the methodology? Do the respondents believe that their data will remain private and anonymous, or do they perceive a threat of their identity being compromised? Is the methodology likely to elicit socially desirable responses? That is, will respondents give the responses they believe to be more socially or politically acceptable than their “true” answer? How comfortable will the respondents be in giving honest responses to questions seeking personal or sensitive information? Low-incidence populations are another consideration. If you are trying to sample very unique or rare respondents or to survey a small geographic area, you may be limited in the methodology you can choose by the sampling frame that is available. Finally, consider how much control for respondents is appropriate for your survey. You might like to give respondents a very long data-collection period to accommodate their busy schedule, but your boss wants the answers yesterday.
There is no perfect all-around methodology for any project. You will have to balance the good with the bad to optimize your survey research. There may be times when you may need to shore up your methodological choice to overcome its weaknesses for your project. Mobile optimization, incentives, reminders, and slider scales are all tools researchers use to optimize respondent participation. The important thing is to understand how your methodology works, how it might impact your project both overall and in data collection in particular, and how those impacts can be taken into consideration as you complete analysis and reporting.
Not sure how methodologies rate on these factors? To see how Symmetric evaluates different survey methodologies, download our summary sheet!