ESOMAR 28 Questions To Help Research Buyers Of Online Samples

1. What experience does your company have with providing online samples for market research?
   Symmetric developed its own mail panels in the 1980s and learned how to manage panel samples during that era. With the development of the Internet, we began to convert our mail panels to online panels in late 1995 and 1996 and started to aggressively recruit panelists for our online panels. Based on our experience and knowledge of telephone sampling, as well as mail sampling, we designed and programmed sophisticated sampling software, Icion ®, to pull representative samples from our panel databases. That software, first developed in 2000, has been continuously improved. Moreover, we have a whole department dedicated to online sampling. This department designs online samples, pulls those samples, executes quality- assurances processes, and monitors the actual deployment and use of the final sample.


2. Please describe and explain the type(s) of online sample sources from which you get respondents. Are there databases? Actively managed research panels? Direct marketing lists? Social networks? Web intercept (also known as river) samples?
   Symmetric’s own proprietary online panels are the primary sources for our online samples. These panels are:
   
   • American Consumer Opinion^®-Worldwide consumer panel in 11 different languages with over 3 million households representing over 7 million individuals.
   • Technology Advisory Board^®-Online panel of programmers/IT professionals and scientists containing 130,000 members worldwide.
   • Executive Advisory Board^®-Online panel of executives with 120,000 members worldwide.
   • Physicians Advisory Board^®-Online panel of 24,000 medical doctors in the U.S., Canada, and Europe.
   • Medical Advisory Board^®-Online panel of 43,000 nurses, dieticians, physicians' aides, and other nonphysician medical workers in the U.S., Canada, and Europe.
   • Contractor Advisory Board^®-Online panel of 25,000 building contractors, including electrical, painting, plumbing, carpentry, etc, primarily in the U.S. and Canada.
   • Imaginators^®-Online panel of 2,000 exceptionally creative individuals in the U.S., Canada, and Europe, used primarily for ideation and innovation projects.
   Panel members are continuously recruited to all panels by a combination of online and offline methods. The recruiting is designed to make the panel representative of the general adult population within each country. The major methods of recruiting include advertising on hundreds of websites, opt-in email lists, search engines, email newsletters and discussion lists, publicity and press releases, and print advertising. The recruiting methods are designed to reach a broad cross-section of people, using a wide variety of sources, appeals, and websites. The Panel Management Department actively monitors the composition of the panel and adjusts recruiting methods and targets as needed to keep the panel balanced by major demographic variables. Panel recruiting is a constant, ongoing activity.


3. If you provide more than one type of sample source, how are the different sample sources blended together to ensure validity? How can this be replicated over time to provide reliability? How do you deal with the possibility of duplication of respondents across sources?
   Symmetric owns and manages several large panels, but most projects do not require mixing of these sample sources. If a client is in need of physicians, we would just use our Physicians Advisory Council ® . If a client is in need of consumers, we would utilize our American Consumer Opinion ® panel. On the occasions where we need to mix panels, we would pull each sample separately and make sure it is representative and balanced through our sampling program, Icion ® , and check for any duplicates between the two at that time.
   
   MaxMind is a program embedded within Symmetric’s survey program, Logician ® , for quality- assurance checks of the respondents within each survey. MaxMind, is a digital fingerprinting technology that is designed to prevent survey respondents from taking the same online survey more than once. This feature “fingerprints” a computer using a large number of data points collected from the machine.
   
   This is accomplished through watermarking and fingerprinting.  A large number of computer characteristics from a respondent's computer are put through MaxMind and Symmetric’s proprietary algorithms to determine a unique digital fingerprint for that machine.  No personally identifiable information is collected; no information on the respondent's computer is altered.


4. Are your sample source(s) used solely for market research? If not, what other purposes are they used for?
   All Symmetric panel members are used solely for marketing research.


5. How do you source groups that may be hard-to-reach on the Internet?
   Symmetric’s online panels are so large that most low-incidence groups can be identified through screening. For groups underrepresented in online panels, such as low income households or minorities, mixed data-collection methods are often used. For example, an online panel survey is frequently supplemented by a mail survey, phone survey, mall-intercept, door-to- door, or recruit-to- central location interviews, depending on the country and needs of the project.


6. If, on a particular project, you need to supplement your sample(s) with sample(s) from other providers, how do you select those partners? Is it your policy to notify a client in advance when using a third party provider?
   Once it is determined that we need to use an outside provider, the client is notified immediately. On the vast majority of projects, we will know at the time of bidding on the project if we will need to bring in another provider based upon incidence provided by the client. Only Symmetric’s preferred sampling suppliers are used. Each supplier is carefully and fully evaluated before addition to the preferred supplier list. All samples are deduplicated using our digital fingerprinting software, RelevantID.


7. What steps do you take to achieve a representative sample of the target population?
   Once the target audience for a survey is determined, a sample is pulled to represent that audience. Samples are defined and pulled via Icion ® , our multivariate sampling software. Samples are balanced by geographic region, county size, and key demographics. Within each cell of the sample, respondent selection is random.
   
   Exclusion procedures are controlled by our Icion ® sampling software. Our standard policy is that once a panel member completes a screener, that member is not eligible to take another screener for four business days. Once a panel member completes a survey, they are precluded from taking another screener or survey for 14 days. Panel members are excluded from taking a second survey on the same topic for six months.
   
   The sample is launched and is slowly sent out throughout each day the survey is online. Reminder emails are sent to nonresponders automatically on the third day. Samples are controlled by geographic area and by time zone.
   
   Our software allows us to schedule and send sample batches at any time.


8. Do you employ a survey router?
   We do not own a router or use one as a standard practice on any of our projects.  We do have the ability to route, if the survey calls for it, through the Lucid routing system and routers employed by sample partners we use.  We alert clients in advance if we will use a router, but by default we do not use one.


9. If you use a router: Please describe the allocation process within your router. How do you decide which surveys might be considered for a respondent? On what priority basis are respondents allocated to surveys?
   Routing is on an individual project basis and discussed with the client if we need to utilize one.  Priority is given to the lowest-incidence project and/or the project with the most completes needed and/or a regional project with limited sample.


10. If you use a router: What measures do you take to guard against, or mitigate, any bias arising from employing a router? How do you measure and report any bias?
    A nationally representative sample is always sent out first prior to allowing respondents to route.  At the time we decide to use a router most projects are almost complete and we are filling in a few hard-to- reach quotas.


11. If you use a router: Who in your company sets the parameters of the router? Is it a dedicated team or individual project managers?
    Our Sampling Manager sets the parameters and consults with our clients before proceeding forward.


12. What profiling data is held on respondents? How is it done? How does this differ across sample sources? How is it kept up-to-date? If no relevant profiling data is held, how are low incidence projects dealt with?
    Each of Symmetric’s different panels collects different profile information since each panel differs in audience and purpose. American Consumer Opinion ® , our consumer panel (www.acop.com), for example, collects name, address, email, phone number, gender, marital status, children in household, ages of children, occupation, spouse’s occupation, education, spouse’s education, dwelling type, dwelling ownership, language spoken in the home, ethnicity, spouse’s ethnicity, and income. Product and service usage are also recorded for hundreds of product/service categories via quarterly screenings.
    
    Executive Advisory Board®, as another example, collects information on industry, company size, number of employees, job responsibilities, and other firmographic data. Likewise, contractor, technology, physician, and medical panels collect profile information specifically tailored to those audiences.
    
    Panel members have complete control of their membership accounts and can log in at any time to update, modify, or delete their information. After completing surveys, panelists are asked to review and update their membership accounts. Additionally, once a year all members of each panel are asked to update their membership information. Finally, as we do large panelwide screeners that data is stored so we can pull sample from it also.


13. Please describe your survey invitation process. What is the proposition that people are offered to take part in individual surveys? What information about the project itself is given in the process? Apart from direct invitations to specific surveys (or to a router), what other means of invitation to surveys are respondents exposed to? You should note that not all invitations to participate take the form of emails.
    After Icion ® selects the respondents for a survey, the Panel Management Department emails those selected respondents the survey URL. The email contains some very basic information about the survey, such as the incentive amount, time required to complete the survey, how long the survey is available for, the URL to the survey, and an email address and phone number to contact Symmetric, in case of problems or questions.
    
    Visually, the emails are in an HTML format for those panel members who can receive HTML emails, or in a plain text format for others. The HTML emails have panel logos and colors throughout.


14. Please describe the incentives that respondents are offered for taking part in your surveys. How does this differ by sample source, by interview length, by respondent characteristics?
    Our panel members are paid in Points for each screener they take. A Point is equal to a penny. The longer the screener, the more points they get paid. Same applies to our surveys. Panel members are paid in Points for each survey they complete, and the longer the survey, the more Points. Screeners are only allowed to be a few minutes in length and respondents are paid between 10 and 30 cents for each screener. Surveys can range from 5 minutes to 30 minutes or longer and the incentive there depends on which panel a respondent is on and how long the survey is. For our consumer panel, survey incentives range from $1 to $25 or more. For our physician panel, incentives are usually in the $75- $250 or more range. Executive or IT panels are in the $5-$100 or more range.


15. What information about a project do you need in order to give an accurate estimate of feasibility using your own resources?
    In order to give an accurate estimate we would need to know the total number of completes needed, qualification requirements, any quotas that the client may have, length of interview, incidence of the survey, and timing requirements.


16. Do you measure respondent satisfaction? Is this information made available to clients?
    Yes. This is done on various surveys so that we are aware of the quality of our survey design process. The client can request to review the panelists' responses.


17. What information do you provide to debrief your client after the project has finished?
    It depends on the type of project, the subject of the project, and the specifications outlined in the proposal.


18. Who is responsible for data quality checks? If it is you, do you have in place procedures to reduce or eliminate undesired within survey behaviors, such as (a) random responding, (b) Illogical or inconsistent responding, (c) overuse of item non-response (e.g. "Don't Know") or (d) speeding (too rapid survey completion)? Please describe these procedures.
    There are many layers to our fraud-prevention efforts.
    
    First, we have a dedicated Quality Assurance Department that checks 100% of all work that is produced. In addition to QCing all surveys before they are launched, they review topline data, open-ends, and tabulation data looking for any errors or discrepancies in the data.
    
    In addition to that, our panels are rigorously managed and continuously cleaned by another department of people who are searching for poor registration data, duplicate registrations, false information during registration, etc. A database of “cheaters” is maintained, so that these individuals can be prevented from registering or reregistering to become a member of any panel.
    
    Our screening questionnaires and survey questionnaires contain “traps” to catch cheaters and sloppy respondents, who are promptly deleted from our panels. Our panel database department reviews all survey completes on a daily basis looking for speedsters, straightliners, etc, and all respondents that have fallen into our cheater traps; and the department determines if respondents are good or bad. Ones that are determined to have cheated or sped through the survey are removed from the survey data and from our panel.
    
    As open-ended questions in surveys are coded by our Coding Analytics Department, respondents who appear to be cheating and/or answering questions in a haphazard manner are deleted. Additionally, during the tabulation process, a series of quality-assurance processes are employed to identify questionable responses. Any cheaters are deleted from the survey and from our panels. Lastly, when checks are mailed to participating panelists, the “payee” list for each study is printed out in alphabetical order and carefully checked by a Symmetric employee (this is our final quality-assurance step).
    
    MaxMind is a program embedded within Symmetric’s survey program, Logician ® , for quality- assurance checks of the respondents with each survey. MaxMind is a digital fingerprinting technology that is designed to prevent survey respondents from taking the same online survey more than once. This feature “fingerprints” a computer using a large number of data points collected from the machine.
    
    This is accomplished through watermarking and fingerprinting.  A large number of computer characteristics from a respondent's computer are put through MaxMind and Symmetric’s proprietary algorithms to determine a unique digital fingerprint for that machine.  No personally identifiable information is collected; no information on the respondent's computer is altered.


19. How often can the same individual be contacted to take part in a survey within a specified period whether they respond ot the contact or not? How does this vary across your sample sources?
    A typical online panelist receives three to four invitations a month to participate in screeners. The average panelist is selected for and completes five to six surveys per year. Limits are placed on screeners and surveys. Our standard policy is that once a panel member completes a screener, that member is not eligible to take another screener for four business days. Once a panel member completes a survey, they are excluded from taking another screener or survey for 14 days. Panel members are excluded from taking a second survey on the same topic for six months. This is the same for all of our panels.


20. How often can the same individual take part in a survey within a specified period? How does this vary across your sample sources? How do you manage this within categories and/or time periods?
    Once a panel member completes a survey, they are precluded from taking another screener or survey for 14 days. Panel members are excluded from taking a second survey on the same topic for six months. This is the same for all of our panels.


21. Do you maintain individual level data such as recent participation history, date of entry, source, etc., on your survey respondents? Are you able to supply your client with a project analysis of such individual level data?
    Yes, detailed records are maintained on each panelist's registration and survey participation history.


22. Do you have a confirmation of respondent identify procedure? Do you have procedures to detect fraudulent respondents? Please describe these procedures as they are implemented at sample source registration and/or at the point of entry to a survey or router. If you offer B2B samples what are the procedures there, if any?
    Yes, at the time of registration we collect and validate email address, mailing address, and company name and phone number if it applies to one of our specialty panels.
    
    RelevantID is also used at the entry point of each survey to prevent survey respondents from taking the same online survey more than once. A large number of computer characteristics from a respondent's computer are put through MaxMind and Symmetric’s proprietary algorithms to determine a unique digital fingerprint for that machine.


23. Please describe the 'opt-in for market research' processes for all your online sample sources.
    Our panels were built for marketing research purposes only and that is clearly explained to everyone before joining our panels. In addition, all of our panels are double opt-in. Prospective panel members come to our website and fill out a registration page, which ranges from 20-30 questions. Once they fill out the registration page, a confirmation link is sent to their email addresses. They must click on the confirmation link in the email. This completes the double opt-in process. Only double opt-in members are permitted in our panels.


24. Please provide a link to your Privacy Policy. How is your Privacy Policy provided to your respondents?
    Each panel has its own privacy policy. Each panel's privacy policy is located on the panel's home page and within the disclaimer of all emails each member receives.
    
    http://www.acop.com/privacy/


25. Please describe the measures you take to ensure data protection and data security.
    Symmetric has publicly accessible web and database servers that are located in a secure facility, which has redundant high-speed Internet access from multiple Tier 1 network service providers. This facility provides 24/7 monitoring and maintenance of the network and server hardware and alerts our technicians in case a problem develops. A service level agreement guarantees 99.99% availability of the network and no more than a two hour turnaround in the event of hardware replacement. The servers are dedicated to Symmetric and housed in high- availability chassis with redundant components such as RAID 5 hard-drive arrays and load balanced to further ensure high availability. The servers are equipped with tape backup to safeguard collected data and a firewall to guard against malicious attacks.


26. What practices do you follow to decide whether online research should be used to present commercially sensitive client data or materials to survey respondents?
    This is one of the first conversations our Client Service staff will discuss with the client when gathering specifications to the project. We will present the client with the pros and cons and work together to make the best decision in regards to that specific project. The fact remains that there are no foolproof methods for protecting audio, video, still images, or concept descriptions in online surveys. In today’s social media world, clients should be aware that the combination of technology solutions and respondent confidentiality agreements are “speedbumps” that mitigate but cannot guarantee that a client’s stimuli will not be shared or described in social media. We utilize the most up-to- date image encryption software. We will place legally binding agreements at the beginning of the survey that our respondents must agree to in order to continue, and we will prosecute to the full extent of the law if they should break that agreement. We use the highest level of data encryption to protect all survey data. But the fact remains that once something is put online, it is never 100% safe, and any company claiming it is 100% safe is not being honest with you.


27. Are you certified to any specific quality system? If so, which one(s)?
    We adhere to the ESOMAR Standards. We are approved under the Safe Harbor Agreement with the European Union. We adhere to extensive and rigid internal quality standards.


28. Do you conduct online surveys with children and young people? If so, do you adhere to the standards that ESOMAR provides? What other rules or standards, for example COPPA in the United States, do you comply with?
    We were the first U.S. marketing research company to be EU Safe Harbor-compliant and continue to be so to this day. We follow ARF and ESOMAR guidelines. In compliance with the Children’s Online Privacy Protection Act, we do not accept registrations from children under age 14. Where rules or guidelines between the organizations differ, we choose the most restrictive set of rules.

White Papers

Over the past decade, many corporations have set up private online panels or online communities as an economical way to conduct surveys and qualitative projects. Read More

The front end of online data collection relies heavily on the honesty of respondents. Read More

Brochures and Additional Information (Click to Download):

Symmetric Brochure
Panel Book
Panel Management and Data Quality
Worldwide Online Panels
Worldwide B2B Panels
Worldwide ACOP Panel
Worldwide Medical Panels
ESOMAR 28 Questions
Sample-Size Determination
Free Satistics Software (Links to Decision Analyst website)
Back to top