ESOMAR 37 Questions To Help Research Buyers Of Online Samples

1. What experience does your company have with providing online samples for market research?
   
   Symmetric developed it’s own mail panels in the 1980s and learned how to manage panels during that era. Over time we moved from only a mail panel to mail and phone and with the development of the Internet, we began to convert our panels to online panels in late 1995 and 1996 and started to aggressively recruit panelists for our online panels. We were one of the first companies in the industry to have online panels and execute online studies. Based on our experience and knowledge of telephone sampling, as well as mail sampling, we designed and programmed a sophisticated sampling software, Icion®, to pull representative samples from our panel databases and to blend in external sample sources into our sampling platform. That software, first developed in 2000, has been continuously improved.




2. Do you have staff with responsibility for developing and monitoring the performance of the sampling algorithms and related automated functions who also have knowledge and experience in this area? What sort of training in sampling techniques do you provide to your frontline staff?
   
   Yes, we have a dedicated department for just sampling along with sister departments for quality assurance and panel recruiting for our online samples. Our staff has an average of 10 years’ experience in this area. In addition to the weekly training meetings and collaboration with other large sample companies in the industry, we have been attending industry workshops for decades and participate in conferences as speakers about the sampling industry and techniques. Our approach to sampling is scientific with the systems we have built and truly understand the importance of proper sampling.




3. What other services do you offer? Do you cover sample-only, or do you offer a broad range of data collection and analysis services?
   
   In addition to sample (whether that be online, offline, mail, in-person, qual or quant) we offer a suite of services that our under our umbrella and owned and operated by us in our headquarters in the Dallas/Ft. Worth area. That includes: Survey programming, mail department for iHuts or mail samples, open end coding, qualitative research team, tabulation, advanced analytics/choice modeling, questionnaire development and analysis, and community/custom panel builds to name a few.




4. 4. Using the broad classifications above, from what sources of online sample do you derive participants?
   
   We fall into the panel definition as we have our own proprietary source which is triple-opt-in (sign up, double opt in by email validation, and then they complete a long quality assurance survey the following day filled with traps and open ends to verify they are a good, valuable panel member) before we allow them to join. Our panels are recruited as Market Research Panels- that is that people are only signing up to take surveys and participate in market research projects and to voice their opinion. This is a rarity these days in the industry as most sources are built where survey taking is a secondary focus. This process and level of steps needed to join our panel go far above and beyond what most companies do in the industry and help keep our average tenure of panel members to over 4 years. We also specialize in blending outside samples with our own whether that is direct relationships built with other high-quality providers and/or through sample platforms.




5. Which of these sources are proprietary or exclusive and what is the percent share of each in the total sample provided to a buyer?
   
   We have 6 proprietary panel sources for sample that have been around for over 20 years:
   • International Consumer panel in 11 languages and over 150 countries
   • A worldwide Technology panel
   • A worldwide Executive panel
   • A worldwide Medical panel
   • A USA Physician panel
   • A USA Contractor panel
   The sample being used depends on the specs and needs provided from the client. We can use just our online proprietary panels or we can blend our panel with other sources to fill a study. We will always disclose to a client if we are using proprietary only or a mix of our panels and trusted partners that we have personally vetted and passed all our quality process. We are firm believers in blending samples to help erase biases that all sources have so we can be a one stop shop for clients. Each project is reviewed by our sampling experts to advise the client the best approach on each individual project.




6. What recruitment channels are you using for each of the sources you have described? Is the recruitment process ‘open to all’ or by invitation only? Are you using probabilistic methods? Are you using affiliate networks and referral programs and in what proportions? How does your use of these channels vary by geography?
   
   We have a dedicated panel recruiting department that has been part of Symmetric for over 20 years. Each panel has its own ways to recruit. For our consumer panel, we run our own affiliate network where most recruiting is done through there and it is online, but also recruit via social media, referral programs and advertising. Our B2B panels were primarily built using industry conferences and direct invite- for example our Technology panel was heavily recruited by attending technology conferences and having a booth there to sign people up. Our Physician panel was built by attending medical conferences and by fax directly to doctors’ offices. We have been recruiting panels since the 1980s and have a lot of expertise here with different methods.




7. What form of validation do you use in recruitment to ensure that participants are real, unique, and are who they say they are?
   
   At the time of registration for each of our panels the respondent fills out a very long, detailed form which includes names, mailing address, and phone number. We focus on validation using both technology and the human element. Our quality assurance department reviews all new members that are applying to be part of our panels for data quality before they are allowed to join. As for technology and software, we use Mailers+ software to validate real address are input at the time of registration along with Strike Iron, SampleChain, MaxMind, Browser Hawk and our proprietary system called Sleuth. For our B2B panels we use LinkedIn and look up the company name, phone number, address and email provided at the time of registration to tie to a company. For the Physician panel we use Certifacts to verify they are a real US Physician.




8. What brand (domain) and/or app are you using with proprietary sources?
   
   We have dedicated panel sites for each panel:
   • Consumer - www.acop.com
   • Technology - www.technologyboard.com
   • Executive - www.executiveadvisoryboard.com
   • Contractors - www.contractorboard.com
   • Medical - www.medicaladvisoryboard.com
   • Physicians - www.physicianscouncil.com
   We also have an App for our consumer panel that is available in the iTunes and google play store.




9. Which model(s) do you offer to deliver sample? Managed service, self-serve, or API integration?
   
   We offer sample via managed services and will occasionally do an API integration. We pride ourselves on not being transactional so each client will be given a dedicated project manager team that will review all specs and make recommendations on what sources (and methods) to use to properly execute the project at hand.




10. If offering intercepts, or providing access to more than one source, what level of transparency do you offer over the composition of your sample (sample sources, sample providers included in the blend). Do you let buyers control which sources of sample to include in their projects, and if so how? Do you have any integration mechanisms with third-party sources offered?
    
    Symmetric is a sampling company that was formed by full-service market researchers so we believe in full transparency with what samples we are providing. If we are using a source outside of our proprietary system, we will discuss that with you and get approval. Transparency and honesty are something that has lacked in this industry which was one of the reasons we created Symmetric.




11. Of the sample sources you have available, how would you describe the suitability of each for different research applications? For example, Is there sample suitable for product testing or other recruit/recall situations where the buyer may need to go back again to the same sample? Is the sample suitable for shorter or longer questionnaires? For mobile-only or desktop only questionnaires? Is it suitable to recruit for communities? For online focus groups?
    
    All of our proprietary panels are built with respondents that are only signing up to take market research projects and help shape and improve products and services. These are true market research specific panels, not rewards programs or taking surveys to earn more playing time for games. Our average tenure is over 4 years. We will use our online panel members for both online and offline work, qual and quant research, iHuts, taste tests, in person focus groups, etc. Most surveys are designed where we are fine using a mix of mobile and desktop users but if there are surveys that need to be for desktop users only, we will screen for that and this really applies to longer surveys or those with advanced analytics techniques in them.




12. Briefly describe your overall process from invitation to survey completion. What steps do you take to achieve a sample that “looks like” the target population? What demographic quota controls, if any, do you recommend?
    
    With our proprietary panels we are survey invite by email only. We pull a specific sample based upon client specs from our panel via our scientific sampling platform, ICION, and then send direct invites to panel members for the available survey and only those invited are allowed in to complete. We will send a reminder to that original sample pull two days later as we want to get as many completes from that balanced or census rep or specific pull, we make from the client specs. Each project has its own specific needs and will recommend the best approach to clients on an individual project basis. We do not use routers.




13. What profiling information do you hold on at least 80% of your panel members plus any intercepts known to you through prior contact? How does this differ by the sources you offer? How often is each of those data points updated? Can you supply these data points as appends to the data set? Do you collect this profiling information directly or is it supplied by a third party?
    
    On all our panel registration pages members complete a long registration form that contains name, address, email, phone, country, occupation, ethnicity, income, education, children in household and their names and ages along with many other data points that are specific to that panel (for example the physicians panel with medical specialties or executive board with Title and company, etc.) We can append this data or we can re-ask in the survey so all the data is in one spot for you. With our proprietary survey software, Logician, all data collected in that is stored and easy for us to access for future projects, IR checks, feasibility, etc.




14. What information do you need about a project in order to provide an estimate of feasibility? What, if anything, do you do to give upper or lower boundaries around these estimates?
    
    The basic information we need is IR, survey length, number of completes needed, qualification requirements and timing of the project. From there we will have an open communication if you want a bid with proprietary sample only or a mix of that and partners.




15. What do you do if the project proves impossible for you to complete in field? Do you inform the sample buyer as to who you would use to complete the project? In such circumstances, how do you maintain and certify third party sources/sub-contractors?
    
    We will give you daily updates on how your study is going and access to online reporting if we are programming and hosting data. If it becomes apparent that the survey will not fill with our current sampling plan then we will have a discussion with the client on how to best complete the project and timing and costs. This could involve bringing on a sampling partner or suggesting using a mixed method approach to hard-to-reach groups. As stated before, we are a sampling company born from a full-service MR firm so we understand and can relate to these issues which is one of the reasons we started Symmetric because of so much overpromising and underdelivering from some sample companies. As for how we verify and utilize third party sources, we have been buying sample for over 40 years. We run all providers through test surveys to see how their sample perform and if they pass those tests, we will bring them on for a small % of completes on live surveys. Our Quality Assurance department tracks deletion rates, poor data quality rates and many other factors on each and every provider and we Tier our sample partners based upon this experience.




16. Do you employ a survey router or any yield management techniques? If yes, please describe how you go about allocating participants to surveys. How are potential participants asked to participate in a study? Please specify how this is done for each of the sources you offer.
    
    No, we do not own or use a sample router. We are sample purists. We do have several partners that do when we have to go to outside sample and understand the pros and cons of that. For our surveys, we directly email our panel member an email where we let them know a survey is available, how much time it will take them to complete and how long the survey will be available along with the survey URL. Only invited participants can login and take the survey.




17. Do you set limits on the amount of time a participant can be in the router before they qualify for a survey?
    
    We do not utilize a router so this does not apply.




18. What information about a project is given to potential participants before they choose whether to take the survey or not? How does this differ by the sources you offer?
    
    On all projects with our proprietary sample, we send a direct email invite to the respondent. In that email it alerts the panel member of how long the survey will be, the incentive they will receive if they qualify and complete the survey and how long the survey is available in field.




19. Do you allow participants to choose a survey from a selection of available surveys? If so, what are they told about each survey that helps them to make that choice?
    
    No. We pull a dedicated sample on each project based upon the client’s needs. As stated before, we are sample purists and firmly believe you pull an ideal sample and send that out, let it simmer and send reminders and try to get as many completes from that original sample pull.




20. What ability do you have to increase (or decrease) incentives being offered to potential participants (or sub-groups of participants) during the course of a survey? If so, can this be flagged at the participant level in the dataset?
    
    We firmly believe that treating panel members fairly is the way to do business and we pay our members a cash incentive for each and every survey they take. We also pay our members for DNQs as we are still taking their time. It drives better data and it helps us achieve panel tenure rates that are over 4 years on average while most sample companies burn through members every 3 months on average. When the original CPI and feasibility is given to a client, we will state what that incentive is to the panel member and we will tell the panel member what the incentive is if they take this project. We are more than OK with increasing the incentive but rarely will ever allow something less to our members as this breaks their trust in us. If we need to vary incentive amounts based upon cells or quotas in a survey then that is perfectly fine. We can flag who is tied to what incentive.




21. Do you measure participant satisfaction at the individual project level? If so, can you provide normative data for similar projects (by length, by type, by subject, by target group)?
    
    We highly encourage our members to reach out to us if they had a poor experience in a survey (for example survey problems or the survey was poorly written and unable to answer the questions properly, etc.) Each survey page has an email and chat link to connect with us and we also will ask Open Ends at the end of the survey gathering their thoughts




22. Do you provide a debrief report about a project after it has completed? If yes, can you provide an example?
    
    We encourage a debriefing meeting at the end of each project (especially low IR, B2B, mixed methodology, or qual projects). When the project is completed, we will reach out with details and are always happy to jump on a call to talk things through too. We are not selling completes here at Symmetric, we are selling quality data from real respondents and we are not a transactional sample company, we consult.




23. How often can the same individual participate in a survey? How does this vary across your sample sources? What is the mean and maximum amount of time a person may have already been taking surveys before they entered this survey? How do you manage this?
    
    Our surveys are by invite only so we can control how often someone is invited for a survey within ICION, our sampling platform. Our guidelines are that once someone DNQs in a screener they are not invited to another screener that same day. Once someone completes a survey, they are not invited to another survey opportunity for 2 days. On average our panel members complete 1-2 surveys per week.




24. What data do you maintain on individual participants such as recent participation history, date(s) of entry, source/channel, etc? Are you able to supply buyers with a project analysis of such individual level data? Are you able to append such data points to your participant records?
    We track what projects our members are on and start (login) and stop times (DNQ or survey complete). We can provide this data to clients that request it.




25. Please describe your procedures for confirmation of participant identity at the project level. Please describe these procedures as they are implemented at the point of entry to a survey or router.
    
    With our proprietary panels we check all this at the time they apply by registering in the panel. All data is reviewed by the quality assurance department along with utilizing the top technology in the industry. We invite our participants via direct email to them with the specific survey URL. If we are programming the survey then we will write a screener and survey with all the proper cheater traps, bot prevention questions, red-herring questions along with using technology to verify the IP address, time zone of the respondent to make sure it matches up properly for your sampling frame and to what we know on the panel member. If we are not programming and just launching to the client’s survey, we will program a pre-screener to do this validation before launching to the client’s screener and survey.




26. How do you manage source consistency and blend at the project level? With regard to trackers, how do you ensure that the nature and composition of sample sources remain the same over time? Do you have reports on blends and sources that can be provided to buyers? Can source be appended to the participant data records?
    
    Within our Panel Management software, we are able to track where a respondent came, prescreen, and set quotas (overall and by provider). For trackers we can set quotas by provider to ensure the same sample blend. At any time reports of sample blends can be provided to the client and a sample source code can be added to the URL.




27. Please describe your participant/member quality tracking, along with any health metrics you maintain on members/participants, and how those metrics are used to invite, track, quarantine, and block people from entering the platform, router, or a survey. What processes do you have in place to compare profiled and known data to in-survey responses?)
    
    Participation metrics are stored for the number of invites, completes, did not qualify, and dropouts. Every 6 months the panel is cleaned for inactivity. Tracking and quarantining/blocking for fraud is done upon entering the pre-screener. Respondents are verified through SampleChain, MaxMind, and our proprietary system called Sleuth. If they fail any metrics a flag is given, and further human checks are done. After a respondent has finished the survey an open-end security question is asked, and data is reviewed. If a member is flagged for fraud (speeding, straight lining, irrelevant answers, etc.) a flag will be placed for further human checks. After 3 flags a panel member is removed from the system.




28. For work where you program, host, and deliver the survey data, what processes do you have in place to reduce or eliminate undesired in-survey behaviors, such as (a) random responding, (b) Illogical or inconsistent responding, (c) overuse of item non-response (e.g., “Don’t Know”) (d) inaccurate or inconsistent responding, (e) incomplete responding, or (f) too rapid survey completion?
    
    Adding red herring questions (OE, grid traps) along with fake brands or products helps trap random responding along with setting speedster checks throughout the survey. Putting in correct validation and in survey terms to prevent illogical, inconsistent/inaccurate responding. To ensure response setting validation on questions that need to have an answer and wording questions in a way that “Don’t Know” is an invalid or illogical answer.




29. Please provide the link to your participant privacy notice (sometimes referred to as a privacy policy) as well as a summary of the key concepts it addresses. (Note: If your company uses different privacy notices for different products or services, please provide an example relevant to the products or services covered in your response to this question).
    
    https://www.acop.com/privacy/ Our privacy policy starts with the various laws and regulations we follow (CCPA, Insights Association, Privacy Shield, GDPR, CCPA, and etc.). From here we have sections on personal information we collect, how we use your information, how you can control your information, and how we secure your information.




30. How do you comply with key data protection laws and regulations that apply in the various jurisdictions in which you operate? How do you address requirements regarding consent or other legal bases for the processing personal data? How do you address requirements for data breach response, cross-border transfer, and data retention? Have you appointed a data protection officer?
    
    Various resources are monitored for changes in data protection (and other) laws and regulations. Consent is built into applications and processes and is generally the foundation for the Legal basis for processing personal information. Data breach requirements, cross-border transfer, and data retention are defined in specific client agreements. Changes to applications and processes to match client requirements regarding consent and legal basis are also performed. We have appointed a data protection officer and their contact info is included in all privacy policies in the relevant sections.




31. How can participants provide, manage and revise consent for the processing of their personal data? What support channels do you provide for participants?
    
    At any point, any member can update their membership to not allow access or use of their personal information. On our member pages and any accessible websites, they are provided a link in which they can request their personal information, and to not sell their personal information. All requests are sent to an email distribution and answered and updated in our system within 72 hours. For European citizens, in accordance with the European Union General Data Protection Regulation they are provided a link to where members can exercise their right to erasure at which point all information will be deleted after 10 days of the notice. If at any point a member is having trouble, they can either call a direct line or email a support group that gets back within 72 hours of all requests.




32. How do you track and comply with other applicable laws and regulations, such as those that might impact the incentives paid to participants?
    
    Respondents are offered a minimum of 2 ways of cashing out their accumulated points. All cash-outs are logged and stored to ensure any honorarium threshold that meets tax laws is followed. For European Citizens’ right to erasure they are warned twice that any accumulated points will be forfeited upon erasure.




33. What is your approach to collecting and processing the personal data of children and young people? Do you adhere to standards and guidelines provided by ESOMAR or GRBN member associations? How do you comply with applicable data protection laws and regulations?
    
    Decision Analyst follows COPPA and EU-GDPR. Upon entry of a new member, anyone that enters an age of +1 to the local regulations is then terminated and information is not stored. For survey’s relating to children participation, express consent is asked to the parent or legal guardian before any involvement with the child.




34. Do you implement “data protection by design” (sometimes referred to as “privacy by design”) in your systems and processes? If so, please describe how
    
    Our Panel management system was redesigned with "data protection by design" implemented in the development process. Other systems and processes will be implementing "data protection by design" when they are due for new versions.




35. What are the key elements of your information security compliance program? Please specify the framework(s) or auditing procedure(s) you comply with or certify to. Does your program include an asset-based risk assessment and internal audit process?
    
    The information security compliance program is currently informal. An ISO compliant security compliance program (and other ISO 27001 compliance programs/standards) are currently being developed. ISO 27001 compliant standards are being implemented over the next few years. All programs and standards will be modified within the ISO 27001 framework during this implementation.




36. Do you certify to or comply with a quality framework such as ISO 20252?
    We are not ISO 20252 certified. We do comply with many market research quality standards and follow laws and rules and regulations globally.




37. Which of the following are you able to provide to buyers, in aggregate and by country and source? Please include a link or attach a file of a sample report for each of the metrics you use.
    
    We can provide any of the below reports upon request.
    1. Average qualifying or completion rate, trended by month
    2. Percent of paid completes rejected per month/project, trended by month
    3. Percent of members/accounts removed/quarantined, trended by month
    4. Percent of paid completes from 0-3 months tenure, trended by month
    5. Percent of paid completes from smartphones, trended by month
    6. Percent of paid completes from owned/branded member relationships versus intercept participants, trended by month
    7. Average number of dispositions (survey attempts, screenouts, and completes) per member, trended by month (potentially by cohort)
    8. Average number of paid completes per member, trended by month (potentially by cohort)
    9. Active unique participants in the last 30 days
    10. Active unique 18-24 male participants in the last 30 days
    11. Maximum feasibility in a specific country with nat rep quotas, seven days in field, 100% incidence, 10-minute interview

Should you have any further questions, please feel free to contact our President at Jason@symmetricsampling.com