Getting Serious About Sample Quality

By Author: Philomath Research
Total Articles: 72
Comment this article

Imagine you’re a market research (MR) analyst. While analyzing data from a recently wrapped survey, you suddenly notice hundreds of entries with gibberish in the open-ended fields. With a sinking feeling, you realize the survey has been compromised by corrupt parties and your study results might be skewed. Sadly, this imaginary scenario is all too common.

Surveys are only as good as their responses. If your respondents are real, engaged participants willing to share opinions, you’ll receive great insights. If click farms, bots, imposters or other fraudulent parties participate in your research, you’re stuck with oversights.

Getting serious about sample quality involves identifying various sources of fraud, and imposing rigorous quality checks on them in surveys and panel studies to filter them out.

Identifying fraudulent parties affecting online surveys and panel results
Click farms (or survey farms)
Click farms are groups of people hired to fill bulk responses on surveys. Just like sweatshops employ workers at low wages, click farms complete multiple surveys daily at minimum wage and ...
... provide bogus responses in massive volumes, skewing sample quality and study results. It’s difficult to detect fraud with click farms because their behavior is very similar to legitimate visitors.

Bots
Did you know more than 50 % of all web traffic is from bots? It’s therefore essential to be aware of their impact on your sample quality. Simply put, bots are computer programs designed for multiple purposes. While bots can be useful in website scraping, they are very destructive when used to impersonate humans in online surveys, skewing study results and sample quality.

Online panel fraud
With many panel studies moving online, it’s becoming easier for fraudulent parties to falsify IP addresses to impersonate others, and botch panel results. For eg. If you’re looking to recruit and incentivize panelists in Europe, tempted fraudsters might route their IP address to a European country, pretend to be locals and give false responses, damaging sample quality.

Even though experts say it’s difficult to get 100% real data with 0% fraud, rigorous quality checks can minimize sample corruption.

Checks to improve sample quality
1. Double opt-in
A single opt-in requires individuals to tick a box to receive future communications from businesses. A double opt-in requires an additional step of email verification to ensure panelists and respondents are fully committed to receiving communications from your research firm. This way, you can approach only real, engaged participants for future studies.

2. Identity verification
Impersonation of participants in online panels has become a concern for many firms, which is why identity verification is necessary. By asking panelists a few self-identification questions, and comparing their responses with their previously-filled panelist profile, you can re-confirm their identity, filter out imposters and ensure better sample and data quality.

3. Subject matter check
Doing a quick screening test before the actual survey to confirm if your participants are well-versed on the research topic, ensures better response quality and minimizes bots, imposters and click farms. You can ask participants to pick the correct terminologies, definitions etc. from a list of options to ensure they are educated on your survey topic.

4. Conflicting answer check
Consistency in responses during online surveys or panels can give you a clear picture of a participant’s opinion and frame of mind. However, if a participant’s survey responses are inconsistent, it could mean your participants aren’t paying attention or there are fraudulent parties. Measuring consistency can definitely help you assess sample engagement.

5. Device fingerprinting
You can use device fingerprinting to stop imposters from taking a survey multiple times (by rerouting their IP addresses and assuming fake identities). Device fingerprinting allows only a single response to be captured per device, so that imposters can’t give multiple responses, even if they change IP addresses.

6.Speed tracking
When participants rush through a 20-minute survey in less than 5 minutes, you know that adequate thought wasn’t given to your survey questions or it could be a click farm infiltration. Calculating participants’ completion times can help you decide how thoughtfully they responded. Anything less than a third of the average survey duration is considered speeding.

7. Open-end validation
When survey farms, bots or other fraudulent sources fill out surveys, they may give randomized, irrelevant, duplicate, misspelled or gibberish answers in response to open-ended questions. At the very least, this is an indication that your participants are not mindful in their responses. Most likely, you’re dealing with a fraudulent party.

Philomath Research recruits expert panel moderators and programs high quality online surveys to rigorously implement these checks and deliver high sample quality for our clients.

About Us
We are a leading market research firm recognized for our cutting edge qualitative and quantitative research solutions. Contact us to know how we’re getting serious about sample quality. For more information on our research services and solutions,

For more information on our research services and solutions, click here .