Does Panel Quality Matter?

By Author: Lucy Carter
Total Articles: 100
Comment this article

Do public relations agencies care about the quality of the online panels they use? In my experience they do, or rather, they would if they had the time to consider the question. The fact is, many busy account managers and execs, focused on delivering to tight deadlines and with little general knowledge about the market research industry, are not aware of the issues of panel quality, unless they have encountered a bad experience which has seriously impacted the credibility of a campaign. In many cases this is not surprising. Given that the wider usage of online panels among public relations agencies is for media relations purposes, it is not surprising panel quality is not top of mind. Such polls tend to be limited in scale, and run very quickly. While every effort is taken to target appropriate audiences, and craft questions which have every chance of highlighting interesting stories, the issue of quality is taken as red. In many ways, why shouldn't it be? Panel users are assuming that they get what they pay for.

Yet, in the market research world, in the ten years or so that online access panels have become an integral ...
... part of survey solutions, the issue of quality has never gone away. There can be little doubt that market research, including polling and large scale consumer surveys, have become significantly cheaper, to the point of making the costs of research to a far wider group of companies than before. Routine polls, flash surveys, trackers or even quick surveys to support a new pitch, have become routine for most PR agencies, including for lower spending clients.

So the online access panels have brought enormous benefits to public relations campaigns, and I suspect most companies would not want to go back to the costs they were paying then years ago.

But if online polling has increasingly come to be seen as a commodity, does quality matter?

Quality does matter

Panel quality does matter, although its perfectly understandable why its not the most compelling issue for PR execs. Why? In the end it matters because people need to be confident that the survey results are credible.

What are the typical quality problems of panels? There are two types of problem. The first relates to identity. Knowing that the participant is who they say they are. In some cases, there are panellists who have multiple accounts completing numerous surveys when they are not eligible to. It may sound amusing knowing that some individuals in China are registered with numerous panels in the UK. But if panel companies don't actively root out these people, not are they fraudulently earning money, but could seriously affect the quality of individual polls. A man in China impersonating a British consumer is just wrong. It does sound ridiculous, but monitoring where survey completions have been undertaken is a constant requirement.

But you don't have to go to China to see that a minority of panellists try to establish multiple accounts with panel providers. This is a particular problem if different profiles have been completed to ensure they receive multiple survey invitations.
The second type of problem relates to the quality of responses from panellists. The most obvious signs are panellists who complete surveys much faster than they should - e.g. if the average completion time is 10 minutes, but individuals rush through the same survey in 1 minute, you know the panellists hasn't even read the questions. Other examples include straight lining (where the panellist races through grid questions ticking the middle answer all the way down, ignoring questions or ticking answers at random.

Quality is as much about incentives as compulsion

Panel providers can use a number of quality checks on individual surveys, and better panels should also run regular audits to trap and remove inappropriate panellists. But that is only part of the answer. The truth is, a large part of quality control stems from the general survey experience and the way panellists are treated.

Conducting opinion surveys which are varied in terms of topic, and engaging in the way questions are asked, are critical tools in panel quality. We also know that shorter surveys are likely to get a better quality of response, so conducting 25 minute surveys, when a thoughtfully designed survey of 10 minutes is possible is the best way forward.

But apart from offering fun and engaging surveys, panel companies have to recognise the need to pay fair incentives. Panellists won't get rich from completing surveys, but they should receive a fair incentive for the time they spend. If panel companies do not treat their panellists fairly, then they can hardly expect them to treat their surveys with respect. We know that as many panellists do surveys because they enjoy them as those whose primary motivation is to earn money. Panel companies need to recognise this as part of their quality control measures.

Of course, paying too much leads to the accusation of professional respondents. One way to mitigate that concern is to limit the number of surveys an individual takes during a specific time period. Some panels have a so called "quarantine period" to ensure that each panellist is not bombarded with survey invitations, even if they wanted to.

Size is not everything

Lastly, there is a strong argument for small is beautiful. Many panel companies still talk about the total number of people they have registered on their panels. But arguably the more important question is what level of response does the panel get. If only 5% of a panel regularly respond to the survey invitations they receive, this is perhaps the biggest indicator of (poor) panel quality you could ask for. The panel needs to be large enough to ensure nationally representative polls, and to offer the flexibility to focus on specific demographics or profiling criteria (e.g. young mother, home owners, directors of small companies, etc) but beyond that, the panel should only be as big as it needs to be. Building an ever growing panel at the expense of quality and high response rates is most definitely not what quality-focused panel providers should be doing.
Neil Cary is a Director of the Market Research Agency, Redshift Research and runs the Online Access Panel, Crowdology.