Choosing the Right Sample Panel Vendor

We know that a study is only as good as the data it produces – garbage in, garbage out. One of the best ways to ensure you receive quality data is to assess the level of respondent quality provided by your sample panel vendor. Below you will find the key questions we ask any potential vendor prior to panel selection. These questions help us determine what type of quality controls have been implemented by vendors when selecting and securing their panel.

Did you build your panel by purchasing other lists?

First, it is important to understand how sample companies have built their lists. These lists are built by either purchasing other existing panels and merging them into a larger one or building a list from the ground up. Depending on how the lists were built, we conduct a follow-up discussion with more clarifying questions.

If the vendor acquired their sample by purchasing other lists, it is important to know how they are managing those lists. For example: Are there duplicate panelists who signed up across multiple panels? Is our supplier merging the lists into one large panel? Or are they keeping the panels they purchased separate? We want to ensure they reconcile all the profiles and data points across the different panels they have acquired, as well as providing relevant IDs for digital fingerprinting. Additionally, we want to know what steps have been implemented to confirm panelists are real people; our preference would include methods like pre-verification or ‘TrueSample’. Also, we need to understand how the panel vendor routinely re-validates the data points collected – what process is used, and how often the re-validation is performed.

Did you build your panel yourself?

If a supplier has built a list on their own, there are a separate set of questions asked. For example, we want to know how they have attracted panelists. Historically, posting banner ads on reputable websites has yielded the best results. This allows the panel vendor to vet where they receive panelists from and what kind of panel they want to build. Vendors with top panelists providing referrals to friends and family members also allows for an additional measure of quality control and has proved successful.

Other methods vendors might use to build their panels include “River Sampling”—where visitors on specific sites are invited to take a survey on a different site. River Sampling tends to be convenient and cost effective for panel vendors but lacks certainty respondents are truly qualified to participate.

Additionally, we want to know the process for confirming panelists are interested in joining the panel. More specifically, are they using a double opt-in method when recruiting panelists? During panel recruiting, what type of profiles or data points are they asking for? The panel vendor might collect info on one or more characteristics – census, panelists’ interests, or even whether they have pets. All of these will help provide you with panelists who are well suited for your surveys. We also want to know how the panel vendor keeps its panelists engaged-such as asking for feedback, communicating with them regularly, etc.

The final piece we need to understand when working with our panel vendors—how panelists are incented to respond. We believe in a balance between providing an incentive good enough to get the panelists interested in responding, but not too good that they become “professional” survey takers.

How does your panel compare to census data?

The best samples are those that are most representative of the overall population. It is just as important to ask the “right people” your questions as the questions themselves. A representative sample is critical for gathering responses that will help you make data-driven decisions. That said, it is impossible for any one survey to completely match census data. A good sample vendor is going to be aware of how their panel compares to the census overall. For example, they may be aware that their lists tend to skew female, or to a younger demographic. Also, they may have lower numbers from a geographic region or a specific income level. Understanding the differences between the panel and the national averages prior to fielding the study will allow you to determine if you will need to weight the data to adjust for those differences. For example, if a specific region is understated on the panel relative to the overall population you will need to provide a weight to keep the comparison apples-to-apples per se.

How do you manage quality control?

Beyond how panels are built and how representative they are to the rest of the country; it is important to know the quality of data you are receiving has been thoroughly checked. Below we have highlighted those questions we always ask to ensure the panel is maintaining a quality control process that will provide us and our clients with quality data.

How do you handle low responders?

It is important to understand how panel vendors handle low response rates from their panelists. Unfortunately, not all panelists are going to be fully engaged and low response rates can lead to longer time in the field overall for surveys and, in turn, driving up costs beyond the scope of the project. Does the panel vendor routinely cull panelists that are not engaged? Do they find some other way to work with their responders to increase engagement? If they are not actively working on increasing response rates it could mean more out-of-pocket costs for you and your client.

How do you  handle “fake” responses?

Some panelists may be taking surveys just for the incentives – shocking, I know. With modern technology, some panelists might decide to cheat the system, by taking multiple surveys at a time, or perhaps just “straight lining” their responses; giving the same answer to every question, regardless of their actual opinions. It is important that sample vendors conduct due diligence and check for irregularities in the data before sending to researchers. We also need to understand the consequences for panelists cheating the system so we can eliminate their participation in the future.

Is your sample base a blend of panels from an Automated Programming Interface?

One trend we are starting to see among panel vendors is programmatic sampling. This is where panelists are selected from more than one panel. We have realized that it is important to know whether your sample vendor is using Automated Programming Interfaces for a few reasons. First, not all panels are created equal and if you have had a negative experience with one previously, you likely do not want to work with them again. Additionally, it may introduce a new level of uncertainty into your study. For example, it may be difficult to discern if your results are being influenced by the mix of different panels.

Overall, choosing the right sample vendor is as critical to the success of your project as the questionnaire itself. Without a representative sample, you put the level of data quality at risk for your study and/or one those you are managing for a client.

The post Choosing the Right Sample Panel Vendor appeared first on The Stevenson Company.