Last month I presented at The Quirks Event in Brooklyn and was honestly blown away by the response I received. To all of you who reached out at the show and after, please accept my sincere heartfelt “Thank you!” Considering the response, I thought it made sense to share the highlights of the presentation in our blog for those unable to attend

A 30-minute presentation is a lot of cover in a single blog so I’ve decided to break it up into 3 which I’ll release each week. This blog is part three where we talk about best practices in fielding.

As I write this is early afternoon on Friday right before the Memorial Day weekend. The end is near of this week and this blog series. I can almost taste the ice-cold beer and good food in my stomach, can’t you? The same is true for fieldwork. You’ve done all the work to design and built the perfect survey instrument and selected the perfect vendors. If you mess up the fielding it will be like a cold rainy Memorial Day weekend. Let’s dig in and figure out how to guarantee a mid-80 degree and sunny days ahead.

First off, make sure to set your survey up for success. If the platform you’re working with has basic deduplication technologies such as IP & cookie deduplication or even digital fingerprinting, make sure they are turned on. If you’ve gotten this far you’re probably working with a supplier who has these things in place but you can never be too careful and they won’t screen out legitimate respondents.

If you’re supplier doesn’t have it and you have the capability, add a reCAPTCHA at the beginning of your survey. Bots are real, wide-spread and it’s just a few lines of code and super simple to install. But don’t go overboard if your supplier has it. Asking respondents to solve two CAPTCHA’s is just mean.

Build attention filters into your survey. But don’t go crazy. One near the front, one in the middle is totally sufficient. I heard a story once about a survey for parents and their children who have ADD to take together. It was a 40-minute survey with many of grid questions in it and about 8 attention filters. One at about 38 minutes that was apparently rather deceiving. Let’s not be the guy that designed that monster…. Please.

If you’re running a B2B or IT study, drop a Red Herring question in it. Red Herrings are questions that require specific professional knowledge to answer. For an accounting professional, it may be a multi-select on the definition of GAAP. For an IT professional, it may be a coding or hardware question. Get creative, ask a resource who works in the field and get a good solid question in there. Even the most validated panels have a few impostors running around in there parading as IT & B2B professionals.

Logic Tests are a rather new development in the past year or so. We call them Logic Tests here at Precision Sample although I’m not sure I’ve seen anyone else give them a name. If you have a different name I’d love to hear it!

I view them as an attention filter and a bot detector in one. Think of the word problem questions you had in math class back in elementary school. Robbie has 6 apples. He gives 2 to Johnny and 3 to Jessica, how many does he have left? Then do a multi-select with possible answers. Keep it simple, we’re just validating that a real person is there and they are paying attention. Don’t go throwing calculus in there. Well, unless you’re surveying mathematics PhD’s then you can and it’s an attention filter, bot detection and red herring all in one. Perfection!

Once you’re ready to launch do some quick checks with your sample supplier. I always recommend reviewing their invitation text and prescreening questions if they are prescreening. High quality suppliers would never intentionally lead participants into a survey but things are over-looked and mistakes are made so you can never be too careful.

If your survey doesn’t require live balancing then ask your supplier(s) to start with the hardest quotas first. The simplest example is a survey balanced to census. Our standard practice is to start fielding to high income, males, ethnicities and young male & female audiences first. These are the hardest groups to find and get into your survey so when they come through, you want all quotas open. If the floodgates are opened and all the quotas are filled up by Caucasians and women what you’re left with is trying to find 18-24-year-old Hispanic males who make $100,000+ which is rather impossible. I always get a question about representativeness when I make this comment. The truth of the matter is that there are very, very few 18-24-year-old males that make $100k in the US much less Hispanics. So even though through our fielding methodology you won’t likely get that person in the survey, they are so rare in the US that it does not affect representativeness with a standard sample size.

Require your supplier to field with a reasonable amount of time. Especially for easy general population studies, most larger suppliers can field these studies in a matter of hours. I’m honestly not qualified to comment on the data skews fielding a survey in a few hours may cause but I promise you they exist. There is a fundamental difference between a person who takes a survey at 10 PM vs someone who takes it at 7 o’clock in the morning.  If your project fields in 6 hours on a Friday afternoon/evening you are missing out on a meaningful amount of psychographic and behavioral representativeness. Our rule of thumb is to always take 48-72 hours to field a survey unless requested otherwise by the client.

Keep an eye on your survey while it’s in field. If your IR comes in less than quoted, many suppliers will come back to reprice the project and it’s never a comfortable conversation. By monitoring your survey in field you can make adjustments to qualifying criteria and quotas as you go to make sure you stay in budget.

Yay! Last slide! I’ll keep this quick. If you are forced to multi-source a project with multiple sample suppliers, here’s a few quick tips. First off, digital fingerprinting is an absolute must. If your survey platform does not have it you absolutely must figure out a solution or you will most definitely have meaningful duplication of respondents.

If your platform does not have digital fingerprinting, ask one of your suppliers to be the central clearing house. All of your suppliers have the ability to issue links for your survey to the other vendors. All traffic will then pass through their system and digital fingerprinting. Extra added benefit is that you will only have to integrate a single set of end links into the survey and they will redirect back to your other vendors for you.

Finally, if you can set vendor level allocations on the quotas I highly recommend it. There will always be a vendor in the mix who will jump in and grab all the easy stuff leaving your other vendors to grind it out on the harder quotas. Aside from the fairness, the main issue is that if the IRs on the harder quotas are also lower there may be a price impact for you and blow out your budget.

Now let’s go blow out some beers…way more fun. Happy Memorial Day everyone! Keep it safe out there.

–Don Golden