There’s a lot of noise about the validity of online surveys these days. People questioning the reliability and accuracy of the results, the so called ‘error margin’, samples sizes and panels, and so forth. We’ve all read about it, and it can get pretty confusing.
Let’s clear some stuff up, shall we?
All online research is not the same. Just because fruit grows on trees, that doesn’t mean that all fruit that grows on trees is the same. Just like fruit, there are a variety of online survey types. Facebook polls, for example, are not meant to be used for serious research purposes, only for entertainment.

True market research requires methodology, scientific principles, oversight, and a plethora of other aspects. This holds true whether the survey is online or offline.
Telephone Surveys VS Online Surveys
What makes one survey more accurate than another? That is the question that people should be asking. Accuracy and reliability of data is the short answer, the long answer consists of the ‘why’.
It doesn’t matter ‘how’ you conduct a survey or a poll (online, offline, interception, etc.), it just matters that you follow and understand the rules of the art. We often see pollsters (political in particular) conducting very similar surveys, yet it happens quite often that each firm arrives at different results. Why is that? If I’m asking aß similar question to 1000 people, shouldn’t the results be statistically similar if I ask that similar question to another 1000 people?
No, the truth is that ‘similar’ questions are not ‘the same’ questions. Similar market research firms are not ‘the same’ firms. Similar pollsters are not ‘the same’ pollsters. The truth is, when it comes to research, everything is in the details.
The Phone Survey Explained
Telephone surveys and polls work in the following way. First, draft up a questionnaire. The questions should be designed by a specialist who understands how questions work, for just as with everything else in surveys, it’s about minimizing error. “Do you like X brand or politician?” is a completely different question from “Would you buy X brand or vote for X politician?” The wording matters.
After making sure that the questions are neutral and don’t ‘lead’ the respondent, tackle sample. For the telephone, this means selecting a few thousand phone numbers in strategic locations which, in theory, could be projected upon the whole population.
(Note: there are many different types of surveys, some have lists of pre-selected eligible respondents (Doctors, Professionals, people between 24-35, etc), and some survey typed are called probabilists (these are the most reliable phone surveys, because the sample is truly random). More on this in later posts)).
Then the survey is programmed into a CATI (Computer Assisted Telephone Interviewing) software, and the telephone interviewers get to work calling people.
Online Surveys Explained
For online surveys, the process is similar, first we design the questionnaire with the same neutrality as the phone questionnaire, but then, instead of random phone numbers to represent the population statistically, we turn to panel management software, such as the excellent one provided by CINT, and select a sample (or Type) of people from the panel list that represent the group(s) of people whose opinion or input we seek. Then, a questionnaire is programmed into an online survey program, like the one provided by Voxco, an invitation is sent to the panellists, and they answer the survey online.
The Difference Explained
For a typical phone survey, say a 15 minute, 1000 completes survey type with a completion rate of 2 per hour, and a deadline of 5 days, we need to complete 100 survey hours per day, and at 5 hours per shift, that equates to 20 telephone interviewers working per day on that project.
Not all telephone interviewers are created equal, some are great at their craft, and can complete 4 surveys per hour, some are terrible and can barely manage 1 per shift. We’ve all been called by a telephone interviewer that was so bad, so incomprehensible, rude, in a bad mood, that we asked to speak to their supervisor to complain. We’ve also all been called by those really good ones that even convince us to do the survey during suppertime simply because they’re that charming.
On the phone, people don’t always answer the same way to each question simply because the person conducting the survey on the other line is either good or bad, charming or boring, conforming to MRIA rules and guidelines or not, called you when you were in a bad mood, or during supper, or too early or too late, and for a plethora of other reasons. That, even though it’s being supervised, causes errors in the results for which ‘the margin of error’ doesn’t account.
Alternatively, even though we circumvent the ‘human aspect’, online research also contains some inherent flaws. For example, you can never be 100% sure that the person to whom the invite was sent, is truly the person answering the survey. Also, some research has shown that not everyone answering an online questionnaire takes the time to read the whole question before answering. Additionally, the general response rate is lower when using online research, and professionals (Doctors, Architects, CEOs, etc,) are quite harder to poll.
So What’s The Best Solution?
Well, that all depends on the type of survey you’re conducting, but generally speaking, if you’re looking for precision, the Mixed Approach is hands-down the best solution.
The analysis of the results gathered by a mix of online and offline surveys allows for greatest reliability of data. (Note: The analysis of survey results (online or offline) is just as responsible for accuracy as the survey itself. More on this in later posts).
By using both methods, online and offline, not only are you getting the best of both worlds, but you’re actually double checking your results using two different, and independent techniques. The mixed approach, in our ever-evolving electronic world, is truly the answer to all your research questions. Both techniques are complimentary to each other, and their weaknesses and strengths are often mutually exclusive, which is why, conducting both simultaneously, yields the most precise and reliable results.
This is a fascinating subject, and in the coming months, we will be exploring it, as well as methodology, phone room management techniques, online sample management techniques, the margin of error, and many other subjects related to the online/offline survey reality.
Meanwhile, if you have any questions about online or offline research services, software, or techniques, please don’t hesitate to give us a call. Also, if you have a chance, come and see us at CASRO Annual Conference (2012, October 8-11), we’d love to meet with you face to face and in this online world, shake hands. We’ll be answering questions and presenting many state-of-the-art research products, and would love to get your insights, comments, and opinions.
Until the next time!
Posted by Eric Perreault