As an information
gathering exercise, surveys are absolutely unreliable. Here’s a list of my top
5 peeves with them.
1.
Flawed design
Most surveys invariable
contain a section that uses demographic data to pigeonhole respondents. Since
people rarely fit into watertight compartments or categories, any ‘insight’
that emerges from such data is highly suspect. (Related: See a post on why psychographicsare a better way to segment here)
2.
Ratings on a scale of 1 to 10
Some things are easy to
measure. In fact, we have dedicated an entire branch of science to it.
Other things are much more difficult to measure. Take the all-too-human pain,
for instance. While I can tell you in Scovilles how pungent the chili I had for
lunch was, I’m still asked by my dentist to rate my pain on a scale of 1 to 10
as he gently probes my horribly aching tooth. Can anyone decipher “Aargh” for
me please?
3.
Self-awareness
How good are you at
recognizing your own strengths and weaknesses? Seriously! If I put you on the
spot and ask you to rate yourself, how truthful would you be? Now how about
rating others while you are at it? How much of a ‘good guesser’ are you? Now
try doing that on a 10-point scale and let me know how scientifically it goes.
I’ll be waiting to hear from you. I promise!
4.
Compiler bias
All too often, survey
compilers are looking to reverse-engineer a finding to conveniently suit there
assumptions. No matter how many trick and placebo questions are included, the
tone, comprehension and response method allows compiler bias to creep in. When
the question, blatantly assume you do something a certain way, how vehemently
would you disagree? Or agree?
5.
Open-ended questions
If you thought 10-point
scales were bad, wait till you get the dreaded open-ended question. This is
around where the survey designer officially gave up and said, “What the heck!
It doesn’t matter anyway.” For those struggling to interpret
statistically-significant insights from the data, sorry! You should have
included the helpful follow-up question right there. Now guess away.
So there! That’s my list
for not trusting surveys. Let me know what your objections are. Even if you don’t
agree! Especially if you don’t!
No comments:
Post a Comment