This may not be the most salacious topic that we’ve ever covered on our blog, but it is one that comes up more and more often in market research as companies want to maximize the amount of information from their quantitative research studies.  However, how much material is too much?  At what point does the length of the survey start to impact the accuracy and integrity of the data?  When does respondent fatigue set in and what can researchers do to tailor our questionnaires accordingly?   

So, first off, let’s be clear…we are talking about quantitative research and not qualitative research, i.e. focus groups.  Qualitative research is usually conducted with smaller samples and tends to be more unstructured and exploratory in nature, typically with the goal of developing insights that can be further examined in a quantitative study.  In order to garner more conclusive results and dependable estimates of market characteristics, quantitative research, or what we call our SP/market segmentation study, is the confirmatory tool in our arsenal of services that, if done the right way, provides the ability to project the findings on to the general population. 

What is the right way then to conduct descriptive quantitative research (studies typically conducted via CAPI/face-to-face, CATI/telephone or CAWI/online & mobile)?  Most of our surveys are either cross-sectional (ad-hoc/taken at a snapshot in time) or tracking studies (repeated over time but with a different sample of demographically similar people each time).  Regardless of the type of survey, the length of the interview always is a major factor in questionnaire design.  For us at Global Media, the first step here is to identify a core problem or question that the study should be designed to answer.  Based on this “core problem definition,” the questionnaire should be constructed so that it covers all the bases in trying to get to the root of the issue.  Once we come up with a preliminary questionnaire, then we need to ask ourselves is every question in this survey a useful one?  How will the data from each question aid us in our analysis and help us to answer the core problem as it was defined?  Are we prepared to act on the data from each question or would the result be more “interesting” but not necessarily “actionable”?

Once this filtering process has taken place whereby certain questions may be eliminated, similarly related questions are grouped together, and/or more effective skip patterns may be put in place, we aim to create a questionnaire that is no longer than about 20 minutes.  This is the industry standard that was detailed in a 2004 study by Sandra Rathod in which she writes: “fatigue effects become more pronounced around the 20 minute mark.”  However, we have found that it’s not just fatigue that sets in after about 20 minutes; an excessively long questionnaire has a negative impact on the integrity of the data.  Our findings show a dramatic decrease in accuracy when a survey (especially an online one) exceeds 20 minutes, on average.  There is a direct correlation in the data whereby respondents increasingly provide “safe” answers and/or answer “don’t know” more often so that they can reach the termination point of the interview.  When surveys exceed 20+ minutes, we see many more instances of “straight lining” in which respondents provide the same answer repeatedly, which for us requires an additional amount of quality control on the back end. 

Now, for the skeptics out there who think it’s not possible to get all the answers to your questions in an interview of this duration, we disagree.  A 20-minute questionnaire provides more than enough time to produce detailed data-driven insights into your primary questions and concerns.  It is just a matter of focusing on the right questions (and only those questions) and then skillfully designing the survey to answer those questions.

So, what can you do to keep your surveys from becoming too long and, in turn, ensure that you are getting the most accurate data possible?   Here are some useful tips:

- Most importantly, challenge the need for each question: What does this particular question tell us?  How will we use the data from this question to steer our strategic or tactical approach to programming a radio station or TV channel/network?

- Avoid long introductions, overly verbose or wordy instructions/questions, and “thank you” messages.  It’s important to provide encouragement to respondents and to remind them how valuable and useful their input is; however, edit these messages down as much as possible.

- Consider pre-populating data fields using existing demographic and geographic information (ask your sample provider or local field service to supply this information on the back end for each completed interview).

- Give participants the opportunity to take a break.  Our online market segmentation studies allow respondents to stop and start as many times as they would like in typically a 24-hour period.  If you are using a CATI methodology, give interviewers the ability to propose to respondents that they take a short break before resuming/concluding the survey.

- For tracking studies, think about dropping a question if the answer to a particular question never changes.

Quantitative research can produce so many valuable insights and should serve as the foundation for sound strategic planning in any successful business.  The team at Global Media specializes in creating questionnaires that extract the maximum value from quantitative research studies and hopes that our recommendations will help you to achieve this objective as well.