Baseline: Writing Effective Questions and Answers

Tags Baseline

Without effective questions and appropriate answer choices you won’t get accurate or valid data from your survey. Below is a list of common pitfalls when writing survey questions. Avoiding these 7 pitfalls will increase the likelihood that your survey provides the data you desire.

 

Avoid Pitfalls when Writing Questions

1. Avoid Leading Questions

Description:

Example of a leading question
Leading questions unknowingly guide the participant toward a certain stance- either positive or negative. Not only may this limit the data you obtain, but in some cases it may invalidate the results. If questions such as this are needed to glean specific information, try to balance out the survey by including a different stance in another question.Example of a leading question

Resolution:

This question has been redesigned by replacing “enjoy” with “satisfied”. Note that the question is also framed to offer a bit more detail about the context and why the question is being asked. In addition, the answer choices are bipolar- both positive and negative options are available.

 

 

 Leading question is resolved

2. Avoid Double-Barrel Questions

Description:

Double- barrel questions ask two or more questions within the same prompt. This is a very common pitfall as surveyors often include details that are related in context within the same question or add a similar prompt in an attempt to clarify a question. But if multiple dimensions exist in one question (such as “enjoy” and “learn”), the data won’t provide the right information, because it isn’t clear which question the respondent Example of a double-barrel questionis answering.

Resolution:

To resolve double-barrel questions, divide prompts or several dimensions into different questions or use a matrix. With a matrix, several related questions can be asked that have the same rating scale but participants answer each prompt independently.

 

 

 

 

 

 

 

Double-barrel question is resolved through separation Double-barrel question is resolved through a matrix

 

 

3. Avoid Questions that Assume

Description:

Asking questions using acronyms, jargon, or technical terms assumes that the participants understand the questions enough to answer accurately. And in some cases, assumes the participants have knowledge about something they may not.Example of an assuming question

Resolution:

Remove any acronyms, jargons or technical terms. If you wish to include questions that some participants may not be able to answer, provide an opt-out option such as “I do not know”. 

 

 

 

 

 Assumptions are now resolved

 

Avoid Pitfalls when Writing Answer Choices

4. Match the Question with Appropriate Answer Choices

Description:

If you are asking about satisfaction, be sure your answer choices relate to satisfaction and not to another measure such as quality. 

 

 

 

 

 Answer choices are incompatible with the question

 

Resolution:

If there is not congruence between the question and the answers, modify the answers so they correspond to the most accurate measure. Baseline offers a list of recommended scales for survey questions at this link: http://baselinesupport.campuslabs.com/entries/21990098

 

 

  Question is now compatible with answer choices

  5. Use Mutually Exclusive Answer Choices

Description:

It is very common to overlap choices, especially with a number range. In the example to the right, respondents would need to select 2 different answers if they attend the theatre 3 times per year. Be sure each answer choice is mutually exclusive of the others.

 

 

 

 Answers are not mutually exclusive

 

Resolution:

To avoid overlapping answer choices, review each answer separately and compare it to the answer before and after it. If there is overlap, modify the ranges as needed.

 

 

 Answers are now mutually exclusive

 

6. Use Collectively Exhaustive Answer Choices

Description:

To aid filtering of final data, it is helpful to list possible options instead of using an open textbox. However, be sure that all possible choices are available in the list of answer choices.

 

 

 

 Answers are not collectively exhaustive

 

Resolution

Try to be as exhaustive as possible. List as many options as are feasible and then include an open textbox to elicit less common answers. 

 

 

 

 

 

  Answers are much more exhaustive

 7. Use Opt-outs Sparingly

Opt-outs are often options such as “I do not know”, “I’m not sure”, “I can’t recall”, or NA. At times, these options can offer more exhaustive data and are helpful for the respondents. However, too many of these options may limit the breadth and value of the data collected.

 

References:

The Research Bunker Blog. (April 27, 2010) Mutually Exclusive & Collectively Exhaustive Survey Tips by George K.  https://rmsbunkerblog.wordpress.com/2010/04/27/mutually-exclusive-collectively-exhaustive-survey-tips-market-research-syracuse-survey/

Survey Monkey. Help Center Design Tutorials.

Campus Labs. Survey Design Best Practices Webinar.