The Complete Guide to Market Research

Basic Questionnaires

Basic Questionnaires

Writing a questionnaire involves the following steps:

Identify everything that you need to measure.

  • Questions that you need to answer. For example, whether people are interested in buying a new product.
  • Diagnostics that will help you understand the answers to your key questions. For example, reasons for liking or disliking a new product.
  • Questions that will be useful for profiling. For example, age and gender.
  • Questions that are useful for checking the representativeness of the study.

Writing one or more question for each thing that you need to measure

This involves:

Ordering the Questions in the Questionnaire.

Questions in a questionnaire are ordered so as to reduce the time taken to complete the questionnaire and to reduce order effects.

Question Types

Open-Ended Questions

Please enter the name of the last soft drink you bought.

_____________

Most data collection programs will let you constrain the type of responses. For example, when collecting numeric data, respondents have to provide a numeric response, and the range of possible responses may be limited:

How many glasses of wine did you drink last night? ____

Open-ended questions are the easiest type of question to ask. However, they should generally be used sparingly,[1] as they take longer for respondents to answer and many respondents will put so little effort into answering the questions that their data becomes unhelpful.

Single Response Questions

Are you...

o Male

o Female

When writing a good single response question it is important to:

  • Ensure that each person will only fit into one and only one category.
  • Ensure that each person can find an appropriate category. Often this is achieved by leaving an “Other” option. It is typically a good idea to require that the respondent type what they mean into a box, as otherwise you can end up with a lot of responses that are not intepretable.
  • Where the question forces people to compare the options (e.g., Which of these best describes your personality?", provide a maximum of seven responses, as otherwise by the time people have gotten to the end of the list they will have forgotten the earlier responses).
  • Where the question is only requiring a simple act of recall (e.g., Where did you go for your last holiday), order the responses alphabetically so they can quickly find the appropriate response (this is particularly important when there are large numbers of options).

Multiple Response Questions

This question type is also known as a pick any or a ‘multi’ or ‘multicode’ question.

Which of the following have you bought in the past week? Tick all that apply.

[] Coke

[] Pepsi

[] Fanta

[] None of these

Grid Questions

There are a variety of questions which are referred to as ‘grid’ questions. They differ in terms of how they permit the respondent to choose options, with common alternatives being to limit respondents to choosing a single option in each row, allowing them to tick multiple options in each row, sliders and text boxes for entering numeric responses.

One response per row

Please rate your satisfaction with the following restaurants

Low Medium High
McDonald’s o o o
Burger King o o o
Wendy’s o o o

Multiple responses per row

Which of these brands are 
        Fun  Sexy  Masculine 
Coke    []    []       []
Pepsi   []    []       []
Fanta   []    []       []

Typically, such questions are best instead asked as a series of multiple response questions, as respondents can be daunted or confused by such a grid layout.

Numeric responses

In the past month, how many flights did you take on...
                  Qantas  United  SAS
Economy           ______  ______  ______
Premium Economy   ______  ______  ______
Business Class    ______  ______  ______
First Class       ______  ______  ______

Date Questions

Date questions can either be similar or open-ended questions, such as in the example below, or can use a data selector (such as a calendar).

What is your date of birth?

____ / ____ / ____

Ranking Questions

The simplest ranking questions use a series of open-ended questions: Rank the following brands according to how much you like them... Please place a 3 next to the brand you like most, a 2 in your next preferred brand and a 1 next to your least preferred brand.

Coke ____

Pepsi ____

Fanta ____

More sophisticated versions use dragging and dropping.

Constant-Sum Questions

A constant-sum question asks the respondent to allocate a specific number of points, tokens, dollars or some other quantity across different alternatives. For example:

If you had $10 to spend on soft drinks, how much would you spend on each of the following?
__ Coca-Cola
__ Diet Coke
__ Coke Zero
__ Fanta

More advanced question types

There are many more advanced question types. Some of these are described in Advanced Questions and Questionnaires.

Phrasing the Question

Writing a question involves selecting a question type and phrasing the question. For the novice researcher, the secret to wording questions well is to copy others’ work. Plagiarism is at the heart of asking good questions, as there are many, many ways of wording a question poorly. The easiest way to get across the basics is to look at a few examples of bad questions (all from real studies): What is your principal brand of soft-drink? What problems can you see with this question? ‘Principal’ is an overly technical word. A better word than principal is ‘main’. However, this does not remove the ambiguity from the question: is it asking which one is liked the most, or bought the most often? Such a question will likely end up measuring brand salience (which brands come to mind) rather than anything else.[1]

And another example:

Do you prefer…  
[] Beer  
[] Wine  
[] Spirits  
[] Non-alcohol drinks

There are lots of problems with this question. What about cider drinkers? They are ignored, as the categories are not exhaustive. Furthermore, people who do drink are likely to drink different types of alcohol at different times, such as beer after sport, wine with dinner and spirits after dinner; how should such a person answer the question?

Key principles when phrasing a question

Principle 1: The question must be answerable

The least-educated person who is likely to have to answer the question needs to be able to figure out how to answer the questions accurately.

Principle 2: The question must not be ambiguous

The issue is not one of respondent confusion, as this is already addressed by the first principle, and respondents are remarkably adept at answering incomprehensible questions (presumably because this is the only way they can finish many questionnaires). The problem with ambiguity is to avoid situations where, once we have collected the data, we cannot discern what it means. For example, one study asked flyers about the importance of getting ‘discounts’ and found that it was important. However, the research did not clarify if the flyers were happy with a 2% or 20% discount. As a result, the research was ambiguous and could not be used to derive valid insights. To avoid being ambiguous, questions need to focus on the current, the specific and the real.[2] A trick to achieving this is to focus on writing questions around the 5Ws – who, what, where, when and why.[3]

A good way of checking for ambiguity is to use think aloud interviews, in which real respondents are asked to answer the question, but are required to verbalize all their thoughts while answering the question.

Principle 3: Incentive compatibility

The third principle is that questions must be incentive compatible, which is a term of art in economics, and it means that questions need to be written in such a way that people have an incentive to provide honest data. Consider the question:
Are you aged under 35? [] Yes [] No

If asked as a screener at the very beginning of a questionnaire, respondents conclude that an answer of No will cause them to be screened out of the study (i.e., asked no more questions). Consequently, when this question is asked at the beginning of questionnaires where people are being paid more if they complete the whole questionnaire, some respondents lie and pretend they are aged less than 35, when they are actually older. By contrast, if asked at the beginning of a questionnaire where people are not being paid to do the study, people aged under 35 pretend to be older, as this becomes a polite way of refusing to participate. An incentive compatible way of screening people based on being aged under 35 instead asks:

How old are you?
[] Under 18 
[] 18 to 24 
[] 25 to 34 
[] 35 to 44 
[] 45 to 54 
[] 55 to 64 
[] 65 or more

and then screens people out if they select one of the older age groups. It is “incentive compatible” because there is no obvious incentive to answer the question dishonestly.

Imagine yourself charged with the problem of pricing an iPad before they were launched. How much should you charge? There were no competitors, and thus no straightforward way of working out even a ballpark price point. A simple approach to this problem is to ask people how much they might pay:

What is the most you would pay for the product?.

The amount that people nominate is referred to as either the reservation price or their willingness-to-pay for the product. The analysis then proceeds by assuming that people will buy a product if its price is less than or equal to the amount nominated by the respondents.

Such questions are not not incentive compatible. Imagine that you loved the concept and would be prepared to pay $2,000 for an iPad. Would you tell this to an interviewer? If you did, it could result in the product being sold at $2,000. Whereas if you lied and said $1,000, you may end up with the product being sold for less and you not having to spend so much. Sure some respondents may be honest, either because they are by nature honest or because they believe that by being honest they will maximize their chance of the product they wish to buy being sold, but others may lie. Observing that respondents have an incentive to lie does not prove that they will lie (unless one is an economist). However, it is better to figure out ways to conduct research that are incentive compatible, as then the risk is removed (this is discussed in Advanced Questions and Questionnaires).

Frequency questions

Perhaps the most frequent mistakes made by novice researchers relate to frequency questions. Consider this question:

How often do you go to the cinema?
[] Never  [] Rarely  [] Sometimes  [] Often

What do the scale points of RarelySometimes and Often mean? One person’s often may be another person’s sometimes. When measuring quantities it is always important to give people precision about time and frequency, such as:

Thinking about the last twelve months, how many times did you go to the movies?
[] None 
[] Once or twice  
[] 3 to 6 times 
[] 7 or more times

A problem with such a question is that respondents may not be able to answer it accurately because they will not remember the precise number of trips to the cinema from the past twelve months. Such a reservation is justified. A solution is to reduce the time interval:

Thinking about the last two weeks, that is, everything from today back to Wednesday two weeks ago, how many times did you go to the movies?
[] None  
[] Once  
[] Twice  
[] 3 to 6 times 
[] 7 or more times

Asking about such a short time period creates a different problem. Attendance at the cinema is sporadic for most people. There is a big difference between somebody who goes six times a year and somebody who does not go at all. However, asking about the last two weeks can result in such people being grouped together, and thus it is better to ask about the longer time interval.

So, what is the best way to ask about frequency of going to the cinema? A question like:

Thinking about the last twelve months, how many times did you go to the movies?
[] None  
[] Once or twice  
[] 3 to 6 times 
[] 7 or more times

perhaps remains the best approach, as even though many people will not be able to answer with complete accuracy, most of the people who have not gone will be able to select None and most that have gone more than seven times will be able to select 7 or more times, whereas with the vaguer Rarely, Sometimes and Often categories, we can not make any deductions about what they actually mean.

Another example:

How often do you visit the web pages of this site?
[] Several times a day
[] Once a day
[] Once a week
[] Once a month
[] Less often than once a month
[] This is the first time I have visited

These categories are not mutually exclusive (anybody who can tick the last category can also tick the second last category). They are also not exhaustive. What should somebody answer if they visit two or three times a week?

Programming Instructions

Once the basic question wording has been finalized it needs have instructions added so that the person who has to program the questionnaire in the data collection software can ensure that it works as intended. Traditionally, these instructions are entered in capitals, italics or another color. In addition to the instructions, sometimes people enter specific values for each of the options. In the example below a Student is represented by a 1; this information is used to determine how the categories are represented in the resulting data file.

Example

Q2.	ASK IF AGED 15 OR OVER:
        To make sure we have a cross section of people in this study, we need to know your work status.
        Which of these best describes your ''work status''?  SINGLE RESPONSE         
	Student	                1	GO TO Q4
	Homemaker	        2	GO TO Q4
	Retired	                3	GO TO Q4
	Not working	        4	GO TO Q4
	Part-time worker	5	GO TO Q4
	Full-time worker	6	
	Don’t know/refused	7	END SURVEY

Ordering the Questions in the Questionnaire

Questions in a questionnaire are ordered so as to reduce the time taken to complete the questionnaire and to reduce order effects.

Minimizing the time to complete the questionnaire

Screeners

A screener is a question that can be asked at the beginning of a questionnaire with the objective of determining eligibility. For example, if conducting a study on cola drinkers, then a question about cola consumption is asked at the beginning of the questionnaire so that non-cola consumers are not asked the rest of the questionnaire (are screened out).

Skips

A skip is an instruction in a questionnaire which makes respondents skip questions for which their answers are not required. For example, if a person has no children then they should skip a question asking about the ages of their children.

Filters

Options that are not relevant to some respondents should not be shown (filtered). For example, if a consumers has never heard of a product they should not then be shown that product in a following question asking which products they consume most often.

Multiple response questions and grids

Questions with similar structures should be grouped together. For example, true/false questions can be grouped together as multiple response questions; questions which share a response scale can be grouped into grids.

Deleting questions

The shorter the questionnaire the better. Consequently, questions should only be included that are believed to be essential.

Minimizing order effects

An order effect occurs when people give answers in questionnaires that are influenced to an extent by the order with which the questions, or options in a question, are presented to respondents.

Knowledge and salience

The simplest type of order effect occurs when earlier questions unintentionally educate a respondent, or, make them think about a particular thing. For example, if a question that asks people if they have heard of Qantas and then, later in the questionnaire asks them to type the names of all the airlines for which they can recall recently seeing TV advertisements, there is a good chance that Qantas will get a higher result in the latter question than otherwise.

Leading questions

Leading questions cause respondents to change their opinions, which can effect either the answer of the question that is leading, or, the answers to following questions. Consider the following from the classic British sitcom Yes Prime Minister:[1]

Sir Humphrey: “You know what happens: nice young lady comes up to you. Obviously you want to create a good impression, you don’t want to look a fool, do you? So she starts asking you some questions: Mr. Woolley, are you worried about the number of young people without jobs?”

Bernard Woolley: “Yes”

Sir Humphrey: “Are you worried about the rise in crime among teenagers?”

Bernard Woolley: “Yes”

Sir Humphrey: “Do you think there is a lack of discipline in our Comprehensive schools?”

Bernard Woolley: “Yes”

Sir Humphrey: “Do you think young people welcome some authority and leadership in their lives?”

Bernard Woolley: “Yes”

Sir Humphrey: “Do you think they respond to a challenge?”

Bernard Woolley: “Yes”

Sir Humphrey: “Would you be in favour of reintroducing National Service?” [A requirement that people join the army after leaving school.]

Bernard Woolley: “Oh…well, I suppose I might be.”

Sir Humphrey: “Yes or no?”

Bernard Woolley: “Yes”

Sir Humphrey: “Of course you would, Bernard. After all you told me can’t say no to that. So they don’t mention the first five questions and they publish the last one.”

Bernard Woolley: “Is that really what they do?”

Sir Humphrey: “Well, not the reputable [market research firms] ones no, but there aren’t many of those. So alternatively the young lady can get the opposite result.”

Bernard Woolley: “How?”

Sir Humphrey: “Mr. Woolley, are you worried about the danger of war?”

Bernard Woolley: “Yes”

Sir Humphrey: “Are you worried about the growth of armaments?”

Bernard Woolley: “Yes”

Sir Humphrey: “Do you think there is a danger in giving young people guns and teaching them how to kill?”

Bernard Woolley: “Yes”

Sir Humphrey: “Do you think it is wrong to force people to take up arms against their will?”

Bernard Woolley: “Yes”

Sir Humphrey: “Would you oppose the reintroduction of National Service?”

Bernard Woolley: “Yes”

Sir Humphrey: “There you are, you see Bernard. The perfect balanced sample.”

Guiding and educating respondents

The opposite to leading a respondent is to guide them, as occurs in Probing and Laddering. In studies for new products it is routine to progressively introduce more information about the products throughout the questionnaire, so that the respondent is gently introduced to the information and also so that the researcher can understand the incremental impact of the additional information.

Fatigue

Harder questions should generally be asked towards the beginning of the questionnaire, so that if respondents become fatigued while completing the questionnaire, the fatigue occurs after they have already completed the more challenging questions. The order of questions can also be used to maximize the energy of the respondent, mixing boring questions with more interesting questions (e.g., projective questions).