Questionnaire Design Flashcards
Learning objectives of the lecture:
- Understand and be able to explain… “rules of thumb” for constructing questionnaires and questionnaire items
- Understand and be able to explain… the advantages and disadvantages of different types of response categories
- Be able to… format a questionnaire and different types of questionnaire items using Qualtrics
- Understand and be able to explain… the advantages and disadvantages of collecting data using online surveys
Rules of thumb for designing questionnaire items…
9 rules of thumb
- Make items clear
- e.g. “What do you think about the proposed changes to income tax?”
- should be clear and unambiguous
- should be precise so that ppts know exactly what the researcher wants an answer to - Avoid presuming questions
- asking “how many cigarettes a day do you smoke?”, for example, presumes that the respondent smokes - Use simple language
- e.g “has it ever happened to you that over a long period of time, when you neither practised abstinence nor used birth control, you did not conceive”
- keep it simple and clear
- keep intended audience in mind - Avoid double-barreled items
- e.g. “The government should cut expenditure on defence and spend more money on education”
- respondents could agree with one part and not the other - Respondents must be competent to answer
- e.g. “ATM has caused many redundancies in British industry”
- must possess knowledge necessary for answering the question
- must explain terms for those who might not know
- is the question likely to make sense? - Avoid using the word “not”
- e.g. “The government is not managing the economy well”
- easily overlooked, rephrase for clarity - Avoid using double-negatives
- e.g. “It is unlikely that the economy will not improve over the next 6 months”
- confusing, may lead to answers where the respondent meant to say the opposite
- make the subject of the question clear, so ppts know what its asking - Avoid using biased items
- e.g. “most reasonable people would agree that police require more powers to fight crime”
- implies that by disagreeing with the statement, they would not be a “reasonable person”
- to reword an item like this, use a statement or ask about perceived social norms - Avoid very mild or extreme statements that everyone will agree or disagree with - this would result in reduced variance in answers
- e.g. extreme - “doctors never listen to anything patients say”, mild - “once in a while, doctors might forget something a patient has told them”
- reduced variance would mean the results wouldn’t correlate with anything, no normal distribution
Guidelines for constructing questionnaire items:
Why are short items best?
Should be able to…
- read item quickly
- understand it’s intent
- give an answer without difficulty
How do we achieve variability in responses?
Using either lots of items or items with response scales.
Variability in answers allows co-variation between factors being investigated
0-100 response scales
e.g. One a scale of 0-100 how confident are you that you could restrain from smoking when drinking a cup of coffee?
Advantages;
- lots of possible variance
Disadvantages;
- may repsond in multiples of 5 or 10, reducing the number of response options/ variability in responses (unlikely to respond with 57 instead of 60)
- creates false precision - the difference between 36 and 38 is fairly meaningless
Visual analogue scales
e.g. Watching football on TV is…
Boring __________________ Interesting
- Mark the line
Advantages;
- Very sensitive, lots of possible variance
- Difficult for respondents to remember previous responses
Disadvantages;
- creates false precision - the difference between one point and another (close together) is fairly meaningless
Worded response categories
e.g. How helpful did you find the advice you were given?
Not at all Not very Somewhat Very
helpful helpful helpful helpful
Advantages;
- works well with young kids or older adults that might have difficulties with a likert scale (words and numbers together can be confusing)
Disadvantages;
- view held goes from one extreme to another in 4 options, resolution of answer options is poor (huge psychological space between responses)
Likert Scale
e.g. Exercise is an essential component of a healthy lifestyle.
Agree 1 2 3 4 5 6 Disagree
Exercise is an essential component of a healthy lifestyle.
Strongly Moderately Mildly Mildly Moderately Strongly
Agree Agree Agree Disagree Disagree Disagree
Exercise is an essential component of a healthy lifestyle.
1 2 3 4 5 6
Strongly Moderately Mildly Mildly Moderately Strongly
Agree Agree Agree Disagree Disagree Disagree
Advantages;
- uses a combination of words and numbers, words used as anchors to understand what the numbers mean
Disadvantages;
- always good to run a pilot study when using likert scales to make sure the respondents know how to answer the questions, some struggle more than others as words and numbers together can be confusing
Semantic differential scale
Semantic differential scales are often used to measure attitudes, using adjective pairs representing opposite ends of a continuum.
e.g.
Exercise is…
Harmful -3 -2 -1 0 +1 +2 +3 Beneficial Bad -3 -2 -1 0 +1 +2 +3 Good
Advantages;
- easy to understand
- reliable responses
- versatile
Disadvantages;
- no right answer
- difficult to know knowledge level of respondent (why they think that)
Thurstone scaling
A Thurstone scale consists of a number of items that have been designed to represent differing levels of an attribute (e.g. conservatism).
- A large pool of items (around 100) ranging in their favourableness are assessed by judges (50 or so).
- Each judge sorts the items into a set number of piles (e.g. 7, 9 or 11) representing differing levels of favourableness which are equally spaced along the attitude continuum.
- The items in the piles are scored 1 to 11 (or 7 or 9) and for each item a median value is calculated.
- On the basis of the judge’s ratings a final choice of 20 or so items is made which cover the entire range of attitudes and are equally spaced along the attitude continuum.
- The items are then presented in a random fashion to respondents who indicate which items they agree with using an agree-disagree response format.
An attitude score is obtained by taking an average of the median values the respondent endorsed.
Thurstone scales assume that an individual will only agree with items around his/her scale position, disagreeing with those more extreme on either side.
Advantages;
- easy to complete, just have to agree/ disagree with statements
- easy to develop alternative forms of the scale, lots of item to pick from
Disadvantages;
- labour intensive
- need lots of judges
- need lots of time and money
- can be individual differences in ratings between judges
Worked example of Thurstone scaling…
- Come up with a list of statements representing levels of conservatism
- The statements (around 100) are places on a continuum (of political affiliation), say 1-7 by 50 or so judges
- The statements sorted to one end of the continuum e.g. in the 7 pile or the 1 pile, would be very conservative or not conservative at all, so conservative ppts would agree with them wholeheartedly or not at all
- pick out 20 items in the end
- 50 ppts then indicate agreement with each of the 20 items
An attitude score is taken by calculating an average of the statements they agreed with (assuming they only agreed with statements that reflected their political affiliation)
Guidelines for questionnaire format
Maximise white space
Use an appropriate font and size
> Maximise white space
- Don’t try to squeeze as many items as possible onto a page
- A longer, well-spaced questionnaire is easier and quicker to complete han a short cluttered questionnaire
> Use an appropriate font and font size
This optimises the “user experience” of the questionnaire
Guidelines for questionnaire format
General Instructions
- Who is it from?
- Purpose of the questionnaire?
- Why them? Why they should complete it?
- How long to complete?
- Is it anonymous and/or confidential?
- How to complete the questionnaire - instructions.
- How to return the questionnaire.
Give the respondent some background information, they don’t want to be left asking questions about why they were completing the questionnaire - it’s only polite
Guidelines for questionnaire format
Question order
> Randomisation of items
- May result in a confusing questionnaire, could block together similar questions to avoid this
- Counterbalancing avoids order effects (either of whole item order or just sections)
> Leave duller questions (e.g. demographics) to the end.
There is lots of debate surrounding item order
Guidelines for questionnaire format
Add instructions to each section
- Helps respondents to make sense of the questionnaire and makes it appear easy to follow and complete.
Examples:
- “This section is concerned with your thoughts about the causes of unemployment”
- “We would like to ask you a few final questions about your personal circumstances, so we can see how different types of people feel about the issues we have asked you about”
- Promotes continued understanding of the questionnaire.
- Orientates them, makes answering easier
