Survey Research Flashcards
What is survey research?
- Data collection tool used to gather information from individuals
- Used across all research approaches
Experimental, correlational, qualitative - Typically, standardised measuring instruments
What it is and how it is administered is clear
Normative data is often available to provide comparison (normally from large standardised representation of population)
How are surveys typically presented?
As questionnaires or interviews
What are surveys often used to do to constructs?
Operationalise them
What can we use surveys to gather data on?
- Attitudes
- Behaviours
- Opinions
- Motivations
- Emotions
- Satisfaction
What type of data do surveys generate?
- Can be used to gather retrospective data (before COVID how frequently did you wash your hands), present data (now during COVID how often do you wash your hands) or future data (after COVID how often will you wash your hands?)
- Sometimes simply records demographic data (most surveys include demographic data
What are the two main purposes of surveys?
- information gathering
- theory testing and building
what types of information gathering is there?
Exploratory (explore a phenomena – doesn’t predict or explain why – what issues are present in a refugee population?)
Descriptive (describe – how many males and females are present in a population)
What types of theory testing and building are there?
Explanatory (explain something – do optimism and pessimism reflect states and traits)
Predictor (does the time a couple spend together before children predict relationship happiness later?)
What are the different types of surveys?
- Self-administered (questionnaires) Postal Online In person - Interview administered Telephone Face to face
When shouldn’t you develop questionairres?
If you don’t have to - you are better off using a pre-existing one or modelling it on a pre-existing one
When should you design a questionnaire?
- Design when:
No established tool exists to measure the phenomena of interest
When the reliability or validity of an established measure is in doubt
What are jangles and should you avoid them when developing a questionnaire?
Jangles are different labels for what are essentially the same thing. You should avoid them if you can
What should you do before widely administering your test?
- Piloting test on a small group of people before administering it more widely:
Administer to small sample
Identify problematic items and revise
Ask respondents for feedback
What are general design principles?
- Keep it short:
omit any questions which will not provide useful data
Don’t just have lots of data ‘just in case you need it’ - Make sure it’s readable:
your Ps must be able to understand the language used
Generally make it 8th grade level - Provide appropriate response options:
Avoid forcing your P to choose between more than one correct option, or not having any ‘correct’ options
What should the instructions be like in your survey?
- Clear instructions
- Standardised specific administration instructions
What should you consider with the order of your survey?
- Can be useful to divide a survey into sections blocked by topic or question type
- Screening – is the participant eligible
- Start with easy and engaging questions
- Use funnelling/branching questions if appropriate. E.g. Do you currently drive a car? If yes go to 14, if no go to 16
What are the demographics and how do they apply in a survey?
- Characteristics of the sample
- Typically contained in single section
- What could be included
Age
Gender (need to be aware it is non-binary)
Ethnic and/or racial background
Place of birth
Religion
Sexual orientation
Relationship status
Occupation or earnings - Only include relevant questions about demographics
What are open questions?
Ask questions and allow participants to respond however they allow
What are the pros of open questions?
Provide more detail
Rich data
Don’t impose assumptions
What are the cons of open questions?
Longer and more difficult to complete
Difficult to analyse responses (often subjective) – more difficult to standardise opinions
Respondents provide meaningless responses
What should you consider when deciding whether or not to use open questions?
- Only use if justified
- Ensure focus of question is clear
- Decide on analysis strategy from outset
- More useful for descriptive, exploratory worked
What are closed questions?
limited responses available
What are the pros of closed questions?
Quick to complete
Easy to analyse (objective)
Standardised responses
What are the cons of closed questions?
Can impose assumptions
Oversimplify complex issues
Frustrate participants if there are no responses for them
What should you consider when deciding whether or not to use closed questions?
- Ensure questions are unambiguous
- Provide clear response options
- Carefully consider style of response options
- More useful for explanatory or predictive work
When writing questions what should you avoid?
- double barrelled questions
- ambiguity
- negations
- double negatives
- value-laden or leading questions
- jargon
- response bias
What are double-barrelled questions?
Composed of two separate issues or topics but allowing just one response
Confusing
Impossible to determine true intention of respondent from data
‘Is your personal style influenced by magazines and TV?’
‘How satisfied are you with your pay and work environment?’
What is ambiguity?
Questions that are vague or over generalised
Open to interpretation
‘Where do you like to shop?’ (depends on what you are shopping for)
‘Is your personal style influenced by magazines?’ (what is personal style?)
What are negations?
- questions that include negatives
Introduce complexity and can be missed
Can result in non-response or misunderstanding
Example ‘Statistics should not be a compulsory unit of study’
Present as a positive statement instead
Be aware that double negatives can occur as a result of negative response options
What should you avoid to make sure that questions are neutral?
Avoid use of emotive language
E.g. ‘Do you believe that the media have a dangerous influence on women’s perceptions of themselves?’ should become:
‘Do you believe that the media have an influence on women’s perceptions of themselves?’
Avoid influencing Ps with leading questions
E.g. ‘Experts suggest that exposure to violent video games can result in behavioural problems in children. Do you agree?’ Should be:
‘To what extent do you believe that exposure to violent video games can result in behavioural problems in children?’
What is jargon?
Technical terms your participant won’t be familiar with
What are social desirability effects?
Participants responds in a positively biased way
Identify with a lie or socially desirability scale
E.g. ‘No matter who I’m talking to, I’m always a good listener’ people will want to answer in a positive way – test by using the same question phased differently and see if the participant is consistent in the way they response to questions
What is Response acquiescence and how do you spot it?
Tendency to agree rather than disagree
Include both positively and negative worded questions:
E.g. ‘It was easy to find what I wanted on the website’ ‘I had no problems navigating the website’ – want them to agree with one and disagree with the other. With response acquiescence they would agree/disagree with both statements
What do rating scales do and what are they particularly useful for measuring?
- Ask people to provide ‘how much’ judgements
- Particularly useful for measuring attitudes (use a number of different statements)
What are the different formats of rating scales?
Dichotomous, multichotomous Likert scale Graphic rating scale Non-verbal scale Ranking scale Semantic differential scale
What are dichotomous scales?
- Two response options
- Simplest type of quantification
- E.g. ‘yes’ ‘no’ responses
What are multichotomous scales?
- One response options: how many hours did you spend revising last week? <5 5-10 10-20 >20 - Multiple response options: which study activities did you engage in? Watching podcasts Making notes Reading textbook Reading papers
What are likert scales?
- Consists of a multi-point response
Often 5 point scales
Sometime 7 point scales - Aim to ensure equal spacing of response options
Use same wording for positive and negative sides of scale
What are some considerations with Likert scales?
Response acquiescence (overcome by using opposing statements that tap into the same attitude)
Potential introduction of double negatives
Verbal responses for all, or only for anchors?
Inclusion of neutral response – remove so Ps don’t have any option for ambiguity
What are non-verbal rating scales?
- Very useful with children and cognitively impaired individuals
- Point to the face that shows your answer
- Sometimes it’s necessary to include labels to clearly define the meanings of each category
What’s a ranking scale?
- Measures the relative importance of several items
- ‘Rank the anxiety you experience in the following group situations (from 1 most anxious, to 4 least anxious)
What is a graphic rating scale?
- Ps asked to mark along a continuous line which is anchored at each end
- ‘How would you rate the music lessons you received in school?’
- Line with not very enjoyable ———————– very enjoyable
- P needs to record a mark on the line where their opinion fits
- Record score by measuring where line was marked
What are semantic differential scales?
- More indirect measurement of attitude
Does not assume an attitude is a cognitive belief - Respondents indicate thoughts and feelings by marking a response on scale between bipolar or opposite adjectives
- E.g. Music lessons in school were:
- Valuable——————–wasteful
What are response rates and why are low response rates a problem?
The percentage of questionnaires completed and returned from those distributed (questionnaires often suffer from low response rates). Low response rates are a problem because they incur a significant cost
How can you maximise response rates?
Keeping questionnaires short
Keeping questionnaires simple and clear
Including a pre-paid envelope with postal surveys
Sending a reminder (after one or two weeks)
Offering an incentive (ethical considerations)
Typically why are there multiple items to measure a single variable, what is the problem with this and how would you calculate the overall e.g. attitude?
Possibly different facets of the same construct
Especially important with fuzzy constructs like attitudes
Minimises the impact of response errors
- Variable score is calculated
Total or average across items – (average is better for missing data than total)
- Better measurements with more items
But inattention might result from too many items
What should you consider about the number of response options?
- Too few will result in low sensitivity
- Too many will result in low reliability
What is psychometrics?
the science of measuring psychological constructs
What two criteria is quality assessed using?
- Temporal consistency
The extent to which our tool would provide the same results under the same conditions - Internal consistency
The extent to which out tool is measuring the construct we are interested in
What is construct validity?
Is the construct we are trying to measure valid? Does the construct itself exist
The validity of a construct is supported by cumulative research evidence collected over time
Each piece establishing an element of reliability/validity
Together, supporting the existence of the construct itself
In the short term construct validity can be assessed in terms of:
Convergent validity correlates with tests of related constructs
Discriminant validity: doesn’t correlate with test of different constructs
Some tests can have convergent but not discriminant validity
What are psychometric tests?
- Standardised questionnaires/tests designed to measure particular traits or abilities
Personal inventories
Cognitive ability tests
Measure of mental health status - E.g. a questionnaire at a doctors office
- Items are published as an inventory
- Norms available allowing for the interpretation of individual participants data
Expressed as standardised scores - Reliability is established, but validity is sometimes questionable
