Chapter 6 Flashcards
Surveys and Observations
The Butterfly Ballot
- Confusion over Palm Beach Count ballot
- The central problem: Poorly designed questionnaire measure which changed course of US history (Bush won over Al Gore)
- Similar challenge for Psychology Research: assess beliefs/moods/intentions/etc., make unobservable into “countable,” assign “values” to psychological states
Measuring Psychological Constructs
- Converting feelings, attiudes, beliefs, and thoughts into numbers
- tallied, averages, divided, correlated; reported verbally; or observed indirectly (neuroscience)
Construct Validity of Surveys and Polls
- Many people may have been asked to participate in surveys/polls, method of questioning people
- Online surveys, in person, written interviews
What is construct validity?
The degree to which a test or measurement tool accurately captures the theoretical concept it is intended to measure
Choosing Question Formats
Open-ended questions
- People can respond in any way they like and can say anything
- Use their own words
- Unpredictable responses
- An example of a self-report question format
- Ex: As you know, this study is primarily concerned with quality of life. What does quality of life mean to you?
Choosing Question Formats
Structured Self-Report Questionnaire
- Converts ideas to numbers
- Self report inventories and behavioral ratin scales
- Characteristics: Standard questions, everyone sees the same test, forced-choice format, highly reliable
What are the strengths of structured self-report?
- Specific administration rules
- Standardized methods for scoring
- Strict guidelines for interpretation
- Strict rules for reporting of results
Fixed Alternative Question aka Fixed Choice
- respondent selects answer from a set of specified responses
Question Formats: Forced Choice
Likert Scales
Respondent indicates extent to whcih they agree or disagree with the statement (strongly disagree, disagree, somewhat disagree, etc)
Question Formats
Semantic differential format
- When there is an anchor at both ends
- Nothing in middle, left to interpretation
Wording Questions Well
Question Wording Matters- Delaware Voters ex.
- Looked at support for voter ID law
- Randomly assigned one of 3 versions of a question
- Leading questions: wording leads people to answer a particular way
Wording Questions Well
Double-barreled Questions
- How much do you love vacation and final exams?- these questions have poor construct validity
- Should instead ask one thing at a time, don’t ask participants to evaluate 2 different things with one response
- Break complex questions into 2-3 simpler questions
Wording Questions Well
Keeping it simple
- Avoiding complexity: one thought or concept at a time
- Avoid negations: “Not” can cause confusion, especially double negatives
- Use similar words instead (not standing–> sitting)
Easter egg: bad one= i do not-should not-have not
REMEMBER
need to watch ted talk on are we in control of our decisions by dan ariely
Wording Questions Well
Question Order
- The order of questions can also affect responses
- One of the best ways to control for this is to randomize question order and create several versions of your survey
Wording Questions Well
Use informal Language
- Consider your intended audience: age/generation, reading skill/level of education, don’t patronize/oversimplify
- No psychology jargon: Hallucinations–>hearing voices, anorexic–>body image, aspirations–>wishes/dreams
Wording Questions Well
Forced Choice and Variance
- Choosing yes or no: one of two responses is the only option
- Problems: may not provide enough information, may not have enough variance, liking apples vs oranges
Wording Questions Well
Variance is Desirable
- Don’t want questions that are answered the same way by everyone. Ex. Do you find puppies cute?
- Recall the Goal: to study differences/changes. People’s answers to questions must vary. Hard if everyone gets 100% or 0% on the test
Wording Questions Well
Variance is desirable- restriction of range
- Restriction of Range: people’s score measure little variation/ few differences
- floor effects: everyone responds at a low level on your measure of DV
- ceiling effects: everyone responds at a high level on your measure of DV
Why is restriction of range bad? What is the solution?
- Restriction of range is not desirable because it is hard to predict anything based on scores that measure a non-varied DV
- Could be your construct (ex puppies) or your measure
- Solution is to pilot your DV measure
Wording Questions Well
Response Bias
- Solutions to the problem:
- reverse-worded items
- neutral language in questions
- use “normalizing” questions (questions designed to make a participant feel like their experiences are common or typical)
- stressing anonymity
Wording Questions Well
Response Bias
- Social desirability bias: answering the way you think is appropriate/expected/desirable, to loaded/sensitive topics
- Response set bias (non differentiation), always giving middle-of-the-road answer (fence sitting), always agreeing (acquiescnce/yea-sayer), alsways disagreeing (naysayer)
Wording Questions Well
How to prevent response bias?
- Guarantee anonymity: participant answers never linked to their actual identity
- Researchers sees only the groups’ answers or see only participants’ secret code
Wording Questions Well
How many questions is enough?
- Sometimes: need only ask one questions
- Often-times: need several times to capture construct
- More is better for reliability, idiosyncratic answers tend to average out
Wording Questions Well
Sensitive to diversity
- Not taking typical for granted: your representative sample is heterogeneous; culture/gender/sexual orientation
- Language can’t alienate sub-groups: mother –> primary caregiver, spouse –> significant other
Wording Questions Well
Sensitive to personal topics
- Your wife is on the roof: when dealing w/ sensitive topics tread lightly (wording) and use questions that progress in intensity
- Easy/innocuous first, and then difficult/sensitive later to get more honest responses
Observer bias
- Observations can often be better than self-report
- Although it’s still work to make you observations reliable and valid
- Observer bias- when expectation influence interpretations of behaviors
Observer bias example
- participants (psychotherapists) watched the same video of a person answering questions about work experiences
- Group 1 was told this man was a patient, and his behaviors were rated as defensive ad frightened of his aggressive impulses
- Group 2 was told the man was a job applicant, rated him as attractive, candid, and innovative
Observer effects examples
- Students w/ maze bright and maze dull rats- rats labeled as bright performed significantly better simply because the researchers expected then to
- Clever hans: when someone or an animal sense what another person wants them to do without being given any deliberate signals. Named after clever hans, a horse that appeared to perform intellectual tasks like arithmetic and reading but was just responding to subtle cues from his trainer
How to prevent observer bias?
- Clear codebooks: instructions on how to rate behavior
- Using multiple observers: for interrater reliability
- Masked (blind design): researchers are unaware of purpose of study and/or who is in which group
Reducing reactivity
- People change behavior when they know they are being watched
- Blend in: unobtrusive observations, researcher is less noticeable
- Wait it out: let participants “get used” to the researcher, works in weight loss studies
- Measure behavioral results: measure traces of behavior/indicators of behaviors
