Lecture 4 - Experimental Design II Flashcards
What are the different kinds of validity?
- Internal validity
- External validity
- Statistical conclusion validity
- Construct validity
What is internal validity?
To what extent does the research tell us what we think it tells us
What are history effects (internal validity)?
An event (other than the manipulation) occurs between pretest and post-test and affects the study outcome
What are maturation effects (internal validity)?
Participants naturally change between pre-test and post-test
What is regression to the mean (internal validity)?
DV scores may vary up and down naturally over time and between individuals and groups.
What is testing effect (internal validity)?
Participants perform better at a post-test because they had practice
What is intrumentation effect (internal validity)?
Performance at pre- and post-test differs because different measures are used
What are selection effects (internal validity)?
Happens when random assignment fails (participants in different conditions are not equivalent)
What factor does mortality play in internal validity?
- Participants with certain characteristics drop out of the study (non-randomly)
- Probably doesn’t involve the death of any Ps, but they may leave the study for other reasons.
What is external validity?
The extent to which results can be generalised beyond the context of the study
What is ecological validity?
The extent to which results are generalised to other environments
What is temporal validity?
The extent to which results can be generalised to other times/ occasions
Why is it important to generalise findings to other populations?
Samples must be representative on theoretically relevant dimensions
What is the problem with having student samples?
- A problem when the process is expected to operate differently in a student population (e.g., social influence)
- Not a problem if we study a process that operates in the same way in all humans (e.g., perception)
What is gender bias?
Generalising from male Ps to people in general, e.g., Kohlberg’s moral development scale or research/advice around cardiac arrest
What is cultural bias?
Generalising to Ps from other cultures or backgrounds
What is statistical conclusion validity?
Is reduced when inappropriate statistical procedures are used
→ E.g. Assumptions made by a statistical test are not met by the data, such as normal distribution, scale type
What is construct validity?
- Adequate manipulations of IVs used
- Appropriate measures of DVs chosen
- In other words, operational definitions of IVs and DVs fit the purpose of the study
→ Allows researcher to test the effect of IV on DV
What is a between participant design?
–When the IV is an individual differences variable (i.e., self-selecting)
–When experience gained from participating in one condition makes it impossible to participate in another (e.g., deception)
What are the advantages of between participant designs?
Each participant is fresh and naive to the hypothesis
What are the disadvantages of between participant designs?
- More participants are needed to be recruited
- There may be unexpected differences between groups of participants
What is random assignment?
- Every participant has same chance of being placed in any of the conditions
- Objective is to spread important individual differences evenly across conditions, thus controlling for possible extraneous variables
- Select Ps and randomly allocate to group
- Goal is to spread important characteristics across groups
What are the possible issues with random assignment?
- Groups may not be equal size
- Random assignment doesn’t necessarily achieve even spread of important characteristics across conditions or groups
- Confounding variable?
What are three ways to create equal-sized groups?
- Block randomisation
- Stratified block randomisation
- Matching procedure, followed by random assignment
What is block randomisation?
- Allocate Ps at random to blocks
- Randomaly allocate Ps in each block to the conditions
What are the possible issues with block randomisation?
- Again, random assignment doesn’t necessarily achieve even spread of important characteristics across conditions or groups
- Confounding variable?
What is stratified block randomisation?
- Identify important characteristics and group Ps accordingly
- Allocate Ps to block based on all important characteristics at random and allocate to each condition or group in turn
What are the possible issues with stratified block randomisation?
- Still doesn’t necessarily result in even spread of important characteristics across conditions or groups
- Distinction between Relaxed and Anxious Ps not subtle enough?
What are the steps to a matching procedure?
- Step 1: Get a score for each person on the matching variable (e.g. pre-existing state of anxiety
- Step 2: Arrange scores in ascending order
- Step 3: Create five pairs of scores; each pair consists of Ps with adjacent scores
- Step 4: For each pair, randomly assign one P to Group 1 and the other to Group 2. We have now effectively controlled for pre-existing state of anxiety.
What is a within-participant design?
- Participants take part in two or more conditions
- Comparison within the same group
- Also known as repeated measures
When is within-participant used?
- When conditions require brief time to test but extensive preparation (e.g., perception research, psychophysiology)
- When population of interest is small
What are the advantages of within-participants designs?
- Samples tend to be small
- More data from each participant
- Reduced error variance
What are the disadvantages of within-participants designs?
- Samples tend to be small (again)
- Threats to internal validity (e.g., maturation, history)
- Order effects
What are order effects?
- Practice effect
- Fatigue effect
- Carryover effect
What is practice effect?
Performance on later trials is improved due to practice
What is fatigue effect?
Performance on later trails is reduced due to fatigue
What is carryover effect?
- One sequence of conditions produces different results than another sequence
- Experience of doing condition 1 can carry over to condition 2
How do you control order effects?
Using more than one sequence of conditions
- Complete counterbalancing
- Partial counterbalancing
How does counterbalancing help?
- Systematically change the order so we can ‘average out’ any order effects across all conditions.
- That would give us more confidence that any differences in DV are due to our manipulation (IV) and not the order with which we present the materials/stimuli.
What is complete counterbalancing?
Every possible sequence is used at least once
- For example: (X!): 3! = 3 x 2 x 1 = 6
i.e. Six different sequences for three conditions:
What is partial counterbalancing?
Using a subset of the total number of sequences
- Either sampling from the list of all possible sequences
- Or randomizing the order of conditions for each participant
What is experimenter bias?
Refers to a situation where an experimenter unintentionally behaves differently towards Ps in one condition compared to another, based on their expectations about how Ps should respond or behave under the different experimental conditions
What is participant bias?
- Hawthorne effect
- Reactivity
- Demand characteristics
What is hawthorne effect?
- Participants change behaviour because they know they are being studied
- Describes that same effect but specifically when the people being observed are actually being studied for research purposes.
What is reactivity?
Describes a common phenomenon where people behave differently if they think they are being observed (e.g., if someone sees a ‘CCTV in operation here’ sign).
What are demand characteristics?
- Being a “good participant” - if participants guess the hypothesis, they may try to help confirm it
- Evaluation apprehension
- Acquiescence effect
How do you control participant bias?
- Deception
- Placebo
What is deception?
Keeping participants naïve to the purpose of the study (reducing demand characteristics) reduces evaluation apprehension and “good participant” effects
What is placebo?
- Condition where participants believe they’re getting a treatment, but in fact they aren’t
- Distinguishes placebo from ‘real’ effects
