WEEK 3 - Design and Controls Flashcards
What is internal validity?
How strongly can we assure that changes in our DV are due to the changes in our IV and not something else we haven’t controlled for (extraneous variable)
What is external validity?
How generalisable are our findings? (tied in with the representativeness of our sample)
How representative of the real world is our study (tied in with how artificial our study is)
What is the relationship between internal and external validity
The more stringently we try to control internal validity the potentially more artificial our study becomes and hence less representative of reality and therefore less generalisable and therefore less externally valid
What are the four steps to internal validity?
- Sound operationalisation of our DV (measure should be reliable and valid)
- Strong design logic
- Sound operationalisation of our IV
- Consideration and use of appropriate remedies to control for extraneous variables
What are the three types of research designs?
- Experimental - cause and effect, manipulation, control
- Quasi-experimental - Similar to experimental designs, however less randomisation of key independent variables.
- Non-experimental - Relationships but NOT Cause and Effect (quantitive or qualitative)
What is an experiment
An experiment is strictly controlled study in which the ultimate aim is to infer causality on the part of the IV and DV
In other words, in order to say that changes in our IV CAUSED changes in our DV we need to make sure that any and (hopefully) all (if not most) alternative explanations have been accounted for
Experimental Vs Quasi-Experimental
Experimental design involves experimental manipulation directly determined by researcher in controlled environment
Quasi-Experimental: where manipulation not controlled by researcher e.g., where levels in IV determined by participant characteristic i.e. individual difference manipulation e.g. demographics, self report measures.
What are the two ways an IV can be manipulated?
Experimental manipulation
Individual difference manipulation
What is experimental manipulation?
Experimenter determines which level of the IV a participant is tested at:
- event manipulation
- instructer manipulation
What is individual difference manipulation?
A characteristic of the participant determines the level of the IV of which they are tested (not strictly an experiment - quasi-experiment)
- demographics
- self-reported measures
What are the types of experimental research designs?
Repeated measures (within groups, dependent group) - each participant tested at each level of the IV
Between groups (intended groups) - each participant only tested at one level of the IV
Mixed: more than one IV with at least one IV manipulated between groups and at least one within groups.
What are repeated measure design’s
- each participant tested at each level of the IV
- Need less subjects
- More sensitive design (easier to detect the effect of interest, as individual differences controlled for)
- Cant always use this design
What are between groups designs
- Each participant tested only at one level of the IV
- less sensitive design
- Often forced to use this design if:
–> If the IV is an individual difference variable (eg. gender)
–> if participating in one condition precludes participating in another
What are mixed designs?
- more than one IV with at least one IV manipulated between groups and at least one within groups
When can you not use repeated measure designs?
- If participants at one level of the IV precludes later participating at another level, for example by causing permanent change in the participant e.g. exposure to one therapy may cause permanent improvement
- It is not physically possible for a participant to participate in all levels e.g. cant be both short and tall
When can you not use a between group design?
Cant use a between-group design if you wish to detect changes in individuals across time example- learning studies or developmental studies
What is a factorial design?
- A design with more than one IV
- May have all repeated measures IV’s or all between group IV’s
- Mixed: more than one IV with at least one IV manipulated between groups and at least one within groups.
- Allows examination of interplay between two or more IVs and the splitting up of these effects into interactions and the main effects
What are strengths of a factorial design?
- more than one IV allows for a more precise hypothesis
- Control for extraneous variables by including it as an independent variable
- ability to determine the interactive effect of two or more variables
What are factorial designs main effects?
- The influence of one independent variable on the main variable
- One main effect for each IV in a study
example: can look at the main effect of age and the main effect of alcohol
What are factorial designs interactive effects?
- looks at whether the effect of one IV is different at different levels of another IV
What is the factorial design notation?
number of numerals = number of IV’s
each number indicates the number of levels for each IV
for example:
2x2 design
IV1 = 2 levels
IV2 = 2 levels
2x 3 design
IV1= 2 levels
IV2 = 3 levels
What are weaknesses of factorial designs?
- using more than two independent variables may be logistically cumbersome
- examples
- 2 x 2 design = 4 cells, 2 main effects, and 1 interaction
- 2 x 3 design = 6 cells, 2 main effects, and 1 interaction
- 2 x 2 x 3 design = 12 cells, 3 main effects, and 4 interactions
- higher-order interactions are difficult to interpret
In order to maximise our chances of getting a true picture of how our independent variable affects our dependent variable we want to…
(separate and compress)
- Maximise the impact on our dependent variable that is related to the independent variable
–> increase between group/condition/level variation
-Minimise variation in our dependent variable that is not related to our independent variable
–> compress within group/condition variation
What is seperate?
maximise the variation between groups/levels of the IV
What is compress?
minimise the variation within groups/levels of the IV
How do you determine levels of the IV?
When considering the operationalisation of an IV you need to consider how to ensure you include as extreme or distinctly separate levels of your IV as possible
If you are looking at the impact of alcohol on performance you would make sure your conditions/levels differ by enough micrograms of alcohol to have a discernible impact
You might choose to include a number of intermediate levels of an increasing intensity IV to help pick up complex patters like linear or curvilinear impacts if you believe they may exist
How is compression achieved?
by reducing error variance
What are the three important sources when aiming to compress a study
- Measurement error
- Individual differences (people differ more within the groups rather than between)
- Other factors that influence peoples scores
What are the two forms of extraneous variables?
Noise creating (nuisance)
confounding
What are nuisance variables?
randomly impact the DV, not related to the IV, but potentially create extra variation in the DV not due to the IV, reducing power
What are confounding?
- systematically impact the DV, related to the IV, potentially
explaining changes in the DV that you would be expecting the IV to make, - want to control for this by eliminating, keeping constant or building into study so can measure impact.
- Confounds reduce internal validity.
- Need to minimize in all forms of research (especially experimental).
What are the troubles with between group desings?
The trouble with Between Groups designs is precisely the fact that they are between groups. Two separate groups of people could differ on a whole range of things. Both relevant and irrelevant to the study at hand
What is selection?
The process of assigning people to experimental conditions.
Types:
self assignment
experimenter assignment
Arbitrary assignment
Random assignment
What is self assignment (in selection)
Subject selects which treatment group
What is experimenter assignment (in selection)
Experimenter selects which treatment group
What is arbitrary assignment (in selection)
- Selection based on seemingly non-relevant criteria
- All above have potential for bias to confound results
What is random assignment (in selection)
- Best to have no criteria for selection
- Provides maximum insurance that groups are equal
- Eliminates systematic differences between groups
- Doesn’t eliminate extraneous variables, but randomly distributes them across groups
- Given sufficient N (sample size) controls for the things you know about as being possible sources of bias
What is matching?
- Use of a variety of techniques to equate participants in the
treatment groups on specific variables - should be done with variables thought to be related to the IV or may confound IV e.g., intelligence, age, gender
What are advantages of matching?
- controls for the variables on which participants are matched
- increases the sensitivity of the experiment
What is individual/precision matching?
creating pairs whose subjects have identical scores on matching variable/s
What are the advantages and disadvantages of precision matching
Advantages:
* groups equated on potential extraneous variable
Disadvantages
* identifying the variables on which to match can be difficult
* difficulty matching participants increases as the number of variables on which to match increases
* decrease in generalizability of results?
If you can’t match variables what should you do?
- Rather than matching you can always just measure the extraneous variable – thus turning it into another IV in your design.
- This does not make analysis harder, but logistically can
be much easier. - Not all variables have to be measured in groups – you
can control for extraneous variables that are measured
on scales. When we control an extraneous variable by measuring it as an IV, and it is measured on a scale – we often call this a “covariate”
Why are repeated measure designs good?
A repeated measures design eliminates the problem
of group differences arising from individual differences in
group make-up
Requires fewer participants to have good statistical power (because it is a more refined measure of the impact of IVs with a major source of error removed)
What are some problems with repeated measure designs?
- Participants may perform differently in each condition based on their prior experience of the study
- repeated measure designs can lead to sequencing effects such as order effects and carry over effects.
What are order effects?
Order effects come in two forms, either practice effects or fatigue effects.
practice effects- If the DV is performance-based (such as reaction time) then differences/improvements may occur simply due to the practice rather than being due to exposure to different conditions of the IV
fatigue effects - where repeated completion of the task may lead to boredom or tiredness (especially if conducted in quick succession)
What is a solution to order effects
counterbalancing
What is counterbalancing?
When we break our sample into subsets who will experience the different conditions in different orders
- sequence of conditions is randomly determined for each
participant
example:
IV with two levels
Two possible sequences (AB, BA)
Each participant randomly assigned to one sequence
What are the different types of counterbalancing?
- Intra-subject counterbalancing
- Complete counterbalancing
- incomplete counterbalncing
What is intra-subject counterbalancing?
- Participants take treatment in more than one order
Example:
IV with two levels
Two possible sequences (AB or BA)
Each participant exposed to ALL sequences
Half the participants randomly allocated to (AB, BA) and other half allocated to (BA, AB)
What is complete counterbalancing?
- all possible sequences of treatment conditions are used
- Participants are randomly assigned to a sequence
- N! = N multiplied by each number below it
Example:
* 2! = 2 x 1 = 2
* AB, BA
* 3! = 3 x 2 x 1 = 6
* ABC, ACB, BAC, BCA, CAB, CBA
* 4! = 4 x 3 x 2 x 1 = 24
* number of possible sequences rapidly becomes very large
What is incomplete counter-balancing?
- Also called Latin Squares
- Most commonly used technique
- Not all possible sequences are used
Criteria
- each treatment condition must appear an equal number of times in each ordinal position
- each treatment condition must precede and be followed by every other condition an equal number of times
Example:
participant Sequence
1 A B D C
2 B C A D
3 C D B A
4 D A C B
What are the two types of carry over effects?
Simple
Differential
What is a simple carry over effect
When performance on the DV in one condition is contaminated by effect of the previous condition
What is differential carry over effects
Where carry-over effects of one condition of the IV differ depending on the order in which the conditions are completed.
Is counterbalancing always possible?
- No, counterbalancing is a great tool and allows you to control for order effects by providing data from a range of different orderings of measurement
However, repeated measure designs don’t always have conditions that lend themselves to counterbalancing
What happens when time is the IV?
A special case of repeated measures designs is when
time itself is an IV, i.e., where it is of interest to examine changes, or perhaps improvements over time, in response to some form of intervention
Participants provide a baseline measurement and then complete measurements at numerous time points
during and after the treatment program
What are some other issues with repeated measure designs?
- Maturation
- History
- Statistical regression
- Morality
What is maturation?
- changes due to natural development, natural expected improvements over time
- If participants improved over time the question becomes whether or not it was because of the IV or if that improvement would have occured anyways
What is history (external events)
- External events that affect participants during the study
- Social-historical-economic changes relevant to outcome
What is statistical regression?
- Refers to the tendency to move up or down towards the mean over time
- In other words, someone that was scoring below or above par on a measurement is potentially likely towards the mean of that variable over repeated measure
What is morality?
Not all participants who take part in the first measurement point of the study will remain in the study till the end
What are other threats to experimenters validity?
Experimenter effects
Participant effects
Situational effects
What are the experimenter’s effects?
Measurement issues
Control of recording errors
Attribute effects
Control of attribute errors
Experimenter expectancies (Rosenthal effects, Golem effects)
What are measurement issues? (experimenter effects)
- Problems with equipment or errors in manual recording of data between measurement points or between participants on the part of the experimenter. If this varies across conditions - could be confounded
What are control of recording errors (experimenter effects)
- Make researchers aware of making careful observations (training)
- Multiple data recorders (computers, tapes etc)
- Have participants make responses on a computer
What are attribute effects? (experimenter effects)
Participants respond differently to different experimenters in the study
What are controls for attribute effects? (experimenter effects)
- Use the same experimenter for all conditions of the experiment
- Standardise how experimenter acts (use a script)
What are experimenters expectancies
Experimenter expectancies as to the hypothesised response of participants to the manipulations. May lead to subtle differences in the way the experimenter interacts with participants which leads to differences in outcomes
What is the Rosenthal effect (experimenters expectancies)
differential attitude or attention conveyed to participant expected to respond most favourably to the study
What is the Golem effect (experimenters expectancies)
differential attitude or attention conveyed to participant expected to respond least favourably to the study
What are the participants effects?
- Demand Characteristics
- Social Desirability
- Hawthorne effects
What are demand characteristics (participant effects)
- Participants get an inkling about what the study is aiming to achieve and perform in a way that conform to those expectations
What is social desirability (participant effects)
Participant performs in a way they think will be most pleasing to the experimenter or will paint them in the best light
What is the hawthorn effect (participant effect)
participant improves or changes performance on the outcome purely as a function of the attention received for being in the study and not as a function of the nature of the
manipulations of the IV
How do you control for expectancy effects?
- Single-blind study
- Double-blind method
- Deception
What is a single-blind study (control for expectancy effects)
- The participant is not made aware of the true purpose of the study or the nature of the group in which they are in
What is a double-blind study (control for expectancy effects)
- neither the experimenter nor the research participant is aware of the treatment condition administered to the participant
What is deception? (control for expectancy effects)
- omission of or altering the truth of information given to the participant during a research study
- used when there is no other way to gain the knowledge and risk does not outweigh the benefit of the information
- must keep the false information constant for all participants
What are the controls of experimenter expectancies
Double blind method
Partial blind method
Automation
What is a double blind method (experimenter expectancies)
- neither the experimenter nor the
research participant is aware of the
treatment condition administered to the
participant
What is partial blind method (experimenter expectancies)
- a method whereby knowledge of each research participant’s treatment condition is kept from the experimenter through as many stages of the experiment as possible
What is automation (experimenter expectancies)
- the technique of totally automating the experimental procedures, so that no experimenter-participant interaction is required
What are situational effects?
- Situational effects refer to the impact of environmental and timing differences on participants’ outcome scores
- These may include time of day, weather/season, lighting, background noise etc
- These can be controlled by attempting to keep these constant for all participants
- Or if multiple measurement points are required, counterbalancing may help
What are control groups?
- Including a control group in your study allows you to directly examine the extent to which changes would have occurred without your intervention
- Make sure you endeavour to have an “equivalent” control group
- Sometimes hard ethically.
- Can be logistically hard at times.
- May have multiple control groups:
- No intervention
- Placebo intervention
- (to counter for Hawthorne effect)
Summary of within groups
- Minimise individual difference error
- Problems with order effects, practice, cumulative effects,
carry-over effects, history, maturation, attrition/mortality - Can control these things somewhat with counter- balancing, random order variation
Summary of between groups
- Eliminate problems of WG
- Individual differences come into play
- Ensuring comparability of groups/matching
What is the purpose of Non- Experimental Research
Description - simple description of a phenomenon,
e.g. How many people currently smoke cigarettes despite the recent anti smoking laws?
- Exploration - preliminary investigation of a research idea,
eg. Survey of reasons why people find it difficult to quit? - Explanation - examine hypothetical relationships
between variables of interest eg. What aspects of smoking
history predict an increased risk of heart disease (Nb. NOT CAUSATIVE -Correlational)
What are the reasons for using a non- experimental design:
Ethical
* In many instances the only way to test hypothesis
ethically.
* Subjects select themselves, they are not assigned to
conditions
Logistical
* Surveys – wide reach to participants
* Examining immutable characteristics
e.g, demographics (demographics cannot be manipulated)
What are the types of Non-experimental research
- Naturalistic observation
* non-invasive recording/measurement of variables
- Case studies
- intensive study of individual subject
- Survey research
- usually, but not limited to, questionnaires. May assist in
developing causative based models - Evaluation research
- examining effect of intervention (policy or practice)
Extraneous variables in non- experimental research
Just as in experimental research, extraneous variables can impact on a non-experimental project also.
- Confounds: Anything which correlates with both the IV and the DV in the way predicted by the hypothesis. In the real
world there are lots of sources of confounds. - Nuisance variables: Anything that correlates with the DV other than the IV itself (lots of possibilities here)
