4. Experimental designs and control Flashcards
what is an experiment?
An experiment is a strictly controlled study in which the ultimate aim is to infer causality on the part of the IV on the DV.
In other words, in order to say that changes in our IV CAUSED changes in our DV we need to make sure that any and (hopefully) all (if not most) alternative explanations have been accounted for
what is something we need to ensure that our study infers causality?
that it is internally and externally valid
what are the two main issues of validity?
internal validity and external validity
internal validity
how sound is the design, how strongly can we assert that changes in our DV are down to our IV and not other things we haven’t controlled for (i.e. extraneous variables)
external validity
how generalisable are our findings (tied in with representativeness of sample), how representative of the real world (tied in with how artificial our study is)
what is the relationship between internal validity, external validity and the artificiality of the study and its generalizability?
The more stringently we try to control or ensure internal validity, the potentially more artificial our study becomes and hence less representative of reality and hence less generalisable… and hence less EXTERNALLY valid.
what are the four steps to internal validity?
- Sound operationalisation of our DV
- Strong experimental design logic
- Sound operationalisation of our IV(s)
- Consideration and use of appropriate remedies to control for extraneous variables
are some weak experimental designs?
One group posttest only
one group pretest-posttest
posttest-only non-equivalent groups
what are the two ways of manipulating the IV?
experimental manipulation and individual difference manipulation
experimental maniplation
Experimenter determines which level of the IV a participant is tested at;
what are the two types of experimental manipulation
event and instructional
individual difference manipulation
A characteristic of the participant determines the level of the IV at which they are tested
three main types of strong experimental design
repeated measures, between groups, factorial (Mixed factorial)
repeated measures (within groups)
each participant tested at each level of the IV
Between groups
each participant tested at only one level of the IV
factorial
when theres more than one IV. May have all repeated measures IV or all between groups IVs
Mixed-factorial
more than one IV with at least one IV manipulated between groups and at least one within groups.
what does mixed-factorial designs allow?
Allows examination of interplay between two or more IVs and the splitting up of these effects into interactions and main effects
what are the strengths of factorial designs?
o more than one independent variable allows for more precise hypotheses
o control of extraneous variables by including as an independent variable
o ability to determine the interactive effect of two or more independent variables
what are the main effects of the factorial design?
o the influence of one independent variable on the dependent variable
o one main effect for each IV in a study
what is the interaction effect of the factorial design?
o the joint, combined, or “interactive” effect of two or more independent variables on the dependent variable
what are the weaknesses of factorial designs?
using more than two independent variables may be logistically cumberstone.
high-order interactions are difficult to interpret
how do you maximise the chances of getting a true picture of how the IV affects the DV?
maximise the impact on our DV that is related to the IV (increase between group/condition/level variation)
Minimise variation in our DV that is not related to IV (compress within group/condition variation)
how many levels must an IV have to allow comparison of performance?
at least two
what is the fundamental thing to consider hen operationalising the IV?
how do you ensure that you will measure the IV’s impact on the DV to its maximum effect?
how do you ensure that you will measure the IV’s impact on the DV to its maximum effect?
To this end you need to try to ensure you include as extreme or distinctly separate levels of your IV as possible
what are the two variables that need to be separated
DV and V
what is the variable that needs to be compressed?
extraneous variable
what are the two forms of extraneous variables?
noise creating
confounding
noise creating , extraneous variable
randomly impact the DV, not related to the IV, but potentially create extra variation in the DV not due to the IV, want to minimise this
confounding - extraneous variable
impact the DV, related to the IV, potentially explaining changes in the DV that you would be expecting the IV to make, want to control for this by eliminating, keeping constant or building into study so can measure impact
what do extraneous variables lead to?
error - variations in measurement due to unwanted, uncontrolled or immeasureable factors need to be minimised
what variables and between groups and repeated measures design bring?
a unique set of extraneous variabes
what is the trouble with between groups?
Two separate groups of people could differ on a whole range of things. Both relevant and irrelevant to the study at hand
how do you make the groups in a between groups experiment as similar as possible?
through careful selecting of participants to the levels of IV
what are the processes of assigning individuals to experimental groups/conditions/levels of the IV?
self assignment, experimenter assignment, arbitrary assignment, random assignment/allocation
self assignment
subjects select treatment group
experimenter assignment
experimenter selects with treatment group
arbitrary assignment
selection based on seemingly non-relevant criteria
random assignment/allocation
ensuring that every member has an equal chance of being assigned to any group.
what issue does self-assignment, experimenter-assignment and arbitrary assignment carry?
bias to confound results
what is the benefits of random assignment?
provides maximum insurance that groups are equal
eliminates systematic differences between groups
randomly distributes extraneous variables across groups
reduces bias
what is matching?
using any of a variety of techniques to equate participants in the treatment groups on specific variables. This should be done with variables thought to be related to the IV or may confound the IV
what are the advantages of matching?
controls for the variables on which participants are matched
increases the sensitivity of the experiment
what are the advantages of individual/perception mathching?
groups equated on potential extraneous variables
what are the disadvantages of matching?
- identifying the variables on which to match
- difficulty matching participants increases as the number of variables on which to match increases
- in generalizability of results?
what is blocking?
building the extraneous variable into the research design. Make the extraneous variable another IV in the study. This should only be used when you are interested in the effect of the extraneous variable
benefits of a repeated measure design
- It eliminates the problem of group differences arising from individual differences in group make-up
- Requires fewer participants to have good statistical power (because it is a more refined measure of the impact of IVs with a major source of error removed)
- Having the same participants take part in each condition or level of your IV eliminates the problems of individual differences and needing to deal with these via random allocation or matching
problems with repeated measure designs
participants may perform differently in each condition based on their prior experiences in the study
can lead to sequencing effects
what are sequencing effects?
order effects (practice and fatigue) and carry over effects
practice effects
Where completing measures of the same DV or outcome variable multiple times may lead to the participant becoming practiced at the measure.
If the DV is performance based e.g. reaction time then differences improvements may occur simply due to the practice rather than being due to different conditions of the IV
fatigue effects
where repeated completion of the IV measure of task may lead to boredom or tiredness
what is the solution to the problems of sequencing effects in repeated measures designs?
coutnerbalancing
counterbalancing
- In order to control for sequencing effects we can use counterbalancing
- breaking our sample into subsets who will experience the different conditions in different orders
- By then collapsing all the data together again hopefully we can also test for the existence of sequencing effect as well as collapsing across them
randomized counterbalancing
sequence of conditions is randomly determined for each participant
what are different types of counterbalancing?
randomised, intrasubject, complete, incomplete
intrasubject counterbalancing
participants take treatments in ore than one order. May not be feasible with longer treatment sequences
complete counterbalancing
all possible sequences of treatment are used. Participants randomly assigned to the sequence
N! = N multiplied by each number below it
rarely used with more than 3 conditions because number of possible sequences (N!) too large
incomplete counterbalancing
most common technique
Not all possible sequences are used.
requires a criteria and formula to determine the sequence
what is the criteria for incomplete counterbalancing?
- each treatment condition must appear an equal number of times in each ordinal position and
- each treatment condition must precede and be followed by every other condition an equal number of times
what are the two carry over effects of counterbalancing?
simple and differential
what is the simple carry over effect for counterbalancing?
Where performance on the DV in one condition is contaminated by the effects of the previous condition
What is the differential carry over effect for counterbalaning?
Where the carry-over effects of one condition of the IV differ depending on the order in which the conditions are completed.
why is counterbalancing a great tool?
It allows you to control for order effects by providing data from a range of different orderings of measurement
what happens when time is an IV
participants provide a baseline measurement and then complete measurements at numerous time points during and after the treatment program.
what is the problem with time as an IV?
cant counterbalance time
what are issues when having time as an IV?
maturation, history, statistical regression, mortality
maturation
- Changes due to natural development, natural expected improvement over time etc
- If participants improved over time the question becomes whether or not it is as a function of the IV, or just a natural improvement that would have occurred anyway
history (external events)
- External events that affect participants during the study
- Socio-historical-economic changes relevant to outcome/DV
- While participants were taking part in a program to reduce smoking there was a massive price rise in cigarettes
statistical regression
- Refers to tendency to move up or down towards the mean over time
- In other words someone who was scoring below or above par on a measure is potentially likely to move towards the mean of that variable over repeated measurements
mortality
• Attrition from the study: Not all participants who take part in the first measurement points will remain in the study until the bitter end
what are other effects to an experiments validity?
- Experimenter Effects
- Participant Effects
- Situational Effects
experimenter effect
- The experimenter as part of the experiment
* Measurement, attributes and expectancies
measurement issues regarding the experimenter effect
o problem with equipment or errors in manual recording of data between measurement point or between participants on the part of the experimenter
how does one control measurement issues regarding the experimenter effect?
• make researchers aware of making careful observations (training) multiple data records (video camera, computers) or observers have participants make responses on computer
attributes effect with regard to experimenter effect
participants responding differentially to different experiments within the study
how does one control attribute errors?
use the same experimenter in all treatment conditions (unless the treatment condition interacts with attributes.
what are expectancies with regard to the experimenter effect?
are the likely response of participants to the study and its manipulations may lead to subtle differences in the way the experimenter interacts with participants which lead to differences in outcomes
what are the two sub categories of expectancies with regard to the experimenter effect?
rosenthal effect and the golem effect
rosenthal effect
differential attitude or attention conveyed to participant expected to respond most favourable to the study
golem effect
differential attitude or attention conveyed to participant expected to respond least favourable to the study
what are controls of the experimenter expectancies?
double-blind method, partial blind technique, automation
double-blind method
neither the experimenter nor the research participant is aware of the treatment condition administered to the participant
partial blind technique
a method whereby knowledge of each research participant’s treatment condition is kept from the experimenter through as many stages of the experiment as possible
automation
the technique of totally automating the experimental procedures, so that no experimenter-participant interaction is required
what are problems that come with participant effects?
demand characteristcs, social desirability, hawthorne effect
demand characteristics
participant gets an inkling about what the study is aiming to achieve and performs in a way to conform to those expectations
social desirability
participant performs in a way they think will be most pleasing to experimenter, or will paint them in the best light
hawthorne effect
participant improves or changes performance on the outcome purely as a function of the attention received for being in the study and not as a function of the nature of the manipulations of the IV
what are the controls of the participant effect?
single-blind double-blind, deception
single blind study
The participant is not made aware of the true purpose of the study or the nature of the group in which they are in
double-blind method
neither the experimenter nor the research participant is aware of the treatment condition administered to the participant
Deception
o omission of or altering the truth of information given to the participant during a research study
o used when there is no other way to gain the knowledge and risk does not outweigh the benefit of the information
o must keep the false information constant for all participants
situational effects
impacts o environmental and timing differences on participants’ outcome scores.
Includes time of day, weather, lighting, noises
how are situational effects controlled?
keep these constant for all participants or counterbalancing
what does control groups allow?
allows you to directly examine the extent to which changes would have occurred without your intervention/s as well as potentially examining a whole raft of other extraneous variables
what is a crucial thing to ensure when forming a control group
that it is equivalent
if you are to have multiple control groups, what are the conditions of the two groups?
no intervention
placebo intervention
what does having multiple control groups counter for?
the hawthorne effect
what are the key features of whtin groups designs?
o Minimise individual difference error
o Problems with order effects, practice, cumulative effects, carry-over effects, history, maturation, attrition/mortality
o Can control these things somewhat with counter-balancing, random order variation
what are the key features of a between groups design?
o Eliminate problems of WG
o Individual differences come into play
o Ensuring comparability of groups/matching
