evaluating quantitative research Flashcards
reliable instrument
instrument that gives the same measurements when you repeatedly measure the same unchanged objects or events
-a test must be reliable for it to be valid
internal validity
the extent to which the change in the dependant variable can be attributed to the manipulation of the independant variable
external validity
extent to which researchers can generalize their findings to other people situations and times
-people : pop validity (do findings generalize from kids to adults)
-situations : ecological validity (real world vs gym situations)
-times : time periods, diff attitudes, beliefs
internal vs external validity
trade-off between internal and external validity
-series of experiments (studies)
-each study has specific goals and limitations
-real world intervention vs. lab based intervention
->control for extraneous variables -> looks less like real world because controlling too much
threats to internal validity
experimental procedures, treatments, or experiences of the participants that threaten the researchers ability to draw correct inference from the data about the population in an expriment
-ability to establish cause and effect
history (threat)
events occuring during the course of experiment that cause changes in the DV and are not the intervention (IV)
-unplanned event that coincides with the IV could be responsible for the observed changes in DV
pretest -> unplanned event -> posttest
maturation
processes within the participant that operate as a result of time passing
eg. developing in highschool during start of year beep to end of year beep
pretest-natural development -> post test
testing
the effects of one test on subsequent administrations of another test
-skipping intervention? pre test right to post
testing - reactivity
exposure to pretest changes behaviour
eg. self monitoring - athlete increase effort bc they were asked to record it
eg. food log
testing - warm-up effect
do better on the subsequent measure/test because they are more familiar with it
-eg. getting better at yoyo test, not improving aerobic capacity just more familiar with test
loss of naivety
people start catching on to what you are testing/measuring
instrumentation
quantitative research relies on measurement
eg. leadership style, MRI , motivation etc.
instrumentation : changes in instrument calibration, including lack of agreement within and between observers
-it is important that the data acquired from instruments be accurate or else the interpretation will be inaccurate
-getting accurate measurements
selection bias
choosing comparison groups in a non-random manner (not randomly assigned)
-pre-existing differences between groups - functions as confounding variables
experimental mortality
loss of participants from comparison groups for non-random reasons
-participants drop out and those who remain tend to be more motivated (results in higher scores on post-test)
selection - maturation interaction
the passage of time affects one group but not the other in nonequivalent group designs
-2 groups
group 1 has more enjoyment than 2
-but; could be due to the fact that
group 2 train on grass in worse weather
group 1 train on turf
expectancy
researchers anticipating that certain participants will perform better
-expectancy has big impact on how we interact with people and how we receive them
-if we expect ppl to do well they will rise to the occasion
could do blinding procedures
controlling threats to internal validity - randomization
random assignment - controls for history
matched pairs - eg. age, gender (next best thing)
randomizing treatments or counterbalancing
-> ABC v ACB c BAV v BCA (order of exercises in s and c)
random selection vs random asignment
population -> (random selection) -> sample -> (random assignment) -> groups
random selection
-effects external validity
-ability to generalize findings
random assignment
-effects internal validity
placebos, blind and double-blind studies
helps reduce experimenter bias/participant expectancies
-attention control group
blind : patients dont know what group they are in
double blind : patients AND researcher dont know what group patients are in
standardize experiments and instruments as much as possible
eg. script every researcher uses, everything in the same order, make sure instruments are calibrated after every test
-order of questionnaires
threats to external validity - two streams
population validity : extent to which findings from a sample can be generalized to a broader population
ecological validity : extent to which research emulates the real world
time
a threat to external validity
-context changes over time, work needs to be contextualized for appropriate time
-covid studies generalized for that time only cant cut across time periods
threats to external validity arise when
experimenters draw incorrect generalizations from the sample data to other persons, other settings, and past or future situations
reactive/interactive effects of testing
the pretest may make the participant more aware or sensitive to the upcoming treatment
-as a result the treatment is not as effective without the pretest
eg. transformative leadership
-more receptive to leadership style because you have thought about your own style beforehand with PRETEST
interaction of selection bias and the experimental treatment
when a group is selected on some characteristics, the treatment may work only on groups possessing that characteristic
eg. gender, age, sport, nationality
eg. gameplan for high perf sport
-helps folks while in sport ; wellbeing. may be effective in sport systems but less effective in other countries
multiple treatment interference
when participants receive more than one treatment , the effects of the previous treatments may influence subsequent ones
eg. not gonna hangclean only for 3 months
-interaction bw clean and sprint
-could be less effective without one of them
-hard to generalize findings bc influencing each other
controlling threats to external validity
selecting participants, treatments, experimental siutation and tests to represent a broader population
-random selection
controlling threats to external validity
researcher might restrict claims about groups to which the results cannot be generalized
-conduct additional experiments with groups with different characteristics
