evaluating research design Flashcards
what are the areas of research design to consider?
statistical conclusion validity, construct validity, external validity, internal validity and reliability
what is a research design?
it’s a plan for how you are going to do the research
what is a reliable study?
one that can be easily replicated
what is internal validity?
the extent to which the results of a study accurately reflect the true relationship between the variables being studied, free from the influence of confounding factors or biases
what does internal validity examine?
examines whether the study design, conduct, and analysis answer the research question without bias
what does internal validity do?
assesses whether the observer effects in a experiment are actually due to the manipulation of the independent variable, rather than being influenced by other factors
what is external validity?
the extent to which the results of a study can be generalized to other settings, populations, times, and measures beyond the specific conditions of the original study
what does external validity examine?
whether the study findings can be generalized beyond the specific conditions of the study into other contexts
what does internal validity focus on?
the accuracy of the study’s conclusions within its own parameters
what does external validity focus on?
the extent to which the study’s conclusions can be applied to broader contexts
what does external validity directly lead to?
generalizability
what do research designs prioritize?
Mostly, research designs prioritize either internal or external validity (so any one study is generally high in either internal or external validity), and we build up a body of work to demonstrate both.
what is ecological validity?
sub-type of external validity, specifically focuses on how well the study reflects real-world settings
what factors do researchers consider for external validity?
emphasizes how external validity goes beyond just the population studied. It also considers the characteristics of the intervention itself and potential biases within the research design. By considering these factors, researchers can increase the likelihood that their findings can be applied more broadly to real-world settings
what does internal validity refer to?
whether the observed changes in the dependent variable can be confidently attributed to the manipulation of the independent variable
what are the three conditions for causality?
covariation between variables, temporal precedence, elimination of third variables
what does internal validity have to do with causality?
the confidence with which we can say A really caused B
what studies are low in internal validity?
correlational studies
why are correlational studies low in internal validity?
they identify covariation, typically cannot identify temporal precedence, struggle to eliminate all possible confounding variables
what studies are high in internal validity?
experimental studies
why are experimental studies high in internal validity?
they identify covariation, set up for temporal precedence (experimental manipulation comes before the effect being measured), design inherently controls for many possible confounding variables
what does ensuring high internal validity involve?
controlling for confounding variables and minimizing the potential for biases in the study design and data collection
how do confounding variables threaten internal validity?
introduce an alternative explanation for the observed relationship between A and B
what threatens internal validity?
confounding variables
how do we ensure high internal validity?
Control for confounding/extraneous variables
Minimizing (as best as possible) any potential cause for biases in our design or data collection methods that might influence our findings
how do you control for confounds?
well designed control group, measuring variables and including them in statistical models, quasi-experimental studies,
what is the difference between random selection and random assignment?
Random selection is when you construct a sample by randomly selecting people from a population: this gives you high external validity.
Random assignment is when you have a sample, and you randomly assign people to experimental and control groups, to make sure that the groups are equal at the start of the study. It gives you high internal validity.
what are quasi-experimental studies?
they look like experiments, but aren’t true experiments. We’ve done our best to make the two groups equal, but can’t be sure
what are threats to internal validity?
history, testing effect, selection bias, demand characteristics
how does history threaten internal validity?
the possibility that external events occurring between the initial and subsequent measurements (or between the intervention and the outcome) can influence the results
what is a control strategy for history threatening internal validity?
collect data on potential historical events and statistically control their influence, conduct the study over a shorter timeframe to minimize the chance of external events
how does testing effect threaten internal validity?
by potentially influencing participants’ performance on subsequent measurements due to their familiarity with the test rather than any experimental manipulation
what is a control strategy for testing effect threatening internal validity?
use a different form of the measure for pre-test and post-test to reduce the familiarity effects, include a buffer period between the pre-test and intervention to minimize carry-over effects
how does selection bias threaten internal validity?
introducing systematic differences between groups that can confound the results. This bias occurs when the participants selected for different groups (e.g., treatment and control groups) are not equivalent at baseline, leading to potential differences in outcomes that are attributable to these pre-existing differences rather than the intervention or treatment being studied
what is a control strategy for selection bias threatening internal validity
emphasize research aspect of the study and not the expected effects of the study during recruitment, use a large enough sample size to allow for randomization to balance out pre-existing group differences
how do demand characteristics threaten internal validity?
influencing participants’ behavior based on their perceptions of what the study is about and what they believe is expected of them
what is a control strategy for demand characteristics threatening internal validity
consider using a placebo for the control group, emphasize the importance of honest reporting
why is achieving both internal and external validity difficult?
Research designs that prioritize internal validity often involve controlled settings and specific participant groups. This can limit the generalizability of the findings to real-world situations.
Factors that enhance external validity, like studying participants in their natural environments, can introduce confounding variables that threaten internal validity.
