evaluating research design

Question 1

what are the areas of research design to consider?

Answer 1

statistical conclusion validity, construct validity, external validity, internal validity and reliability

Question 2

what is a research design?

Answer 2

it’s a plan for how you are going to do the research

Question 3

what is a reliable study?

Answer 3

one that can be easily replicated

Question 4

what is internal validity?

Answer 4

the extent to which the results of a study accurately reflect the true relationship between the variables being studied, free from the influence of confounding factors or biases

Question 5

what does internal validity examine?

Answer 5

examines whether the study design, conduct, and analysis answer the research question without bias

Question 6

what does internal validity do?

Answer 6

assesses whether the observer effects in a experiment are actually due to the manipulation of the independent variable, rather than being influenced by other factors

Question 7

what is external validity?

Answer 7

the extent to which the results of a study can be generalized to other settings, populations, times, and measures beyond the specific conditions of the original study

Question 8

what does external validity examine?

Answer 8

whether the study findings can be generalized beyond the specific conditions of the study into other contexts

Question 9

what does internal validity focus on?

Answer 9

the accuracy of the study’s conclusions within its own parameters

Question 10

what does external validity focus on?

Answer 10

the extent to which the study’s conclusions can be applied to broader contexts

Question 11

what does external validity directly lead to?

Answer 11

generalizability

Question 12

what do research designs prioritize?

Answer 12

Mostly, research designs prioritize either internal or external validity (so any one study is generally high in either internal or external validity), and we build up a body of work to demonstrate both.

Question 13

what is ecological validity?

Answer 13

sub-type of external validity, specifically focuses on how well the study reflects real-world settings

Question 14

what factors do researchers consider for external validity?

Answer 14

emphasizes how external validity goes beyond just the population studied. It also considers the characteristics of the intervention itself and potential biases within the research design. By considering these factors, researchers can increase the likelihood that their findings can be applied more broadly to real-world settings

Question 15

what does internal validity refer to?

Answer 15

whether the observed changes in the dependent variable can be confidently attributed to the manipulation of the independent variable

Question 16

what are the three conditions for causality?

Answer 16

covariation between variables, temporal precedence, elimination of third variables

Question 17

what does internal validity have to do with causality?

Answer 17

the confidence with which we can say A really caused B

Question 18

what studies are low in internal validity?

Answer 18

correlational studies

Question 19

why are correlational studies low in internal validity?

Answer 19

they identify covariation, typically cannot identify temporal precedence, struggle to eliminate all possible confounding variables

Question 20

what studies are high in internal validity?

Answer 20

experimental studies

Question 21

why are experimental studies high in internal validity?

Answer 21

they identify covariation, set up for temporal precedence (experimental manipulation comes before the effect being measured), design inherently controls for many possible confounding variables

Question 22

what does ensuring high internal validity involve?

Answer 22

controlling for confounding variables and minimizing the potential for biases in the study design and data collection

Question 23

how do confounding variables threaten internal validity?

Answer 23

introduce an alternative explanation for the observed relationship between A and B

Question 24

what threatens internal validity?

Answer 24

confounding variables

Question 25

how do we ensure high internal validity?

Answer 25

Control for confounding/extraneous variables
Minimizing (as best as possible) any potential cause for biases in our design or data collection methods that might influence our findings

Question 26

how do you control for confounds?

Answer 26

well designed control group, measuring variables and including them in statistical models, quasi-experimental studies,

Question 27

what is the difference between random selection and random assignment?

Answer 27

Random selection is when you construct a sample by randomly selecting people from a population: this gives you high external validity.
Random assignment is when you have a sample, and you randomly assign people to experimental and control groups, to make sure that the groups are equal at the start of the study. It gives you high internal validity.

Question 28

what are quasi-experimental studies?

Answer 28

they look like experiments, but aren’t true experiments. We’ve done our best to make the two groups equal, but can’t be sure

Question 29

what are threats to internal validity?

Answer 29

history, testing effect, selection bias, demand characteristics

Question 30

how does history threaten internal validity?

Answer 30

the possibility that external events occurring between the initial and subsequent measurements (or between the intervention and the outcome) can influence the results

Question 31

what is a control strategy for history threatening internal validity?

Answer 31

collect data on potential historical events and statistically control their influence, conduct the study over a shorter timeframe to minimize the chance of external events

Question 32

how does testing effect threaten internal validity?

Answer 32

by potentially influencing participants’ performance on subsequent measurements due to their familiarity with the test rather than any experimental manipulation

Question 33

what is a control strategy for testing effect threatening internal validity?

Answer 33

use a different form of the measure for pre-test and post-test to reduce the familiarity effects, include a buffer period between the pre-test and intervention to minimize carry-over effects

Question 34

how does selection bias threaten internal validity?

Answer 34

introducing systematic differences between groups that can confound the results. This bias occurs when the participants selected for different groups (e.g., treatment and control groups) are not equivalent at baseline, leading to potential differences in outcomes that are attributable to these pre-existing differences rather than the intervention or treatment being studied

Question 35

what is a control strategy for selection bias threatening internal validity

Answer 35

emphasize research aspect of the study and not the expected effects of the study during recruitment, use a large enough sample size to allow for randomization to balance out pre-existing group differences

Question 36

how do demand characteristics threaten internal validity?

Answer 36

influencing participants’ behavior based on their perceptions of what the study is about and what they believe is expected of them

Question 37

what is a control strategy for demand characteristics threatening internal validity

Answer 37

consider using a placebo for the control group, emphasize the importance of honest reporting

Question 38

why is achieving both internal and external validity difficult?

Answer 38

Research designs that prioritize internal validity often involve controlled settings and specific participant groups. This can limit the generalizability of the findings to real-world situations.
Factors that enhance external validity, like studying participants in their natural environments, can introduce confounding variables that threaten internal validity.