Validity in designs

Question 1

What are the different types of validity in experimental research?

Answer 1

• Statistical conclusion validity
• Construct validity
• External validity
• Internal validity

Question 2

What is statistical conclusion validity?

Answer 2

• The degree to which conclusions about the relationship based on the data are correct.
• You need to do the correct analysis without violating any assumptions
• Determine your hypotheses and primary outcome variables before the data has been collected.
• Report all analyses, even the ones you don’t like.
• Don’t go fishing (this is a type I error)
• Make sure your measures are reliable, so that if there is an effect you can find it (otherwise you run the risk of a type II error)

Question 3

What is a type I error in terms of statistical conclusion validity?

Answer 3

• The more the analyses, the higher the chance of a false positive.

Question 4

What is a type II error in terms of statistical conclusion validity?

Answer 4

A false negative.

Question 5

What is construct validity?

Answer 5

• The extent to which a measure of a construct is empirically related to other measures with which it is theoretically associated.
• Does a test truly measure what it purports to measure?
• Are your operational definitions adequate?

Question 6

What is external validity?

Answer 6

• The degree to which the findings/conclusions can be generalized beyond the confines of the design and study setting.
• Generalisability to: other populations, other environments, other times.

Question 7

What is internal validity?

Answer 7

• The degree to which a study is methodologically sound and confound-free.
• Ensure all third variables controlled.
• The degree to which we can be sure that the dependent variable changed as a result of the independent variable.
• Concerns the research design and validity of the conclusions.

Question 8

What are the 7 threats to validity?

Answer 8

• History
• Maturation
• Testing
• Instrumentation
• Selection bias
• Attrition (mortality)
• Regression to the mean.

Question 9

What is discriminant evidence?

Answer 9

Measures that aren’t theoretically related aren’t empirically related either. Convergent evidence is the opposite.

Question 10

What are pre-post studies?

Answer 10

• The dependent variable is measured; there is an intervention; the dependent variable is measured again.

Question 11

What is the con to pre-post studies?

Answer 11

They are low in internal validity. This is why we need control groups and placebos

Question 12

What is History and how do we counteract it as a threat to internal validity?

Answer 12

• History is any event between pre and post test that is outside of research control and impacts entire sample
• Counteract it by having a control group that provides point of comparison.

Question 13

What is Maturation and how do we control it?

Answer 13

• How participants naturally mature (better or worse) during a study
• Particularly a problem with younger people/children
• Counteract it by having a control group that provides point of comparison, using random assignment, and having a large sample.

Question 14

What is testing and how do we counteract it?

Answer 14

• When pretest (baseline) changes behaviour in participants
• Counteract by using a control group as a point of comparison

Question 15

What is instrumentation and how do we control for it?

Answer 15

• This happens when measures are not operationalised accurately or reliably.
• Counteract by: ensure operationalisation is accurate and reliable, use random assignment and large sample (both groups will experience the same but will not be reliable)

Question 16

What is selection bias and how do we control for it?

Answer 16

• Participants being picked according to a criterion that biases results
• Usually not a conscious decision, but can cause massive problems for studies.
• Volunteers means more eager people
• Selection bias means the sample population will be highly specific on important variables.
• Counteract this by using randomization and large samples.

Question 17

What is attrition and how do we counteract this?

Answer 17

• People dropping out or dying
• This can confound the study if it isn’t happening randomly
• Can only be countered statistically. You can also use a control group as a point of comparison

Question 18

What is regression to the mean and how do we counteract it?

Answer 18

• Extreme high/low scores on the first test tending to move towards the mean on the subsequent test
• Extremely low scorers are probably scoring below their true score, very likely for next time to be closer to their true scores.
• Can result in selection bias when using premeasure to select high/low scoring participants.
• Counteract by not using premeasure to select high/low scoring participants.
• Random assignment

Question 19

What is the placebo effect and how do we counteract it?

Answer 19

• Thinking that something will have an effect can make it have an effect.
• Can lead to a measurable, but often temporary benefit which makes it look like there’s a difference, or that a treatment has worked.
• Relevant for any active manipulation including medical tests, emotion induction, therapies, support groups etc.
• Counteract by having a control group as it provides a point of comparison.