Unit 3: Validity and Control

Question 1

Define validity

Answer 1

an indication of accuracy – in terms of how closely the research conclusion corresponds to reality

Question 2

internal validity

Answer 2

The extent to which the experiment demonstrates a cause-effect relationship. The degree to which the experimental manipulation alone caused the change in behaviour.

Question 3

construct validity

Answer 3

the degree to which the study measures and manipulates the underlying concepts the researcher claims. Does the experiment study what it is supposed to?

Question 4

external validity

Answer 4

the degree to which the results could be generalized to different subjects, settings, or times.

Question 5

statistical conclusion validity

Answer 5

the degree to which the observed results are not due to chance only

Question 6

several common categories of confounding variables, known as threats to internal validity

Answer 6

History
Maturation
Testing
Instrumentation
Regression effects
Selection biases
Mortality
Subject variables

Question 7

History

Answer 7

Specific events other than the independent variable that occur between the first and second test an influence the dependant variable.

Question 8

Maturation

Answer 8

Internal changes to the subject (biological and psychological) that happen over time.

Question 9

Testing effects

Answer 9

Any effects on the dependant variables as a result of repeated testing.

Question 10

Instrumentation

Answer 10

Changes over time to the way the dependant variable is measured.

Question 11

Regression effects

Answer 11

Subjects with extreme scores give scores closer to the average on subsequent testing.

Question 12

Selection biases

Answer 12

When different selection procedures are used for placing subjects in the groups.

Question 13

Mortality

Answer 13

Subjects from different groups dropping out at different rates. It is a threat when the ones who drop out are different than the ones who stay, thus biasing the subject group.

Question 14

Subject variable

Answer 14

When characteristics can only be selected, not controlled.

Question 15

Control

Answer 15

Any technique used to reduce threats to validity

Question 16

Randomization

Answer 16

an unbiased assignment process that gives all individual subjects equal chance at being placed into any of the experimental groups.

Question 17

Matching

Answer 17

A control technique used to ensure that groups are equated on one or more variable before the experiment.

Question 18

Statistical control

Answer 18

A mathematical way of equating subjects on paper when they can’t be equated in fact.

Question 19

Statistical control

Answer 19

A mathematical way of equating subjects on paper when they can’t be equated in fact.

Question 20

Confounding

Answer 20

An error that occurs when the effects of more than one variable can not be separated -> confused interpretation of results.

Question 21

Why is confounding an issue with subject variables?

Answer 21

When characteristics can only be selected, not controlled, there is a greater chance of other influences on the subjects and therefore on the results they give, which can not be separated from the variables being tested.

Question 22

What are ways to determine construct validity?

Answer 22

1. Manipulation check - part of the experimental design that checks to see whether the independent variable is changing in the intended way.
2. Run another experiment to test other possible explanations.

Question 23

2 threats to construct validity.

Answer 23

1. When theory and method are too loosely connected.
2. Ambiguous effects of independent variables - when subjects don’t behave naturally - good-subject tendency (giving the tester what you think they want), or evaluation apprehension (pretending to be the more socially acceptable way).

Question 24

3 threats to external validity

Answer 24

1. People - subjects are not representative of the population
2. Time - tests done at another time yield different results (social factors change over time)
3. Place - behaviour can change depending on the location.

Question 25

3 threats to external validity

Answer 25

1. People - subjects are not representative of the population
2. Time - tests done at another time yield different results (social factors change over time)
3. Place - behaviour can change depending on the location.

Question 26

2 broad categories of bias resulting from researcher-subject interaction?

Answer 26

1. Role demands / demand characteristics

2. Researcher bias

Question 27

What are role demands / demand characteristics?

Answer 27

Subject’s expectations about what the experiment requires them to do.

Question 28

What can be done about role demands / demand characteristics?

Answer 28

1. Deceive the subjects
2. Design the experiment so the hypothesis is not obvious
3. Use a measure that unlikely to be effected by role demands

Question 29

What is experimenter bias?

Answer 29

When the experimenter’s biases influence the results.

Question 30

What can be done about experimenter bias?

Answer 30

1. double blind - so the person presenting to the subject does not have full knowledge of the experiment
2. standardize. use written/recorded means to communicate to the subjects.

Question 31

16. What does the concept of control mean in experimental research?

Answer 31

Any means to rule out threats to an experiment’s validity. Achieve constancy of the effects of the confounding variables across the levels of the independent variable.

Question 32

Control experiment vs. experimental control

Answer 32

Control experiment: using a control condition or group as a standard of comparison.
Experimental control: reducing variability to the independent variable to reduce potential confounding variables

Question 33

3 strategies for achieving control

Answer 33

1. Laboratory to control the environment
2. Research setting as a preparation: and environment that is selected or constructed for a particular purpose
3. Control of the instrumentation of the response: all the ways you could turn the observable response into statistical information

Question 34

19. Briefly describe the strategy of subject as own control. What are the limitations of this strategy?

Answer 34

Each subjects experiences both control and experimental conditions (all conditions).
Limitations: sometimes one condition influences behaviour in the others. Some are mutually exclusive (learning same info by one method vs. another)

Question 35

What is random assignment?

Answer 35

An unbiased method of assigning subjects to conditions/groups that gives all subjects an equal and independent chance to be assigned to any one condition.

Question 36

How can random assignment be performed?

Answer 36

1. Assigning everyone a number, then using a random number table or generator to determine who goes in which condition.
2. Computer randomizer.

Question 37

Why is matching a strategy for achieving control?

Answer 37

It is a strategy for control because it ensures that groups are equal on one or more variable before the experiment, thus reducing the likelihood of those variables being confounded.

Question 38

Under what conditions should matching of subjects be done?

Answer 38

Matching should be done when subjects differ on a variable suspected to affect the variable of interest.

Question 39

Describe the proper matching procedure.

Answer 39

1. Do a pre-test
2. Rank them
3. Randomly assign the individuals from each pair to the conditions
   * they stay in their matched pairs for analysis - data is analyzed for the pairs, not for the individual.

Question 40

Briefly describe building nuisance variables in an experiment as a strategy for achieving control.

Answer 40

Nuisance variables are ones that can not be easily excluded, so they are built into the experimental design as a way of controlling for their effects.

Question 41

How is statistical control used as a strategy for achieving control?

Answer 41

Statistics equate subjects on paper when they can not be equated in fact. You can compare not just the results to each other, but also within the context of other data, thus accounting for variables that may have influenced the result.

Question 42

What is replication?

Answer 42

Repeating the experiment to see whether the results are the same.

Question 43

Distinguish two types of replication

Answer 43

Direct replication - doing the experiment exactly. No one does this because it’s not exciting or glorious and no one wants to fund it.
Systematic replication - repeating the experiment but with some tweaks. If results are consistent, it supports the first experiment. Tests the first experiment’s external, construct, and statistical validity.