Unit 3: Validity and Control Flashcards
Define validity
an indication of accuracy – in terms of how closely the research conclusion corresponds to reality
internal validity
The extent to which the experiment demonstrates a cause-effect relationship. The degree to which the experimental manipulation alone caused the change in behaviour.
construct validity
the degree to which the study measures and manipulates the underlying concepts the researcher claims. Does the experiment study what it is supposed to?
external validity
the degree to which the results could be generalized to different subjects, settings, or times.
statistical conclusion validity
the degree to which the observed results are not due to chance only
several common categories of confounding variables, known as threats to internal validity
History Maturation Testing Instrumentation Regression effects Selection biases Mortality Subject variables
History
Specific events other than the independent variable that occur between the first and second test an influence the dependant variable.
Maturation
Internal changes to the subject (biological and psychological) that happen over time.
Testing effects
Any effects on the dependant variables as a result of repeated testing.
Instrumentation
Changes over time to the way the dependant variable is measured.
Regression effects
Subjects with extreme scores give scores closer to the average on subsequent testing.
Selection biases
When different selection procedures are used for placing subjects in the groups.
Mortality
Subjects from different groups dropping out at different rates. It is a threat when the ones who drop out are different than the ones who stay, thus biasing the subject group.
Subject variable
When characteristics can only be selected, not controlled.
Control
Any technique used to reduce threats to validity
Randomization
an unbiased assignment process that gives all individual subjects equal chance at being placed into any of the experimental groups.
Matching
A control technique used to ensure that groups are equated on one or more variable before the experiment.
Statistical control
A mathematical way of equating subjects on paper when they can’t be equated in fact.
Statistical control
A mathematical way of equating subjects on paper when they can’t be equated in fact.
Confounding
An error that occurs when the effects of more than one variable can not be separated -> confused interpretation of results.
Why is confounding an issue with subject variables?
When characteristics can only be selected, not controlled, there is a greater chance of other influences on the subjects and therefore on the results they give, which can not be separated from the variables being tested.
What are ways to determine construct validity?
- Manipulation check - part of the experimental design that checks to see whether the independent variable is changing in the intended way.
- Run another experiment to test other possible explanations.
2 threats to construct validity.
- When theory and method are too loosely connected.
- Ambiguous effects of independent variables - when subjects don’t behave naturally - good-subject tendency (giving the tester what you think they want), or evaluation apprehension (pretending to be the more socially acceptable way).
3 threats to external validity
- People - subjects are not representative of the population
- Time - tests done at another time yield different results (social factors change over time)
- Place - behaviour can change depending on the location.
3 threats to external validity
- People - subjects are not representative of the population
- Time - tests done at another time yield different results (social factors change over time)
- Place - behaviour can change depending on the location.
2 broad categories of bias resulting from researcher-subject interaction?
- Role demands / demand characteristics
2. Researcher bias
What are role demands / demand characteristics?
Subject’s expectations about what the experiment requires them to do.
What can be done about role demands / demand characteristics?
- Deceive the subjects
- Design the experiment so the hypothesis is not obvious
- Use a measure that unlikely to be effected by role demands
What is experimenter bias?
When the experimenter’s biases influence the results.
What can be done about experimenter bias?
- double blind - so the person presenting to the subject does not have full knowledge of the experiment
- standardize. use written/recorded means to communicate to the subjects.
- What does the concept of control mean in experimental research?
Any means to rule out threats to an experiment’s validity. Achieve constancy of the effects of the confounding variables across the levels of the independent variable.
Control experiment vs. experimental control
Control experiment: using a control condition or group as a standard of comparison.
Experimental control: reducing variability to the independent variable to reduce potential confounding variables
3 strategies for achieving control
- Laboratory to control the environment
- Research setting as a preparation: and environment that is selected or constructed for a particular purpose
- Control of the instrumentation of the response: all the ways you could turn the observable response into statistical information
- Briefly describe the strategy of subject as own control. What are the limitations of this strategy?
Each subjects experiences both control and experimental conditions (all conditions).
Limitations: sometimes one condition influences behaviour in the others. Some are mutually exclusive (learning same info by one method vs. another)
What is random assignment?
An unbiased method of assigning subjects to conditions/groups that gives all subjects an equal and independent chance to be assigned to any one condition.
How can random assignment be performed?
- Assigning everyone a number, then using a random number table or generator to determine who goes in which condition.
- Computer randomizer.
Why is matching a strategy for achieving control?
It is a strategy for control because it ensures that groups are equal on one or more variable before the experiment, thus reducing the likelihood of those variables being confounded.
Under what conditions should matching of subjects be done?
Matching should be done when subjects differ on a variable suspected to affect the variable of interest.
Describe the proper matching procedure.
- Do a pre-test
- Rank them
- Randomly assign the individuals from each pair to the conditions
* they stay in their matched pairs for analysis - data is analyzed for the pairs, not for the individual.
Briefly describe building nuisance variables in an experiment as a strategy for achieving control.
Nuisance variables are ones that can not be easily excluded, so they are built into the experimental design as a way of controlling for their effects.
How is statistical control used as a strategy for achieving control?
Statistics equate subjects on paper when they can not be equated in fact. You can compare not just the results to each other, but also within the context of other data, thus accounting for variables that may have influenced the result.
What is replication?
Repeating the experiment to see whether the results are the same.
Distinguish two types of replication
Direct replication - doing the experiment exactly. No one does this because it’s not exciting or glorious and no one wants to fund it.
Systematic replication - repeating the experiment but with some tweaks. If results are consistent, it supports the first experiment. Tests the first experiment’s external, construct, and statistical validity.
