Artifacts and biases Flashcards
Artifacts and biases can…
Threaten the validity of an experiment
Artifact
Extraneous influence
Includes all of the variables that the experimenter is not interested in examining
Virtually unlimited number of factors
Artifact or bias?
Depends on the focus on the investigation
Influences commonly identified as artifacts evolve through stages
ICE
- Ignorance
- Coping
- Exploiting
Ignorance
First stage of artifact influence
Investigator = unaware
When source of artifact is posed, may be denied
Coping
Second stage of artifact influence
Existence and possible importance of artifact = recognized
Implement control procedures to direct, estimate, reduce, or prevent its impact
Exploiting
Third (final) stage in artifact infl
Rather than try to minimize its impact -
“Artifact” becomes an area of research
Sources of biases
Investigator
Experimenter
Investigator
Individuals designing the experiment
Experimenter
Individuals executing the experiment
Nature of the problem w rationales, scripts, and procedures
Potential sources of biases incl instructions and experimental materials or procedures in which the subj is exposed
Scripts
Major source of bias - can result from imprecision
Script = specific activities, tasks, and instructions administered by the experimenter
Loose protocol effect
Failure of investigator to provide well-specified script
Problems resulting from failure to specify how experimenter should behave
- Investigator won’t know what was actually done w the subjs
- Inconsistencies obscure the effects of the data and introduces “noise”
Within grp variability (error variance) = increased
Recommendations for script etc problems
- Procedures should be explicit and standardized
- Use automated or previously taped procedures
- Anticipate range of questions that may arise and provide guidelines for answering them
- Train experimenters together
- Incl confederates (increases vigilance)
- Interview subjs after to evaluate consistency of experimenters
- Encourage experimenters to report deviations from the script
Experimenter expectancy effects - nature of the problem
Source of potential bias
Effects refer to the infl of the experimenters’ beliefs and desires about the results on how the subj performs
Effects considered unintentional
Inapprop statistical analyses or selective omission of data
Expectancies can systematically relate to teh magnitude of expectancy effect
Pose a threat to construct validity
Recommendations for experimenter expectancy effects
- Double blind study
- Use different staff for data collection and administration of intervention
- Use manuals
Experimenter characteristics - nature of the problem
Characteristics of the experimenters may infl subj bx
Incl age, gender, race, friendliness, etc.
Usually - only restricts external validity - relationship w IV and DV may only hold up w experimenters who have specific characteristics
Threaten construct validity when one experimenter administers one condition and the other administers another
Tp characteristics infl outcome incl level of empathic understanding, amt of experience, degree of openness, and directness
Recommendations for experimenter characteristics
Not easily balanced bc so many characteristics can be identified
- Data can be analyzed for differences
- Carefully specify characteristics in reports
Situational and contextual cues - nature of the problem
Demand characteristics refer to cues in the experimental situation that may infl how subjs respond
Possible that these cues rather than or in conjunction with the IV account for results
Any facet of experimenter bx, setting, materials, or context of research
Cues that are plausibly related to the pattern of results and are confounded
Recommendations for demand characteristics
Three procedures:
- Post experimental inquiry
- Preinquiry
- SImulators
Post-experimental inquiry
What:
Ask subjs at the end of an experiment about their perceptions as to the purpose, what was expected, and how they were supposed to perform
How to interpret:
If subjs ID responses that are consistent w expected performance - raises the possibility that demand characteristics contributed to results
Pre-inquiry
What:
Subjs exposed to the procedures (told what they are), see what subjs would do, hear the rationale and instructions, but do not actually run through the study itself. They are then asked to respond to the measures
How to interpret:
If subjs respond to measures consistent w hypothesized or expected performance - the possible that demand characteristics account for results
Simulators
What:
Subjs asked to act as if they have received the procedures and then to deceive the assessors who do not know whether they have been exposed to the actual procedures
Similar to pre inquiry, except the subjs actually go thru part of the experiment in which experimenters evaluate performance
How to interpret:
If simulators can actually deceive experimenter - consistent w possibility that demand characteristics contribute to the results
Subject roles - nature of the problem
Subjs may adopt different ways of responding to the experimental cues of the experiment
Reflects how subjs intend to respond to the nature of the task
Types of subj roles
- Good
- Negative
- Faithful
- Apprehensive
Good subject
Subjs attempt to respond in a way that would corroborate the hypothesis
Role reflects subj concern that responses will be useful to science
To adopt this role - must ID hypotheses and act in a fashion consistent w the hypotheses
Negative subj
Attempt to refute hypotheses
Assumed to provide evidence for some alternative or to provide info that is not useful
May result from subjs concern over being controlled, predictable, or in a position of being forced to respond
Faithful subj
Attempt to follow carefully instructions and avoid acting on the basis of any suspicions that they might have about the purpose of the investigation
Role may be performed passively - if instructions followed apathetically
Performed actively - if highly motivated to help science and take special care to avoid infl of suspicions
Apprehensive sub
Concerned that performance will be used to evaluate their ability
Often motivated to present themselves in a favorable light
Subject roles can threaten validity in different ways
Construct validity
Statistical concl
External
Roles threaten construct validity…
If roles are likely to vary systematically w conditions or groups
Roles threaten statistical concl validity
If a particular condition fosters diverse roles, variability of performance among subjects may be increased
Roles threaten external validity
If the results apply only to subjs who adopt a particular role
Recommendations for Subj roles
- Convey that there are no right or wrong answers and that responses are valuable no matter what
- Inducements given prior NOT after
- Convey that responses are anonymous and confidential (reduce apprehensiveness)
- Feedback from experimenter
- Make sure subjs do not perceive hypotheses
Data recording and analysis - nature of the problem
- Errors in recording or computing data
- Analyzing select portions of the data/biases related to data analysis
- Fabricating data
File-drawer problem
Refers to the possibility that the published studies represent a biased sample of all studies that have been completed for a given hypothesis
Recommendations for data recording and analysis problems
- Keep indivs recording data uninformed
- Have subjs enter data directly
- Check data scoring and transposition at each stage
- Threat of expulsion for lying
- One of best checks = replication
Subject-selection bias - nature of the problem
Refers to the infl attributable to types of subjs who participate in experiments
Sample of convenience
Refers to the selection and use of subjs merely because they are available
May not be approp for what is being studied
Often use college students
Problems with volunteers
Possible that those who volunteer differ in important ways from those who do not
Key question = whether volunteers and nonvolunteers differ in ways that affect the generality of the results
Several variables related to volunteering for experiments
Education - Better educated SES - Higher occupational status IQ - higher Need for approval - higher Sociability - more Arousal seeking - more Conventionality - less conventional Sex - female Authoritarianism - less Religious affiliation - Jewish more likely than Protestant; Protestant more like than Catholic Conformity - less Town of origin - smaller town Religiousity - more interested Altruism - more Self-disclosure - more Adjustment - better Age - younger
Recommendations for sampling
- Increase range of persons from among whom volunteers are sought
- Awareness of variables that infl volunteering
- Differences b/w vol and nonvol do not necessarily restrict generality
- Consult findings from population studies
- Consensus data - to see if study sample represents characteristics of population
Attrition, who remains in the study - nature of the problem
Impacts virtually all facets of validity
- Subjs who drop out likely differ from those who remain
- # of subjs who drop out may vary significantly b/w or among groups
- Type of person who drops out varies among conditions or grps w/in the study
- Possible that so many cases drop out that valid concl cannot be made (stat concl)
Recommendations - attrition
- Special orientation, interviews, various mailings during the course of tx, $
- ID variables correlated w attrition and utilize info to decide who participates in subsequent research
- Devise procedures to combat attrition
