Artifacts and biases

Question 1

Artifacts and biases can…

Answer 1

Threaten the validity of an experiment

Question 2

Artifact

Answer 2

Extraneous influence
Includes all of the variables that the experimenter is not interested in examining
Virtually unlimited number of factors

Question 3

Artifact or bias?

Answer 3

Depends on the focus on the investigation

Question 4

Influences commonly identified as artifacts evolve through stages

Answer 4

ICE

1. Ignorance
2. Coping
3. Exploiting

Question 5

Ignorance

Answer 5

First stage of artifact influence
Investigator = unaware
When source of artifact is posed, may be denied

Question 6

Coping

Answer 6

Second stage of artifact influence
Existence and possible importance of artifact = recognized
Implement control procedures to direct, estimate, reduce, or prevent its impact

Question 7

Exploiting

Answer 7

Third (final) stage in artifact infl
Rather than try to minimize its impact -
“Artifact” becomes an area of research

Question 8

Sources of biases

Answer 8

Investigator

Experimenter

Question 9

Investigator

Answer 9

Individuals designing the experiment

Question 10

Experimenter

Answer 10

Individuals executing the experiment

Question 11

Nature of the problem w rationales, scripts, and procedures

Answer 11

Potential sources of biases incl instructions and experimental materials or procedures in which the subj is exposed

Question 12

Scripts

Answer 12

Major source of bias - can result from imprecision

Script = specific activities, tasks, and instructions administered by the experimenter

Question 13

Loose protocol effect

Answer 13

Failure of investigator to provide well-specified script

Question 14

Problems resulting from failure to specify how experimenter should behave

Answer 14

1. Investigator won’t know what was actually done w the subjs
2. Inconsistencies obscure the effects of the data and introduces “noise”
   Within grp variability (error variance) = increased

Question 15

Recommendations for script etc problems

Answer 15

1. Procedures should be explicit and standardized
2. Use automated or previously taped procedures
3. Anticipate range of questions that may arise and provide guidelines for answering them
4. Train experimenters together
5. Incl confederates (increases vigilance)
6. Interview subjs after to evaluate consistency of experimenters
7. Encourage experimenters to report deviations from the script

Question 16

Experimenter expectancy effects - nature of the problem

Answer 16

Source of potential bias
Effects refer to the infl of the experimenters’ beliefs and desires about the results on how the subj performs
Effects considered unintentional
Inapprop statistical analyses or selective omission of data
Expectancies can systematically relate to teh magnitude of expectancy effect
Pose a threat to construct validity

Question 17

Recommendations for experimenter expectancy effects

Answer 17

1. Double blind study
2. Use different staff for data collection and administration of intervention
3. Use manuals

Question 18

Experimenter characteristics - nature of the problem

Answer 18

Characteristics of the experimenters may infl subj bx
Incl age, gender, race, friendliness, etc.
Usually - only restricts external validity - relationship w IV and DV may only hold up w experimenters who have specific characteristics
Threaten construct validity when one experimenter administers one condition and the other administers another
Tp characteristics infl outcome incl level of empathic understanding, amt of experience, degree of openness, and directness

Question 19

Recommendations for experimenter characteristics

Answer 19

Not easily balanced bc so many characteristics can be identified

1. Data can be analyzed for differences
2. Carefully specify characteristics in reports

Question 20

Situational and contextual cues - nature of the problem

Answer 20

Demand characteristics refer to cues in the experimental situation that may infl how subjs respond
Possible that these cues rather than or in conjunction with the IV account for results
Any facet of experimenter bx, setting, materials, or context of research
Cues that are plausibly related to the pattern of results and are confounded

Question 21

Recommendations for demand characteristics

Answer 21

Three procedures:

1. Post experimental inquiry
2. Preinquiry
3. SImulators

Question 22

Post-experimental inquiry

Answer 22

What:
Ask subjs at the end of an experiment about their perceptions as to the purpose, what was expected, and how they were supposed to perform
How to interpret:
If subjs ID responses that are consistent w expected performance - raises the possibility that demand characteristics contributed to results

Question 23

Pre-inquiry

Answer 23

What:
Subjs exposed to the procedures (told what they are), see what subjs would do, hear the rationale and instructions, but do not actually run through the study itself. They are then asked to respond to the measures
How to interpret:
If subjs respond to measures consistent w hypothesized or expected performance - the possible that demand characteristics account for results

Question 24

Simulators

Answer 24

What:
Subjs asked to act as if they have received the procedures and then to deceive the assessors who do not know whether they have been exposed to the actual procedures
Similar to pre inquiry, except the subjs actually go thru part of the experiment in which experimenters evaluate performance
How to interpret:
If simulators can actually deceive experimenter - consistent w possibility that demand characteristics contribute to the results

Question 25

Subject roles - nature of the problem

Answer 25

Subjs may adopt different ways of responding to the experimental cues of the experiment Reflects how subjs intend to respond to the nature of the task

Question 26

Types of subj roles

Answer 26

1. Good 2. Negative 3. Faithful 4. Apprehensive

Question 27

Good subject

Answer 27

Subjs attempt to respond in a way that would corroborate the hypothesis Role reflects subj concern that responses will be useful to science To adopt this role - must ID hypotheses and act in a fashion consistent w the hypotheses

Question 28

Negative subj

Answer 28

Attempt to refute hypotheses Assumed to provide evidence for some alternative or to provide info that is not useful May result from subjs concern over being controlled, predictable, or in a position of being forced to respond

Question 29

Faithful subj

Answer 29

Attempt to follow carefully instructions and avoid acting on the basis of any suspicions that they might have about the purpose of the investigation Role may be performed passively - if instructions followed apathetically Performed actively - if highly motivated to help science and take special care to avoid infl of suspicions

Question 30

Apprehensive sub

Answer 30

Concerned that performance will be used to evaluate their ability Often motivated to present themselves in a favorable light

Question 31

Subject roles can threaten validity in different ways

Answer 31

Construct validity Statistical concl External

Question 32

Roles threaten construct validity...

Answer 32

If roles are likely to vary systematically w conditions or groups

Question 33

Roles threaten statistical concl validity

Answer 33

If a particular condition fosters diverse roles, variability of performance among subjects may be increased

Question 34

Roles threaten external validity

Answer 34

If the results apply only to subjs who adopt a particular role

Question 35

Recommendations for Subj roles

Answer 35

1. Convey that there are no right or wrong answers and that responses are valuable no matter what 2. Inducements given prior NOT after 3. Convey that responses are anonymous and confidential (reduce apprehensiveness) 4. Feedback from experimenter 5. Make sure subjs do not perceive hypotheses

Question 36

Data recording and analysis - nature of the problem

Answer 36

1. Errors in recording or computing data 2. Analyzing select portions of the data/biases related to data analysis 3. Fabricating data

Question 37

File-drawer problem

Answer 37

Refers to the possibility that the published studies represent a biased sample of all studies that have been completed for a given hypothesis

Question 38

Recommendations for data recording and analysis problems

Answer 38

1. Keep indivs recording data uninformed 2. Have subjs enter data directly 3. Check data scoring and transposition at each stage 4. Threat of expulsion for lying 5. One of best checks = replication

Question 39

Subject-selection bias - nature of the problem

Answer 39

Refers to the infl attributable to types of subjs who participate in experiments

Question 40

Sample of convenience

Answer 40

Refers to the selection and use of subjs merely because they are available May not be approp for what is being studied Often use college students

Question 41

Problems with volunteers

Answer 41

Possible that those who volunteer differ in important ways from those who do not Key question = whether volunteers and nonvolunteers differ in ways that affect the generality of the results

Question 42

Several variables related to volunteering for experiments

Answer 42

``` Education - Better educated SES - Higher occupational status IQ - higher Need for approval - higher Sociability - more Arousal seeking - more Conventionality - less conventional Sex - female Authoritarianism - less Religious affiliation - Jewish more likely than Protestant; Protestant more like than Catholic Conformity - less Town of origin - smaller town Religiousity - more interested Altruism - more Self-disclosure - more Adjustment - better Age - younger ```

Question 43

Recommendations for sampling

Answer 43

1. Increase range of persons from among whom volunteers are sought 2. Awareness of variables that infl volunteering 3. Differences b/w vol and nonvol do not necessarily restrict generality 4. Consult findings from population studies 5. Consensus data - to see if study sample represents characteristics of population

Question 44

Attrition, who remains in the study - nature of the problem

Answer 44

Impacts virtually all facets of validity 1. Subjs who drop out likely differ from those who remain 2. # of subjs who drop out may vary significantly b/w or among groups 3. Type of person who drops out varies among conditions or grps w/in the study 4. Possible that so many cases drop out that valid concl cannot be made (stat concl)

Question 45

Recommendations - attrition

Answer 45

1. Special orientation, interviews, various mailings during the course of tx, $ 2. ID variables correlated w attrition and utilize info to decide who participates in subsequent research 3. Devise procedures to combat attrition