4: evaluating casual claims Flashcards
interrogating construct validity dependent variables
-Evaluate the reliability and construct validity of a
dependent variable’s operationalization
interrogating construct validity
-Empirical evidence of the measure’s reliability:
• Test-retest reliability
• Inter-rater reliability
• Internal reliability
- Subjective evidence for the measure’s construct validity:
•face or content
-Empirical evidence of the measure’s construct validity:
• criterion, convergent or divergent
interrogating construct of independent variables
empirical methods of assessing of a manipualation
-Manipulations (operationalizations of independent variables) must also demonstrate reliability and construct validity
• Reliability: Can the IV consistently be manipulated in the same way?
•Construct validity: Does the manipulation change the value of the conceptual variable as intended?
• Start with face validity
-Empirical methods of assessing construct validity of a
manipulation
• Manipulation check: collect data measuring IV during the experiment (after the manipulation) to show that a manipulation worked
• Pilot study: a separate simple study (with separate participants) may be used to measure IV to demonstrate effectiveness of a manipulation
interrogating stat validity
-stat signifigance
• p < .05 establishes covariance
-Effect size is only relevant if difference is significant
• Larger d less overlap stronger effect
watch out for stat probs:
• Outliers – very high or very low DV values
• Restriction of range – too few levels; floor/ceiling effects
• Curvilinear relationship
interrogating internal (V imp)
Three fundamental internal validity questions:
1. Were there any design confounds? Control for design
confounds by holding extraneous variables constant.
- If a between-subjects design was used, did they control for selection effects using random assignment
(possibly preceded by matching)? - If a within-subjects design was used, did they control for order effects by counterbalancing?
internal validity threat 1
- Design confounds: Another variable accidentally varies systematically along with the IV
- Correction: control variables
Hold extraneous variables constant (the same) in all
conditions!
internal validity threat 2
- Selection effects: Systematically different types of participants are in each of the groups
- Between-subjects designs only (independent-groups
-Correction: random assignment or matching
internal validity threat 3
• Order effects: Participants’ later responses are
systematically affected by their earlier ones (due to
carryover such as fatigue, practice, or contrast effects)
• Within-subjects designs only (within-groups)
-Correction: counterbalancing
interrogating external
-Generalizing to other people
• What is the intended population, and how were the participants in the sample selected from that population?
• Random sampling can generalize to intended population
• Generalizability decreases as differences between populations increase
-Generalizing to other situations
• Does finding apply to other settings or other tasks?
• Such as to a real-world context (ecological validity)
- Media often over-generalizes causal claims
mozart effect
- Supposedly intelligence-enhancing music products for babies
- Georgia governor recommended that the state provide classical music recordings to every newborn
- Florida passed a law requiring state-funded programs to play classical music daily for children under 6
external validity is poor
- Experiments must emphasize internal validity, often with tradeoff of external validity
• Highly controlled and artificial laboratory settings
• Nonrandom samples
• Homogenous samples due to inclusion/exclusion criteria, etc. - Suboptimal external validity in experiments is often
acceptable
• Later experiments conducted with other participants and/or in other contexts can support generalization of finding
