Week 4 (reliabilty & validity) Flashcards
Explain measured scores?
Measured scores = true score+noise
The assumption is everything we want to measure has a true value or score
The goal is that measurement will give us a pure score
Reality: measurement will always include some amount of random noises
The measurement decide or person measuring is going to have some degree to which there will be fluctuations around true score which will be different from time to time
Explain the relationship between consistency (reliability) and variability
Consistency is measure by variability
INVERSE RELATIONSHIP
high variability = low reliability
Low variability= high reliability
What are the three types of reliability?
Test-retest reliability (consistency over time): if we do measurements on multiple occasions and the true score doesn’t change
Internal reliability (consistency across items): eg. Surveys need more than one item to measure abstract constructs
Inter-rater reliability (consistency across researcher): different people collecting the data are consistent. Assessed using a statistic called cohens kappa (k) which is analogous to chronbachs alpha
Is a reliable measure valid?
A measure that is reliable is not automatically valid
Why does variability occur?
- Casual relationships between the IV and DV (effect) - when IV changes DV changes
- Variability due to random factors (noise) - unknown factors affect the DV
Can research design be used to seperate effect from noise?
Noise can be reduced but not eliminated, there is always some amount
What is the split-half reliability?
Rosenberg self-esteem scale
Divide items into two halves and correlate scores on the two halves
Should be strong and positive
What is cronbach’s alpha?
Measures internal consistency
Conceptually related to split half reliability
Calculate all possible split-half correlations for a set of items
Cronbachs is the mean of the set of correlations
Explain parallel forms reliability?
Two versions of test for same test
Scores for the two versions should be highly and positively correlated
Need different set of questions due to testing someone twice
What is validity?
Validity is the extent to which measures represent what they are intended to
Loosely the truthfulness of a measure
What are the forms of validity?
Construct
Internal
External
Face validity
What is construct validity?
Construct validity is “the degree to which a test measures what it claims, or purports, to be measuring.”
What is internal validity?
Extent to which casual statements about the relations among variables can be made
Dependent on study design
High control over extraneous variables have high internal validity
Why is external validity?
Extent to which you can generalise your conclusions to other settings, participants, populations etc
Sample size doesn’t necessarily affect
When does externally invalid research occur?
When conclusions are limited to the original setting, sample etc
What is face validity?
Isn’t a true form of validity but is used
Do IV and DV appear to reflect or measure what the ya re intended to
Research is often unfairly challenged when it lacks face validity - less accepted by general public
What are the threats towards internal validity?
History Maturation Instrumentation decay Statistical regression Selection Testing Morality
How does history threaten internal validity?
Longitudinal studies use risk- measurements that occur overtime are susceptible to effects from factors other than IV. Risk increases as time between measures increases
How does maturation threaten internal validity?
Change may be a result of natural maturation processes
How does instrumentation decay threaten internal validity?
Changing the measurement instrument during a study can affect measurement of the DV
Includes questionnaires and researchers
How does statistical regression threaten internal validity?
Regression to the mean when groups are formed by selecting extreme scores at pretest these scores tend to be less extreme at post test they move closer to true scores
How does selection (biased assignment) threaten internal validity
When groups are not formed through random assignment there is potential bias in group formation
How does testing (protest sensitisation) threaten internal validity
Some studies required participants to do a pre-test (for task familiarisation or baseline measure) after an intervention participants are tested again using the same test this pre-test my influences post test outcome it may alert participants to the expected outcome affecting behavioural test maybe sensitive to learning affects
How does mortality (differential attrition) threaten internal validity?
Differential attrition in studies with more than one group - nonrandom factor that results in greater drop out in one group than the other
Explain subject roles
Good subject: behave in ways that they believe will help the researcher
Faithful subject: they will do precisely what you’re asking them to do
Negative subject: will try to undermine the study and do the opposite of what the researcher wants
Apprehensive-subject: anxious and trying to do things well but anxiety may interfere and they second guess themselves etc
What are some additional forms of validity?
Convergent validity: tests whether constructs/measures that should be related, are related
Discriminant (divergent) validity: tests whether constructs/measures that should be related are unrelated
Predictive validity: tests whether current performance in some measure predicts future performance on another
Concurrent validity: test whether performance on a validated test correlated with performance on a new test
What can be used to test construct validity?
Can be tested through convergent and discriminant (divergent) validity
How do we quantify test-retest reliability?
Using the correlation coefficient
In excel =CORREL
Strong test-retest reliability is .8 or larger
What is used to calculate convergent validity statistically?
Same as test-retest reliability
=CORREL In excel
r >.8 is good
Explain correlation of split-half reliability
Should be strong and positive
