L12 - Psych Assessment: Validity Flashcards
What happens to the CI with low reliability?
Confidence intervals get wider.
so lower reliability = LESS PRECISE RESULTS
Although obsolete, what are some ways of measuring internal consistency?
- KR20 - dichotomous items
- Cronbach’s alpha - continuous scale items
They are the average correlation between each item and the total of other items in the list
what is internal consistency?
An internal consistency coefficient sets an upper limit to the reliability of tests constructed according to the domain-sampling model.
It also provides a good estimate of reliability for a test in most situations since a major source of measurement error is because of poor sampling of test item content. Poor item sampling (lack of item consistency) is a major enemy of variance (sources of error variance) and reliable assessment. It is important to be familiar to the reliability tests that are relevant to our study.
What is test standardisation?
Process of turning a test into something that is administered to every person in the same way and scored in the same way - to the extent that that is possible
What is convergent validity?
When there is higher correlation amongst ‘like’ constructs
What is discriminant validity?
Lower correlation amongst dissimilar constructs
what is The attenuation (or disattenuation) formula used for?
It is used to work out the true correlation, when perfect reliability is assumed.
What is the spearman brown prophecy formula used for?
It shows us how much longer to make a test in order to increase reliability.
Produces a negatively accelerating growth curve
How reliable should a test be?
- greater than 0.9 for an individually administered test
- 0.7 min for a test used in group research (larger samples –> measurement errors should cancel out)
What is validity?
Validity is “scientific usefulness” and validity is a term with “surplus meaning”. A test “is valid if it does what it is intended to do.
Establishing test validity is the “scientific inquiry into test score meaning” (Messick 1989).
What is construct validity?
➢ Theoretical understanding of test scores
➢ Relationship of particular test score(s) to other relevant constructs or latent variables (often also measured by test scores)
➢ Convergent and discriminant validity BEST METHOD FOR THIS IS FACTOR ANALYSIS
What is criterion related validity?
➢ Practical evaluation of the usefulness of the test in relevant population(s).
- to what extent does the test scores agree with established criteria? does a new test for depression give similar results to another evaluation process?
two types: PREDICTIVE and CONCURRENT
Examples: (PREDICTING)
- Tertiary academic success from VCE results
- need for psychotherapy from depression test results
- job performance from online selection test
–> these might not be perfect.. but theyre trying to identify “needs” and are cost effective
What is sensitivity (Se)?
The proportion of cases who have the target condition or diagnosis in whom the test is positive.
Equal to the ratio of True Positive or Hits divided by the total number with the target condition (a/a+c). Sensitivity ranges from 0-1 (or 0%-100%). A good test has Se close to 1.
What is prevalence/base rate/pre-test probability?
The frequency of the target condition or diagnosis in a particular population.
In most professional settings, the prev of a diagnosis is higher than in the general population.
(a + c) / (a + b + c + d)
What is positive predictive power (PPP)?
The probability that a positive test result is correct, at a specific base-rate = (a/a+b).
Varies from 0 to 1 and a useful test has a value over .5
“top row ratio”
