Week 3 Flashcards
How many components does a test score have?
A test score has two components - the obtained score and the error
Can we apply a CI to obtained scores?
Yes.
Xi = +/- 1.96 x SD
ti = +/- 1.96 x SEM
We do not know ti and SEM and need to estimate them
How do we estimate the true score?
ti = M + rxx (Xi - M)
When reliability (rxx) = 1 (perfect score) then the obtained score will be the true score as both M’s cancel each other out.
If the reliability = 0, the test is considered unreliable.
ti = estimated true score
Xi = individual raw score
M = mean of scores
rxx = reliability of the test
How do we estimate the standard error of measurement?
SEM = SD √1-rxx
If rxx = 0, that is the test is unreliable, the SEM is equal to the SD of the sample
If rxx = 1, the test is perfectly reliable and the SEM = 0.
Define validity
The extent to which evidence supports the meaning and use of a psychological test.
Describe the key difference between reliability and validity.
Reliability = consistency
Validity = accuracy
Reliability is necessary but not sufficient for validity i.e. if a test does not provide reliable (consistent) scores, the test cannot be valid (accurate).
Face validity is
The test appears to test what it sets out to test
Criterion-related validity
The test score predict performance
Concurrent validity (right now, existing state, useful to diagnose). Used to determine learning difficulty.
Predictive validity (future, how will this person learn in the future). Used most commonly for selection/classification
Example: VCE scores to determine entrance to Uni
Content validity
The test covers a representative sample of the domain being measured
More relevant for achievement tests (i.e. exams)
Built into the test from the outset
Construct validity
The test scores reflect individual differences in a psychological construct
for example anxiety, intelligence, extraversion are not directly observable but age differentiation, relationship to behaviour, experimental manipulation and criterion groups are
convergent validity
divergent validity
What is a good level of validity?
About .2 to .5
Typically lower than reliability coefficients (.8)
What is the role of validity in decision theory?
Even if rxy is low, a test may appreciably improve predictive efficiency.
We use a test if the benefit gained in terms of predictions outweighs losses in terms of costs incurred as a result of testing. They can increase base rate successes.
Detail the factors that influence validity
Low sample size (may not reach statistical significance)
Restriction of range in either the test or criterion scores > creates a selectivity problem
non-linear relationships between test and criterion scores (assumption of Pearson’s correlation - when non linear a non-parametric test is required.
Problems with the criterion.
Solutions for the selectivity problem are
test all applicants (not practical)
test a random sample of applicants
use statistical correction
What are some of the problems with criterion?
different criteria may not relate well to another
criterion contamination
criterion requirements change
_________ validity usually determines abstract attributes?
convergent?
content?
construct?
concurrent?
construct. the test scores reflect individual differences in a psychological construct.
The accuracy of decisions is directly linked to the _______ of test scores.
validity?
reliability?
representativeness?
sample?
validity. Validity is about accuracy and reliability is about consistency.
By showing that the behaviours sampled by the test are a representative sample of the domain of behaviours you could have measured, you have demonstrated that…
construct validity?
content validity?
concurrent validity?
both a & b
content validity. the test covers a representative sample of the domain being measured.
Two general strategies for assessing criterion related validity are…
content and construct validity
predictive and construct validity
concurrent and predictive validity
construct and concurrent validity
concurrent and predictive validity. concurrent: valid when tests are employed for diagnosis of an existing state. predictive: give a test to a group of people and measure then on the criterion at a future point in time.
The test scores ‘predict’ performance.
Range restriction is most likely to occur in..
concurrent validity studies
predictive validity studies
construct validity studies
both a & b
concurrent validity studies
Face validity
cannot influence the usefulness of a test
is typically examined in modern validity studies
depends on subjective reactions of those who take the test
depends heavily on objective data
depends on subjective reactions of those who take the test.
Obtaining both test scores and criterion scores at the same time from an intact preselected population, and then correlating the score is an example of
concurrent validity
predictive validity
construct validity
criterion-referenced validity
concurrent validity: tests are employed to diagnose an existing state
If multiple measures of the same construct yield high correlation coefficients, __________ is demonstrated
divergent validity
convergent validity
discriminant validity
concurrent validity
The ideal criterion related validity strategy is
predictive validity
concurrent validity
construct validity
face validity
predictive validity: give a test now to a group of people and measure them on the criterion at some future point in time.
