Week 3

Question 1

How many components does a test score have?

Answer 1

A test score has two components - the obtained score and the error

Question 2

Can we apply a CI to obtained scores?

Answer 2

Yes.
Xi = +/- 1.96 x SD

ti = +/- 1.96 x SEM

We do not know ti and SEM and need to estimate them

Question 3

How do we estimate the true score?

Answer 3

ti = M + rxx (Xi - M)

When reliability (rxx) = 1 (perfect score) then the obtained score will be the true score as both M’s cancel each other out.

If the reliability = 0, the test is considered unreliable.

ti = estimated true score
Xi = individual raw score
M = mean of scores
rxx = reliability of the test

Question 4

How do we estimate the standard error of measurement?

Answer 4

SEM = SD √1-rxx

If rxx = 0, that is the test is unreliable, the SEM is equal to the SD of the sample

If rxx = 1, the test is perfectly reliable and the SEM = 0.

Question 5

Define validity

Answer 5

The extent to which evidence supports the meaning and use of a psychological test.

Question 6

Describe the key difference between reliability and validity.

Answer 6

Reliability = consistency

Validity = accuracy

Reliability is necessary but not sufficient for validity i.e. if a test does not provide reliable (consistent) scores, the test cannot be valid (accurate).

Question 7

Face validity is

Answer 7

The test appears to test what it sets out to test

Question 8

Criterion-related validity

Answer 8

The test score predict performance

Concurrent validity (right now, existing state, useful to diagnose). Used to determine learning difficulty.

Predictive validity (future, how will this person learn in the future). Used most commonly for selection/classification

Example: VCE scores to determine entrance to Uni

Question 9

Content validity

Answer 9

The test covers a representative sample of the domain being measured

More relevant for achievement tests (i.e. exams)

Built into the test from the outset

Question 10

Construct validity

Answer 10

The test scores reflect individual differences in a psychological construct

for example anxiety, intelligence, extraversion are not directly observable but age differentiation, relationship to behaviour, experimental manipulation and criterion groups are

convergent validity
divergent validity

Question 11

What is a good level of validity?

Answer 11

About .2 to .5

Typically lower than reliability coefficients (.8)

Question 12

What is the role of validity in decision theory?

Answer 12

Even if rxy is low, a test may appreciably improve predictive efficiency.

We use a test if the benefit gained in terms of predictions outweighs losses in terms of costs incurred as a result of testing. They can increase base rate successes.

Question 13

Detail the factors that influence validity

Answer 13

Low sample size (may not reach statistical significance)

Restriction of range in either the test or criterion scores > creates a selectivity problem

non-linear relationships between test and criterion scores (assumption of Pearson’s correlation - when non linear a non-parametric test is required.

Problems with the criterion.

Question 14

Solutions for the selectivity problem are

Answer 14

test all applicants (not practical)

test a random sample of applicants

use statistical correction

Question 15

What are some of the problems with criterion?

Answer 15

different criteria may not relate well to another

criterion contamination

criterion requirements change

Question 16

_________ validity usually determines abstract attributes?

Answer 16

convergent?
content?
construct?
concurrent?

construct. the test scores reflect individual differences in a psychological construct.

Question 17

The accuracy of decisions is directly linked to the _______ of test scores.

Answer 17

validity?
reliability?
representativeness?
sample?

validity. Validity is about accuracy and reliability is about consistency.

Question 18

By showing that the behaviours sampled by the test are a representative sample of the domain of behaviours you could have measured, you have demonstrated that…

Answer 18

construct validity?
content validity?
concurrent validity?
both a & b

content validity. the test covers a representative sample of the domain being measured.

Question 19

Two general strategies for assessing criterion related validity are…

Answer 19

content and construct validity
predictive and construct validity
concurrent and predictive validity
construct and concurrent validity

concurrent and predictive validity. concurrent: valid when tests are employed for diagnosis of an existing state. predictive: give a test to a group of people and measure then on the criterion at a future point in time.

The test scores ‘predict’ performance.

Question 20

Range restriction is most likely to occur in..

Answer 20

concurrent validity studies
predictive validity studies
construct validity studies
both a & b

concurrent validity studies

Question 21

Face validity

Answer 21

cannot influence the usefulness of a test

is typically examined in modern validity studies

depends on subjective reactions of those who take the test

depends heavily on objective data

depends on subjective reactions of those who take the test.

Question 22

Obtaining both test scores and criterion scores at the same time from an intact preselected population, and then correlating the score is an example of

Answer 22

concurrent validity
predictive validity
construct validity
criterion-referenced validity

concurrent validity: tests are employed to diagnose an existing state

Question 23

If multiple measures of the same construct yield high correlation coefficients, __________ is demonstrated

Answer 23

divergent validity
convergent validity
discriminant validity
concurrent validity

Question 24

The ideal criterion related validity strategy is

Answer 24

predictive validity
concurrent validity
construct validity
face validity

predictive validity: give a test now to a group of people and measure them on the criterion at some future point in time.