Wk 5 - Validity

Question 1

If the validity coefficient between a test and its criterion measure (where a high test score should predict a high criterion score) is -.97 (minus point nine seven) and is statistically significant then this probably indicates… (x1)
Because… (x4)

Answer 1

The test could be reliable but is definitely not valid
The negative correlation indicates that the test has an inverse relationship with the criterion -
When it ought to have a positive correlation if it was valid.
However high correlation suggests the test is probably reliable (if unreliable, correlation would be much closer to zero)

Question 2

Factors that may affect a predictive validity coefficient do NOT include… (x1)
Because… (x1)

Answer 2

The mean score on the test (assuming no ceiling or floor effects)
As calculating the correlation coefficient involves standardizing the variables anyway

Question 3

True or false, and why? (x2)

Construct irrelevant variance refers to the variance in a CONSTRUCT that does not covary with TEST scores

Answer 3

False
Construct irrelevant variance refers to the variance in the TEST that does not covary with the CONSTRUCT
(not the other way around, as stated in the question)

Question 4

True or false, and why? (x2)
If the variances of a test and the construct it is attempting to measure only overlap by a small degree then the test is likely to have low reliability.

Answer 4

False
If variances of test and the construct it’s measuring only overlap by a small degree then the test is likely to have low VALIDITY
(not RELIABILITY as it says in the question)

Question 5

True or false, and why? (x4)
Non-random attrition between two time points in a longitudinal validation study is one of the factors that could potentially compromise the evaluation of the CONCURRENT validity of a test (assuming the test is administered during the initial time point).

Answer 5

False
CONCURRENT validity involves administering both test and criterion measures AT SAME TIME
So if people drop out after the initial time point it won’t matter -
We already have all the data we needed
(though it may affect any evaluation of the test’s PREDICTIVE validity)

Question 6

True or false, and why? (x2)
In a validation study for a behavioural measure, you discover that self-selection biases in your sample are influencing the spread of scores for the measure. This could compromise the evaluation of the CONCURRENT validity of the test.

Answer 6

True
Anything that affects the spread of scores in a test may affect its correlations with other variables
(which is what we’re analysing when we evaluate the concurrent validity of a test)

Question 7

True or false, and why? (x1)
Evaluating a test by seeing if it does not correlate highly with a construct it is not supposed to be measuring is an example of deviating validity.

Answer 7

False.

It’s an example of discriminant or divergent validity

Question 8

True or false, and why? (x2)

A factor analysis involves mathematically grouping items according to the similarity of their content.

Answer 8

False
Factor analysis involves mathematically grouping items according to their inter-correlations
Not similarity of content (there’s no way the factor analysis can “know” what the content of the items is)

Question 9

True or false, and why? (x1)

If a test has poor face validity then this may have implications for the data that the test yields.

Answer 9

True

Poor face validity can lead to things like missing data

Question 10

True or false, and why? (x2)
Content validity is not important for a university examination as long as that examination is supported by empirically-based validity evidence

Answer 10

False.
Even if we could create a test that discriminated between good and poor students in the course (i.e. it had empirically-based validity),
Still a problem if it did not do this by measuring knowledge of course content directly.

Question 11

True or false, and why? (x1)
You can test the incremental validity of a test by seeing whether it can predict some relevant criterion measure in isolation from other measures.

Answer 11

False
Incremental validity is about whether a test contributes to predicting some outcome IN ADDITION TO the effect of other measures

Question 12

True or false, and why? (x2)
If we had an established intervention known to reduce state anxiety then we potentially could use this to test the validity of a new measure of state anxiety.

Answer 12

True
We can use intervention as part of experiment to see if intervention reduces scores in the new test in ways we’d predict if test is valid
(compared with some placebo intervention)

Question 13

True or false, and why? (x2)

It is possible for a test to have excellent reliability but poor validity.

Answer 13

True
You don’t need validity to have good reliability
(your test can be consistent in the scores it produces without measuring what you want it to)

Question 14

True or false, and why? (x3)

It is possible for a test to have excellent validity but poor reliability.

Answer 14

False
Need good reliability for chance of test being valid
(because the level of reliability places a ceiling on how high your validity coefficient can be).
If your measure is producing wildly inconsistent scores then it’s probably not measuring anything

Question 15

True or false, and why? (x3)
If the reliability of both a test and a criterion measure are high then this means the correlation between them should also be high.

Answer 15

False
If reliability of both a test and a criterion measure is high then the correlation between them is not restricted –
However, this doesn’t mean it can’t be small
(the correlation can be high or low, depending on the validity of the test)

Question 16

True or false, and why? (x2)

Content validity is empirical data supporting the hypothesis that the content of a test is appropriate

Answer 16

False
Content validity involves opinions
And is not generally based on empirical data.

Question 17

When students complain that, in a course examination, a lecturer did not ask any questions from a particular lecture, they are effectively complaining about the examination having… (x1)

Answer 17

Potentially poor content validity

Question 18

Why is it not strictly accurate to talk about the validity of a test (hint: one test could be used in more than one context)? (x3)

Answer 18

It’s interpretations of test scores required by proposed uses that are evaluated, not the test itself.
When test scores are used or interpreted in more than one way, each intended interpretation must be validated
Because test can be used in different contexts – validity can change

Question 19

What are constructs? (x2)

Plus egs x4

Answer 19

Unobservable, underlying hypothetical traits or characteristics
That we can try and measure indirectly using tests
Intelligence, anxiety, plumbing skill, speeding propensity

Question 20

What is contruct underrepresentation? (x1 plus e.g. x1)

Answer 20

The portion of variance in the construct that is not captured by our test
self report assumes insight, into eg being slow or fast drivers, but the chances of perfect insight are very slim – so there’s things you don’t capture in the test

Question 21

What is construct irrelevant variance? (x1 plus e.g. x 1)

Answer 21

Stuff that’s captured by the measurement, but not part of the construct
Eg speed questionnaire – variance that’s to do with interpreting the wording, social desirability issues, not reflection of speed of driving

Question 22

What are the similarities (x2) and difference between content and face validity? (x2)

Answer 22

Both are opinion-based, not empirical, but…
Face is how valid the test appears to be, from the perspective of the test taker (usually), while
Content is judgment (usually by experts) regarding how adequately a measure samples behaviour representative of the universe of behaviour it was designed to for

Question 23

Can we say that ‘test is valid’? (x2)

Answer 23

No

Only that validity hypotheses are supported - not all or nothing

Question 24

What is ‘valid measurement’? (x2)

Answer 24

That aspect of the characteristic/ability/trait which IS captured by both the construct and the test
Variance that is due purely to the trait, not construct irrelevant variance or construct underrepresentation

Question 25

Variance in test measurement is made up of what two components?

Answer 25

Valid measurement

Construct irrelevant variance

Question 26

Variance in a construct is made up of which two components?

Answer 26

Construct under-representation

Valid measurement

Question 27

How can face validity be useful (x2), or not (x2)?

Answer 27

Good for PR - people can see the point
They accept test and take it seriously, less missing values
But may have terrible actual validity -
Just because it feels like driving a car, doesn’t mean it tells us about RL driving behaviour

Question 28

How could you go about evaluating content validity? (x2)

Answer 28

Getting experts to rate each item for its relevance to the construct - e.g. exam questions for course content
Getting those experts to make judgements on whether particular lectures were over- or under-represented.

Question 29

How could I create an exam that had great empirical validity but poor content validity? (x7)

Answer 29

Ask about known correlates of academic success:
hours spent studying, assignment marks, motivation, GPA, questions based on other prerequisite courses, number of lectures/tutorials attended

Question 30

What is the general process involved in testing empirical validity? (x2)

Answer 30

Create hypotheses regarding how your measure ought to perform if it is valid
Then design and run studies to test these hypotheses

Question 31

Give two examples of things that might restrict the range of scores in a test and indicate what influence this could have on the validity coefficient.

Answer 31

Non-random attrition: certain types of people dropping out of your longitudinal study - e.g. those too sick or not very sick dropping out of study
Self-selection: only certain people in the sample in the first place - we can’t test randomly in RL

Question 32

What is criterion validity? Give 5 examples.

Answer 32

Judgment regarding how adequately a score on a test  can be used to infer an individual’s most probable standing on some measure of interest (the criterion)
Method of contrasted groups
Concurrent
Predictive
Incremental
Convergent

Question 33

What is the method of contrasted groups? (x2 plus egs x 2)

Answer 33

Establish criterion validity by determining whether test scores of groups of people vary as expected
Eg groups we think will score high or low – does the test tell them apart?
Clinical group vs non-clinical controls (e.g. dental phobia patients vs controls for dental phobia questionnaire)
Experts vs novices for a skill or ability test

Question 34

Give two examples of criterion contamination, explaining what is contaminated in each case. (x2 and x3)

Answer 34

Validate schizophrenia test by finding it can tell apart those with/out
BUT then discover that people with schizophrenia were originally diagnosed using my test (validity test is circular)
Zuckerman sensation seeking scale: validated by comparing scores with a risk-taking behaviour scale (the criterion),
But virtually the same items appeared in both test and criterion
ie many questions directly ask if you like to take risks

Question 35

What are the effects of restriction of range of continuous variables? (x1)

Answer 35

Giving us a much smaller correlation than we would get from full range

Question 36

How big does a validity coefficient have to be? (x1 plus egs x 2)

Answer 36

Depends entirely on the context
.2 is awesome for crash risk – most people only crash every ten years, and only small portion of any sample will have had a crash in eg past yeardepends entirely on the context
But, if correlating a new test of intelligence with an older one, might expect more like .80 (no excuse for it to be smaller if it’s measuring the same thing)

Question 37

What is concurrent validity? (x2)

Answer 37

Subset of criterion validity

Where test and criterion are measured for each person at the same time/in same session

Question 38

What is predictive validity? (x2 plus e.g. x1)

Answer 38

Subset of criterion validity
Where the test is trying to predict what the criterion will be at some future time
Speed test predicts crashes over next year

Question 39

What is incremental validity? (x3)

Answer 39

Type of criterion validity
Can be predictive or concurrent
How much each individual predictor ADDS to predicting the criterion in addition to the effect of other predictors

Question 40

Discuss two examples of incremental validity (x3 each)

Answer 40

Predict bungee jump:
Sensation-seeking, Past risk-taking behaviour, Susceptibility to peer pressure, Fear of heights
Alone might have poor criterion validity, using together (e.g. combined using Multiple Regression) improves it
Eg crash risk:
Speeding propensity; hazard perception skill; fatigue; km driven/wk; drunk; tailgaiting propensity; traffic violations; distraction; years experience; age
(last two correlate highly, so using both doesn’t add much value – doesn’t change you validity coefficient)

Question 41

What is convergent validity? (x2 plus e.g. x1)

Answer 41

Does it correlate with similar things?
Convergent and criterion are not mutually exclusive – many things might be equally described by either term
eg if anxiety measure is valid, we can expect it to correlate with established measures of anxiety

Question 42

What is discriminant/divergent validity? (x1 plus e.g. x2)

Answer 42

Does it not correlate with dissimilar things?
e.g. new measure of depression does NOT correlate highly with validated measures of anxiety
(shows it isn’t just a general measure of maladjustment and is specifically measuring depression, because depression and anxiety are different things)

Question 43

What is the criterion validity coefficient?

Answer 43

Correlation between the test scores and the criterion scores

Question 44

Why do both the test and its criterion have to have decent reliability? (x2 plus explain x4)

Answer 44

Reliability of each limits size of validity coefficient
So, noise/measurement error in the data makes the correlation smaller, due to scores being more spread out
Eg, imagine a perfect match of test onto criterion (correlation of 1)
• Then introduce uncertainty/inaccuracy/unreliability into the tests scores…
• Perfect correlation is spoiled by smearing the scatterplot
• Same would happen if the criterion scores were unreliable

Question 45

What questions does factor analysis answer? (x4)

Answer 45

Is the internal structure as expected? Does it map onto theory?
Multi-faceted (multiple subscales; heterogeneous)?
Or no subscales, homogeneous?

Question 46

How does factor analysis work? (x2)

Answer 46

Uses mathematical techniques to group the items into clusters/factors/components based on how well they correlate with each other
Picks out the distinct clusters of items in our data based on their inter-correlations -tells us to what extent the constructs tie together

Question 47

What is construct validity? (x3)

Answer 47

Umbrella term covering water we are measuring the right thing…
Argued to cover all validity methods
How well the scores on your test reflect the construct (i.e. the trait or characteristic) that your test is supposed to be measuring.

Question 48

Define criterion (x1)
Plus eg x1
And four contingencies for deciding on a criterion measure

Answer 48

The standard against which the test is evaluated (e.g. actual driving speed was one of the criteria used to validate the speed questionnaire).
Needs to be reliable, valid, relevant, and not subject to criterion contamination

Question 49

Give 6 egs of criterion validity checks

Answer 49

Surgery competence simulator test: correlation between test score and patient outcome rating
University admissions test, with GPA at end first year;
Depression inventory with clinician rating of severity;
Salesperson personality scale with amount of worthless crap sold to gullible people;
Clerical aptitude with supervisor’s job performance rating;
Creative thinking test with panel rating of product

Question 50

Are face and content validity necessary for actual validity? (x3)

Answer 50

No
Measure with terrible face validity could still be great predictor, e.g. risk-taking and driving behaviours
Could have psychometric validity sans content - e.g. GPA as predictor of course performance, rather than exam

Question 51

What size correlation is generally necessary to say that an item belongs in a particular cluster (in factor analysis)?

Answer 51

Greater than .4, with larger being better

Question 52

How do we use experimental effects to check validity? (x2 plus egs x2)

Answer 52

Are they as expected?
ie it’s been shown in a bunch of studies what the effect is supposed to be
Eg new test of state anxiety:
• Give them all the test, split into two groups (treatment/placebo)
• If valid, intervention group should differ from placebo
Eg hazard perception: subject 25 people to a validated training course and see if scores improve on the test compared with control group/ pre-training scores

Question 53

How might we use developmental changes to assess validity? (x1 Plus e.g. x1)

Answer 53

Diffs in test scores as you might predict if your measure was valid?
Eg everyone would agree that language skills improve with age

Question 54

What are the components of the WISC-IV Wechsler Intelligence Scale for Children

Answer 54

10 core subtests arranged into 4 groups:
•	Verbal comprehension index
•	Perceptual reasoning index
•	Working memory index
•	Processing speed index
Plus 5 supplementary tests for additional/replacement tests

Question 55

Describe the properties of the WISC-IV (x6)

Answer 55

Large, representative normative sample across 11 age groups (6 - 16 yrs)
Mean = 100m, SD = 15
Excellent internal consistency, test-retest and inter-rater reliability
Need diff of 7.58 IQ points (2 x SEdiff =3.79) for confidence that not due to chance
Good content validity
And empirical: predicts academic achievement, diffs between normal/special groups, confirmatory factor analysis

Question 56

Factors that may affect a predictive validity coefficient do include… (x3)

Answer 56

Internal consistency of the criterion
Certain types of people dropping out of the sample between the original test and when the criterion is measured
Certain types of people agreeing to be in the sample.