Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

Psychometric Final! > Validity > Flashcards

Validity Flashcards

1
Q

What is the distinction between validity and reliability?

A 
Validity = measurement of the test is valid when it measures what it is supposed to.
Reliability = consistency of measurement
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Why is it imprecise to say that a test is valid?

A

• Because validity refers to the measurement that is attained by use of the test

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

Construct underrepresentation:

A

Construct underrepresentation: aspects of the construct not assessed by the measure (maybe due to poor construction of the measure or changing knowledge of the construct)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

Construct irrelevant variance:

A

variance in test scores related to things unrelated to the construct (e.g., random and systematic error). Reduces valid measurement.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Valid measurement:

A

overlap where measurement is measuring what it is supposed to

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

How does increased or decreased validity play into the relationship between these?

A

How does increased or decreased validity play into the relationship between these?
Decreased validity = more construct underrepresentation and/or more construct irrelevant variance.
Increased validity = less of those.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

What is validity of measurement?

Already discussed above… maybe look into in more detail in the additional readings section.

A

overlap where measurement is measuring what it is supposed to… maybe look into in more detail in the additional readings section.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

What are content and construct validity? How do these differ?

A
  • Content validity: examination of aspects of the test itself. Confirmatory, non-statistical/non-quantifiable process of determining how well a measure overlaps with the construct.
  • Construct validity: examination of relationship between test scores & scores of other measures

• Content validity only examines aspects of the test itself, whereas construct validity compares the test scores with scores on another measure of the same construct

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

Construct validity

A

Construct validity: examination of relationship between test scores & scores of other measures

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Content validity

A

Content validity: examination of aspects of the test itself. Confirmatory, non-statistical/non-quantifiable process of determining how well a measure overlaps with the construct.
Content validity only examines aspects of the test itself, whereas construct validity compares the test scores with scores on another measure of the same construct

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

Determine the areas of the content domain that are measured by each test item

A

Boundary = identification of all sub-constructs that exist within that construct.
• Structure = relative weight or importance of each sub-construct within the larger construct (determined by examining research)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Determine the areas of the content domain that are measured by each test item

A

Making sure all sub-constructs are represented by items:
o To account for weight: adjust # of items or adjust values in scoring
o This avoids construct irrelevant variance

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

Compare the structure of the test with the structure of the content domain

A

Need to ensure your measurement reflects construct

o Increases content validity, decreases construct irrelevant variance & underrepresentation

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

Why is an assessment of content validity seen less in psych measurement than educational testing?

A

It is harder to assess content validity with psychological constructs. Our knowledge and conceptualization of the constructs change, making it difficult to identify boundary and structure.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

What is face validity?

A

The degree to which a measure appears to measure the target construct.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Why is it good to have a measure that appears to be valid, but not at the expense of empirical validity?

A

High face validity can increase participants’ likelihood of continuing to respond. People hate wasting their time.
However, you may need low face validity to get honest answers for some constructs (e.g. racist attitudes).

17
Q

What are the methods of determining construct validation?

A
  1. Correlational study: Correlations between measures of certain behaviors (either related or unrelated to our construct) and our test.
    • Likelihood of 2 measures failing to tap into the same aspect of a construct is small.
    • Easiest and cheapest option
  2. Factor analysis: analyzing which groups of items “hang” together.
    • Compares factor structure that results from a measure with the assumed factor structure of the construct.
    • Gives a determination of which items conceptually intercorrelate/covary
    • Hard to do and requires huge sample size
  3. Experimental manipulation: manipulate the construct of interest (e.g., induce fear) and see if it relates to scores on your test.
    • Best way EXCEPT problems:
    o ethics (e.g., unethical to induce PTSD)
    o only works with stable constructs
    o more expensive and time-consuming
18
Q

What is the difference between convergent and discriminant validity? What would you expect from each in terms of the resulting correlation analysis?

A

Convergent validity: we want significant correlations between test & measures of behaviors related to construct
Discriminant validity: we want non-significant correlations between our test & behaviors unrelated to construct

We expect a significant positive correlation when things are related (convergent validity).
We expect a nonsignificant correlation when things are not related (discriminant validity).

19
Q

What types of information can be gained from use of the multitrait-multimethod matrix approach?
Note: You will need to determine and interpret this information from a matrix on the exam (see slides).

A

MTMM allows you to determine 3 things:

  1. Convergent validity
  2. Discriminant validity
  3. Check for method bias
20
Q

What is method bias?

A

If there is a significant relationship (one or a couple) within the same method, it may indicate a training issue or a conceptual overlap between two traits.

21
Q

What is a criterion?

A

A measure that could be used to determine the accuracy of a decision. Outcome measure.

22
Q

What is criterion-related validity? (aka decision-making validity)

A
  • Quantitative way to evaluate how successful decisions made from the test’s results are.
  • We compare scores on our measure with scores on a criterion assessment (an outcome measure that helps you determine the accuracy of decisions).
  • There is high criterion-related validity when there are significant correlations between our test and the criterion assessment.
  • Assessed using either prediction validation strategies or concurrent validation strategies.
23
Q

Why is a predictive validation strategy considered the ideal approach? Why isn’t it always used?

A
  • Maximizes potential for covariance (avoids truncated range)
  • BUT not always ethical or cost effective
24
Q

How does one conduct a predictive validation procedure?

A

You screen everyone and then accept everyone (so you can see the full possible range in later assessment).

  1. Obtain test scores
  2. Obtain performance measures
  3. Correlate these with the test scores
25
Q

How does this differ in a concurrent validation procedure?

A

Timing:
Test scores and criterion scores from a preselected population are obtained at the same time (after decision has been made) and then a correlation is conducted between the two.

26
Q

Know the strengths and weaknesses of the predictive and concurrent validation approaches:

A

Predictive Validation:
• Strengths: maximizes potential for covariance (avoids truncated range), screens out failures.
• Weaknesses: not always ethical or cost effective
Concurrent Validation:
• Strengths: much more practical, easier, validity coefficients are often similar to predictive anyway
• Weaknesses: only get info from people already accepted, range is more restricted (reduces correlation)
Range restriction not that important unless it’s highly restricted already

27
Q

What is range restriction? How can this impact our validity for decisions?

A

• Limited variance (if you limit who makes it through, you only know how good your screen is for top performing individuals).
Reduces correlation between predictor (test score) and criterion (outcome measure). Less predictable.

28
Q

How do you determine the variance in the criterion that can be explained by the predictor measure?

A

Square the validity coefficient.
Ex. Using a test of IQ to predict job performance results in a validity coefficient of 0.5 so we can say that 25% of the variance in job performance (criterion) is accounted for by IQ (predictor measure)

29
Q

Know the differences between and how to calculate base rate and selection ratio.

A
  • Base rate: level of performance on criterion in the general population (e.g., if 75% of the population would be successful, the base rate = .75). Easy task = higher base rate. Hard task = lower base rate.
  • Selection ratio: ratio of positions to applicants (e.g., if 30 people apply for 3 jobs, the selection ratio is .10 or 10%)
30
Q

Know the possible decision outcomes. Which of these outcomes are we looking to maximize?

A 
LOOK IN SLIDES AT CHART

Accept
Reject
Actual Performance
Success
True Positive
BR x SR
False Negative
BR – (BR x SR)
Failure
False Positive
SR – (BR x SR)
True Negative
1 – all of those (they add to 100%)
31
Q

What is the impact of a large base rate?

A

High true positives and high false negatives

32
Q

What is the impact of a small base rate?

A

High false positives and high true negatives

33
Q

What level of base rate is ideal in terms of accurate decision making?

A

.50 which is moderate

34
Q

What level of selection ratio has the biggest impact on correct decision making?

A

Low level of selection ratio.
• High selection ratio means nothing matters because you’re accepting everyone.
• Low selection ratio is much more stringent.

35
Q

Do know the formulas that I presented in class regarding how to calculate the likelihood of obtaining each of the possible decision outcomes… both with and without the use of a test as a predictor.

A 
P(TP)=BRxSR+r Sqroot of {BR(1-BR)}{SR(1-SR)}
Where P(TP) = probability of true positive
BR = base rate
SR = selection ratio
Then related cells can be determined the same was as before (see table above).

Psychometric Final! (3 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions