Comps - Assessment (Validity)

Question 1

Validity

Answer 1

The degree to which evidence and theory support the interpretations of test scores entailed by the proposed uses of a test

Question 2

If validity is not a property of a test, what is it?

Answer 2

It is a property of how the clinician uses the test

Question 3

Tests are not valid in and of itself; __________ made about test scores and use of test scores are more or less valid

Answer 3

Inferences

Question 4

Content Validity

Answer 4

Examining the match between the content of the test and the content that should be included on a test of that particular attribute (extent to which questions on a test represent the construct being measured)

Question 5

What is a good question to conceptualize content validity?

Answer 5

How well does the content match what should be on the test?

Question 6

What do you want to avoid if you want a high content-valid test?

Answer 6

Construct-irrelevant content (items that are not related to the content being measured)

Question 7

Threats to Content Validity

Answer 7

-Inaccurate knowledge
-Construct underrepresentation (something important about the content is missing)

Question 8

Face Validity

Answer 8

The extent to which the items look like they are measuring the construct

Question 9

Content Validity Example

Answer 9

If a therapist is using a questionnaire to assess depression, the questionnaire should include questions that cover all the symptoms of depression, such as mood, energy levels, sleep patterns, and appetite. It ensures that the test comprehensively covers all relevant aspects of depression

Question 10

Criterion Validity

Answer 10

Correlation between a test score and some external criterion (e.g., a validated test/measure) – compares it against some other measure or outcome already considered to be valid

Question 11

Forms of Criterion Validity

Answer 11

-Convergent Validity
-Discriminant Validity
-Concurrent Validity
-Predictive Validity

Question 12

Convergent Validity

Answer 12

This checks if the test correlates well with other tests that measure the same or similar constructs

Question 13

Discriminant Validity

Answer 13

This checks if the test does not correlate too strongly with tests that measure different constructs

Question 14

Concurrent Validity

Answer 14

This checks if the test correlates well with a criterion measure taken at the same time

Question 15

Predictive Validity

Answer 15

This checks if the test can predict future outcomes or behaviors

Question 16

Convergent Validity Example

Answer 16

A new depression questionnaire (Designer Depression Inventory) should have similar results compared to an established depression test (Beck Depression Inventory)

Question 17

Discriminant Validity Example

Answer 17

The depression questionnaire should not have high correlations with a test measuring something different, like delusions

Question 18

Concurrent Validity Example

Answer 18

If a therapist gives a new depression test (Designers Depression Inventory) and the established depression test (Beck Depression Inventory) at the same time, they should both show similar results

Question 19

Predictive Validity Example

Answer 19

If a therapist uses an assessment to predict how well a client will respond to a specific therapy in the future, the results should match the actual outcomes

Question 20

Factors impacting the size of the reliability coefficient

Answer 20

-Restriction of range issues
-Reliability of predictor and criterion measures
-Sample size
-Screening people into dichotomous categories
-Sensitivity and specificity

Question 21

Restriction of Range Issues

Answer 21

Not representative of a larger population (e.g., if I am looking at depression in college students, it will not be representative of an adult population)

Question 22

Reliability of Predictor and Criterion Measures

Answer 22

As the reliability for either the predictor or the criterion gets lower, the coefficient is less likely to be significant

Question 23

Spearman’s Correction for Attenuation

Answer 23

-Correcting for measurement error to strengthen your predictor
-Takes out the effective measurement error for the coefficient
-Reduces the weakening effect of measurement error and specifies the actual correlation between the constructs if they were both perfect and unaffected by error

Question 24

Sample Size

Answer 24

As the sample size increases, the reliability coefficient needs to be smaller in order to be considered significant (you want it to be 0 to account for as little error as possible)

Question 25

Screening People into Dichotomous Categories

Answer 25

Involves setting a cut score between two categories

Question 26

Sensitivity

Answer 26

Proportion of actual positives; appropriately identifying people with the construct (low type II error – miss)

Question 27

Specificity

Answer 27

Proportion of true negatives; appropriately identifying people without the construct (low type I error – false positive)

Question 28

Construct Validity

Answer 28

How well a test measures the theoretical construct it is supposed to measure. This involves ensuring that the test truly reflects the concept in question and that the concept itself is well-defined. Subsumes both content and criterion-related validity

Question 29

Construct Validity Example

Answer 29

If a therapist is using a test to measure self-esteem, the test should measure self-esteem and not some other related concept like confidence or social skills

Question 30

Messick’s Unified Theory of Construct Validity (MUTCV)

Answer 30

Six aspects to construct validity

Question 31

MUTCV Aspect 1: Content Validity

Answer 31

-Content of a test
-Ex: content of Designers Depression Inventory represents all the facets of depression

Question 32

MUTCV Aspect 2: Criterion Validity

Answer 32

-Relationship to external factors
-Ex. when the Designers Depression Inventory is compared to the BDI, clients should receive similar scores

Question 33

MUTCV Aspect 3: Psychological Processes

Answer 33

-How well do processes that respondents actually use match the processes they should theoretically be using?
-Ex 1. does a processing speed test require processing speed skills/abilities?
-Ex 2. when taking the Designers Depression Inventory, they should naturally have negative affectivity and pessimistic traits that align with depression symptoms

Question 34

MUTCV Aspect 4: Internal Structure of a test

Answer 34

-How parts of the test are related
-Factor analysis (evaluates the internal structure/dimensionality of the test; the goal is to find the smallest number of meaningful underlying variables/factors that account for the pattern of relationships among responses)
-Ex. Designer Depression Inventory – when testing for depression, wanting to see if mood, cognitive, and behavioral overlap within the inventory or if they are separate

Question 35

Exploratory Factor Analysis

Answer 35

Wanting to find which factors overlap

Question 36

Confirmatory Factor Analysis

Answer 36

Wanting to confirm that the factors do overlap

Question 37

MUTCV Aspect 5: Generalizability

Answer 37

-Extent to which score properties and interpretations generalize across population groups/settings/tasks
-Ex. Designer Depression Inventory – testing on IUP students, would want to make sure an urban/diverse population has similar results

Question 38

MUTCV Aspect 6: Consequences of Test Use

Answer 38

-Validity includes unintended and intended consequences of test use
-Does the test unfairly or adversely impact some people more than others?
-Ex. Designer Depression Inventory – testing on a Western population with a focus on emotional focus on depressive symptoms, but in Latin populations they have more of a focus on physiological symptoms of depression

Question 39

Type I Error

Answer 39

False positives

Question 40

Type II Error

Answer 40

False negatives