Validity Flashcards
Define Validity.
The extent to which the test measures what it purports to measure
What is Construct Validity?
The meaning of a test score made possible by knowledge of the pattern of a relationships it has with other variables and the theoretical interpretation of those relationships
Explain Face Validity.
The extent to which a measurement or assessment appears, on the surface, to measure what it intends to measure.
It is a subjective judgment based on whether a test or measurement looks like it is measuring the intended construct or concept.
What is Content Validity?
The extent to which a measurement or assessment covers all the important aspects or content of the construct being measured.
It is concerned with whether the items or questions included in the measurement tool represent the full range of the construct being studied.
What are the differences between Face Validity and Content Validity?
- Face validity is a subjective judgment based on appearance or perception, while content validity is an objective evaluation based on the coverage of important content
- Face validity is a surface-level evaluation of whether a measurement “looks like” it measures the intended construct, while content validity involves a thorough analysis of whether the measurement adequately represents the full range of the construct
- Face validity is an initial and quick assessment, often used as a rough indicator, while content validity involves a more rigorous and systematic assessment to ensure comprehensive coverage.
What is Predictive Validity?
The extent to which a score on a psychological test allows a statement about standing on a variable indexing important social behaviour independent of the test.
What is Concurrent Validity?
A form of predictive validity in which the index of social behaviour is obtained close in time to the score on the psychological test
What is a method used to determine one’s predictive validity?
Regression
How is the correlation between test and criterion often evaluated?
r^2
Under Coefficient of Determination, what does correlation of 0.3 suggests?
This suggests that the scores on the test account for 9% (r^3) of the variance in the criterion
Explain Incremental Validity.
An extension of predictive validity
It pertains to the evaluation of unique contribution a test has over other existing tests in predicting an outcome
Explain Standard Error of Estimate.
An index of the amount of error in predicting one variable from another
What are the 4 types of decision-theoretic approach?
- Valid Positive Decisions
- Valid Negative Decisions
- False Positive Decisions
- False Negative Decisions
Explain valid negative decision (C)
Assumption: Does not have characteristic
Reality: Does not have characteristic
Explain valid positive decision (B).
Assumption: Have characteristic
Reality: Have characteristic
Explain false negative decision (A).
Assumption: Does not have characteristic
Reality: Have characteristic
Explain false positive decision (D).
Assumption: Have characteristic
Reality: Does not have characteristic
How to calculate the Base rate (BR) of criterion behavior in the population of interest?
(A + B) / N
What is Base rate?
The proportion of individuals who have the characteristic in a population (or sample of)
What is Selection ratio?
Proportion of individuals selected (or categorised positively) based on test cut-off score
How to calculate selection ratio?
(B + D) / n
How to calculate valid positive rate?
B / n
The number of valid positives divided by the total number of persons tested
How to calculate hit rate?
(B + C) / n
The sum of number of valid positives and valid negatives divided by the total number of person tested
Define test sensitivity.
B / (B + A)
How well does the test do in identifying those who have the characteristic
Define test specificity.
How good is the test in ruling out those who do not have the characteristic
C / (C + D)
What is Multitrait-multimethod Matrix?
A correlation matrix of scores from various tests administered via different methods (i.e., self-report, observation)
If there is a low correlation observed on two tests that are supposed to be measuring the same construct, what does it imply?
It suggests that the test has failed convergent validity because tests measuring the same construct should have a high correlation.
What is criterion validity?
It focuses on the overall construct that the measure’s scores are supposed to be indicative of, rather than the actual content of the measure.
(gold-standard)
Explain what Y = mX + C each represents.
X = linear relationship between test/predictor
Y = Criterion
This allows for the prediction of Y through sheer knowledge of the X value itself
Explain Factor Analysis.
It is a mathematical method of summarising a matrix of values (such as the inter-correlation of test scores) in terms of a smaller number of values (factors) from which the original matrix can be reproduced
What is the purpose of Rotation?
It helps to maximize the loadings on one factor and minimizes loadings on another
List down 2 different types of Rotation techniques.
- Orthogonal Rotations
- Oblique Rotations
What is the angle for Orthogonal rotations and are they correlated?
90 degrees; not correlated
What is the angle for Oblique rotations and are they correlated?
< 90 degrees; correlated to a certain extent
What does h^2 refers to?
It refers to the proportion of variance (multiply by 100 to obtain % of variance)
When h^2 = 0.9832 for a specific item, what does it mean?
98% of the specific item’s score variance is accounted by Factor A and B combined
How do we calculate V for a Factor?
Square each of its loadings and summing them up, then dividing by the number of items
V = 0.54 means?
The factor accounts for 54% of the entire test’s score variance
What is Exploratory Factor Analysis?
This method of factor analysis is typically used during early test development
It lets the data “speak for itself”, where software decides on the dimensional cuts that best accounts for most of the variance in the data
What is Confirmatory Factor Analysis?
Used when you have an existing measure where you know what it is supposed to measure
You tell the software that you are supposed to see a specific factor structure just indicating one construct, and the software will tell you if it fits well with the data obtained
What are the 2 types of EFA?
- Principal Components Analysis (PCA)
- Principal Axes Factoring (PAF)
