F&B Validity Flashcards
What is meant by validity?
the degree to which evidence and theory support the interpretations of test scores entailed by the proposed uses
What part of a test is seen as valid or invalid
The interpretation of the score and its implications; not any part of the test
How is validity traditionally a three faceted concept?
Content validity- the degree to which test scores can be interpreted as as reflecting a particular psychological construct
Criterion validity - the degree to which test scores can predict specific criterion variables.
Construct validity
What five types of evidence relevant for establishing the validity of test score interpretation was published in the Standards for Educational and Psychological Testing?
Internal structure Response processes Test Content Consequences of use Associations with other variables
Name 2 threats to content validity
Construct-irrelevant content
Construct underrepresentation
What is meant by face validity?
the degree to which a measure appears to be related to a specific construct, in the judgment of non-experts, such as test takers and representatives of the legal system
Why may test validity be relevant?
The apparent meaning and relevance of a lest’s content might influence lest takers’ motivation to respond in a serious and honest manner.
What is meant by the internal structure of a test?
The way the parts of the test are related to each other (correlated items or clusters etc)
What should the internal structure of a test match if is to be internally valid?
Internal structure of the theory
What test examines the internal structure of a test?
Factor analysis
How are the results of a factor analysis displayed for “global” and for multidimensional tests
Global- scree plot (eiganvalues)
Multidimensional- orthogonal rotation
What is the facet “associations with other variables” concerned with?
The match between a measure’s actual associations with other variables and the associations is actually has with other variables
When evaluating the pattern of validity associations between a measure and other measures, what evidence should we consider? (2)
Convergent evidence- the degree to which scores are correlated to tests of related (or the same) constructs
Discriminant evidence- the degree to which test scores are uncorrelated with tests of unrelated constructs
Distinguish between concurrent validity evidence and predictive validity evidence
Concurrent validity evidence refers to the degree to which test scores are correlated with other relevant variables that are measured at the same time as the primary test of interest.
Predictive validity evidence refers to the degree to which test scores are correlated with relevant variables that are measured at a future point in time.
concurrent validity evidence and predictive validity evidence are variants of what type of what type of evidence
Convergent evidence
Name two big differences between the contemporary and traditional perspective on validity
-the contemporary perspective emphasizes the primacy of construct validity over content
validity and criterion validity
-The assertion from the contemporary perspective that the social consequences are a facet of validity
According to the contemporary view, what should criterion validity be categorised under?
Construct validity
How would an inductive view of testing validity look?
by examining the associations between test scores and a large set of potentially
important and relevant psychological variables rather than testing the hypothesis of what is is assumed to test
When would an inductive method usually be utilised?
In a research context
What is meant by the nomological network?
The interconnections between a construct and other related constructs
Name four methods for evaluating convergent and discriminant validity
Focused associations
Sets of Correlations
Multitrait-Multimethod Matrices
What is involved in focused associations?
When measures have clear relevance for very few very specific variables
What are meant by validity coefficients?
Correlations between test scores and crucial variables
What is meant by validity generalisation?
Process of evaluating a tests validity coefficients across a large set of studies, it is a form of meta-analysis- it studies a range of settings, times, situations etc
What three issues does a validity generalisation study address?
- Can reveal the general level of predictive validity across all of the smaller individual studies
- Can reveal the degree of variability among the smaller individual studies (determining generalisability)
- The source of variability among studies (confound/ moderate etc)
When is a set of correlations utilised?
When a constructs nomological network touches on a wide variety of other constructs with varying levels of association rather than a small set of extremely relevant constructs
How is a set of correlations utilised?
First researchers compute the correlations between a test of interest and measures of many criterion variables. They will then ‘eyeball’ the correlations and make a somewhat subjective judgement about the degree to which the correlations match.
How is a mutitrait multi method study carried out?
Researchers obtain measures of several traits, each of which is measured through several methods (the test, self report, friends, psychologist etc)
How does MTMMM study variance?
By both studying trait variance and method variance
What is meant by shared method variance?
When there is a correlation because it is derived from two measures from the same source (bias in self report etc)
How may method variance affect results?
A strong correlation could either mean the two traits have some commonalities (share trait variance) or that they share method variance. On the other hand a weak correlation could indicate very few commonalities or that they share no method variance
Name the four different types of correlations in a MTMMM and their expected strengths
Heterotrait- heteromethod correlations- weakest
Heterotrait- mono method correlations- moderate
Monotrait- heteromethod correlations- moderate
Monotrait- mono method correlations- strongest
What is convergent validity represented by in MTMMM?
Monotrait- heteromethod correlations
What requirement is there for strong evidence for convergent validity?
The mono trait- heteromethod correlations should be higher than heterotrait- mono method measures or heterotrait- heteromethod measures
