Lecture 2 Validity & Reliability (Catherine) Flashcards
To provide an overview of the content of Lecture 2
Distinguish between precision, accuracy, reliability & validity in relation to measuring instruments
An instrument has:
- Precision if it has fineness of discrimination
- Accuracy if it gives the correct value and has no systematic bias
- Reliability if the instrument has measurement stability, with no random substantial fluctuations
- Validity if it measures what it proports to measure
Name the key types of Reliability
- Test-retest Reliability: Correlating pairs of scores on 2 different administrations of the same test
- Internal Consistency Reliability: split-half testing Cronbach (nondichotomous items) Kuder & Richardson (dichotomous items)
- Inter-scorer Reliability: The degree of agreement between scorers
What are the key challenges of test reliability?
- Stability over time?
- Internal consistency
- Test scores are made up of the true score plus error
- There is always variability in test scores as a result of error
What prevents scores being stable over time?
-Stability over time: central problem is the interpretation of individual scores changes when the test is administered multiple times
What does internal consistency mean?
The extent to which a psychological test is homogenous (tests one item) or heterogeneous (tests more than one item)
- DASS tests depression, anxiety, & stress is therefore heterogeneous
What are the types of error that are included in the final test score (Test scores are made up of the true score plus error)?
-Test Construction Item or content sampling
-Test Administration (environment, test-taker & examiner related variables)
-Test Scoring & Interpretation
(Hand scoring or subjective judgements)
What is the main problem with Test Construction?
-Systematic error: an ambiguously asked question could be interpreted differently by 2 people
-Errors in Item or content sampling
-Random error: e.g. Catherine is morning person, Donna is afternoon person if we both have our exam in the morning, Catherine has an advantage
NB: Can use Alternate Forms to identify source of error and internal consistency for fatigue as source of error variance
What are the main problems with Test Administration?
-Inconsistent environmental factors (e.g. air con vs no air con)
-Test-taker (individual differences not taken into account like age)
-Examiner related error (fatigue, boredom, etc)
NB: Can use Test-Retest to identify source of error
What are the main problems with Test Scoring & Interpretation?
- Hand scoring open to error
- Subjective judgements
- Computer aided scoring cannot be used for qualitative data
List the different forms of Reliability Estimates
- Test-Retest Reliability
- Parallel Forms Reliability
- Alternate Forms Reliability
- Internal Consistency Reliability using
- Split-Half Reliability
- Chronbach’s alpha
- Kuder-Richardson
- Inter-Scorer Reliability
Which source of Error Variance does Test-Retest Reliability attempt to account for?
Test-Retest Reliability Testing attempts to account for Errors in Test Administration
What are the important considerations to successfully apply Test-Retest Reliability Testing?
- The Test is taken twice and the results are correlated
- It is important to have an appropriate amount of time between tests (this will vary depending on the type of test - e.g. MSE needs 18 months,)
- Systematic Changes should not affect the scores (e.g. everyone in a cold room)
- Unpredictable changes will affect the correlation (such as?)
- A reliable test will be able to sustain greater levels of fluctuation (what does that mean?)
What are the factors that affect Test-Retest Reliability?
individual differences, experience, practice effects, memory, fatigue, motivation.
Which source of Error Variance does Parallel Forms or Alternative Forms Reliability Tests attempt to account for?
Parallel Forms or Alternative Forms Reliability Tests attempt to account for errors in Test Construction
When would a test administrator implement a Parallel Forms or Alternative Forms Reliability Test?
In a situation where it is not possible to conduct a Test-Retest Reliability test
In What ways is a Parallel Forms or Alternative Forms Reliability Test similar to a test-retest reliability test?
- In both cases the participant completes two tests
- The aim of both is to minimise error variance
What are Parallel Forms Reliability Tests?
Parallel forms of a test exist when for each form of the test the means and variances of observed test scores are equal
What are Alternate Forms Reliability Tests?
Alternate forms are simply different forms of a test that have been constructed to be parallel. They are designed to be equivalent with regard to content and level of difficulty, but do not meet the same stringent criteria as parallel forms (so means & variances have not been made equivalent)
What is the main draw back with Alternate Form Tests of Reliability?
Because the means and variances have not been made equivalent (as they have in parallel form) it leads to highly ambiguous test confounds: as now have two sources of error: Time and Content, whereas with parallel forms just time as a confound.
What methods can be employed to achieve internal consistency reliability?
Split-half reliability testing can be employed to achieve internal consistency reliability
What are the main considerations when implementing Split-half reliability testing?
-Ensure the split is in a meaningful way i.e. not first-last half of test (fatigue effects)
better to do odd-even split
-if its heterogeneous test ensure this is also split in meaningful way
What statistical analysis does a test administrator employ to assess Split-Half reliability of a homogeneous test?
A test administrator can obtain a correlation coefficient of a homogeneous test using a Spearman-Brown Formula
-The Spearman-Brown in effect converts the split-half test into a full test.
It cannot be used for a heterogeneous test!
What statistical analysis does a test administrator employ to assess Split-Half reliability of a heterogeneous test?
A test administrator can test the internal consistency reliability of any heterogeneous split-half test using a Chronbach’s Alpha Formula
-The Chronbach’s Alpha is a generalised reliability coefficient for scoring systems that are graded by each item.
(With the DASS we would need a Chronbach’s alpha for each trait measured)
It cannot be used for either a homogeneous or a heterogeneous test But NOT dichotomous answers (yes/no; true/false)
What statistical analysis does a test administrator employ to assess Split-Half reliability of a dichotomous test?
A test administrator can test the internal consistency reliability of any dichotomous split-half test using a Kuder-Richardson Formula
The Kuder-Richardson essentially provides an estimate of all test-retest or split-half coefficients for yes/no or true/false answers
What are acceptable Reliabilities for Clinical and Research situations?
Acceptable reliabilities for Clinical settings is:
r > 0.85 acceptable
Acceptable reliabilities for Research settings is: r > ~0.7 acceptable
What are the Internal Consistency and Test-Retest Reliabilities of the WAIS and MMPI?
Internal Consistency of WAIS: r = 0.887
Internal Consistency of MMPI: r = 0.84
Test-Retest Reliabilities of the WAIS: r = 0.82
Test-Retest Reliabilities of the MMPI: r = 0.74
NB: WAIS test-retest is just outside acceptable limits
the MMPI is suseptable to change over time as its a personality inventory used on clinical patients whoa re more likely to change over time
What type of reliability would a test administrator be assessing if they utilised a Kuder-Richardson, Chronbach’s Alpha or Spearman-Brown?
The Test Administrator would be assessing a tests Internal Consistency
A Correlation coefficient can be used to check all other types of reliability except Internal Consistency. What are these types of reliability?
- Test-Retest Reliability
- Alternate Form Reliability
- Inter-scorer Relaibility
What factors does a Test Administrator need to bear in mind when measuring reliability?
- Is the test measuring state or trait? (trait is more enduring)
- The range of possible responses (ideally 5-7 responses - 0 - 10 is not ideal as people tend to cluster around the middle)
- Speeded tests - towards end of test test taker may not have had time to attempt a number of items, does not mean they would have been incorrect, did not get time to answer
There are seven methods utilised to improve reliability, what are they?
- Quality of items (need to be clear, concise, homogeneous)
- Ensure consistent testing conditions
- Reduce Test-Retest time intervals
- Longer assessments
- Develop a robust scoring plan
- Test items for reliability & adapt the measure
- Ensure Validity
There are 3 classes of Validity, what are they?
- Internal Validity
- External Validity
- Test Validity
What is Internal Validity interested in?
Relevant to Experimental Validity
Confidence in making causal statements about study outcomes
What is External Validity interested in?
Relevant to Experimental Validity
Confidence you can generalise results to people outside of the study
What is Test Validity interested in?
Relevant to this Unit!!!!
Confidence that what you are measuring truly represents what you think you are measuring
What are the 3 forms of assessing test validity?
- Content Validity
- Criterion-Related Validity
- Construct Validity
There are 3 traditional measures of test validity, name them
- Content Validity
- Criterion-Related Validity
- Construct Validity
There are 3 methods to assess each of these forms of validity, what are they?
- Scrutinise test contents
- Comparing Scores on this test to other tests
- Perform an analysis of how scores on this test relate to scores on other tests and theories
There is another form of validity, Face Validity, what is it?
Face Validity relates to whether the person being assessed believes the test appears to measure what is actually being measured
Which is the most important form of validity?
Construct Validity
What are the implications for low face validity
- indirectly tests some aspect not perceived by the test-taker (e.g. MMPI asks about icecream as part of a personality assessment)
- may result in negative consequences such as poor test taker attitude, or disgruntlement
- Some tests have low face validity and others have high face validity
What does Content Validity assess?
Content Validity Scrutinises the test’s content
What is Content Validity concerned with?
Content validity is concerned with how well does each item on the test measure what it intends to measure?
- Tests should capture all aspects of the target behaviour
- e.g. for HR test items should directly relate to the job role we are hiring for
How do we measure Content Validity?
-We use a Content Validity Ratio (CVR - Lawshe, 1975)
-We ask N number of experts to rate each item to reflect performance on an item in terms of essential, useful, not necessary
-We remove items based on % of people who state the item is not necessary
If less than 5% say necessary/essential it should be removed
What does Criterion-Related Validity assess?
Criterion-Related Validity relates scores obtained on the current test to other test scores or other measures
What is Criterion-Related Validity & what are the 2 varieties of Criterion-Related Validity?
Criterion-Related Validity is a judgement of how adequately a test score can be used to infer an individual’s most probable standing on some measure of interest.
The two varieties of Criterion-Related Validity are:
*Concurrent Validity
*Predictive Validity
What is Criterion-Related Validity concerned with?
Criterion-Related Validity is interested in how well the test item reflect an individual’s actual score on the criterion of interest
What are the types of Criterion-Related Validity?
Concurrent Criterion-Related Validity
-The degree to which the score relates to the criterion measure at that time (measure a new test against a gold standard)
Predictive Criterion-Related Validity
-Degree to which the score relates to a criterion measure in the future (i.e. uses regression to predict a person’s future reading ability)
What is Concurent Criterion-related Validity?
Concurrent Validity is an index of the degree to which a test score is related to some criterion masuree obtained at the same time
What are the important considerations when assessing Criterion-Related Validity?
Is the criterion: -Relevant -Valid & Reliable -Uncontaminated We need to ensure the test we are comparing is relevant, valid, reliable & uncontaminated, hence we use gold standard tests rather than just random other tests
How does one measure Concurrent Validity?
By performing a correlation and comparing how well the outcome of the new test compares with the outcome of a well-known, reliable, well-validated test
What is Predictive Criterion-Related Validity?
Predictive Validity is an index of the degree to which a test score predicts some criterion measure (in the future)
How does one measure Predictive Validity?
By obtaining test scores now, and criterion measure in the future using multiple predictors.
-considering the validity coefficient in the context of corresponding issues including
-incremental validity and expectancy data
NEEDS work - not clear
What is the validity coefficient?
The Validity coefficient is a correlation that provides a measure of the relationship between test scores and scores on the criterion measure.
What is incremental validity?
Uses more than one predictor
- Additional predictors used in ascertaining criterion-related predictive validity should possess incremental validity.
- That is the degree to which an additional predictor explains something about the criterion measure that is not explained already by predictors already in use.
What is Expectancy Data?
Expectancy Data provides useful information to evaluate the criterion-related validity of a test.
Using a score obtained from one test or measure expectancy tables illustrate the likelihood that the test-taker will score within some interval of scores (such as pass or fail).
How does one create an Expectancy Table?
An expectancy table can be created by a scatterplot according to
An expectancy table shows the relationship between scores on e.g. high school exams and relationship to university grade
What is Construct Validity concerned with?
Construct Validity is concerned with how well inferences drawn from a test score relate to current theories or knowledge (i.e. constructs)
What are the 6 sources of evidence for Construct Validity?
- Evidence of Homogeneity
- Evidence of Changes with Age
- Evidence of Distinct Groups
- Convergent Evidence**
- Discriminant Evidence**
- Factor Analysis
What issues does one need to be aware of when considering Evidence of Homogeneity, one of the 6 sources of evidence for Construct Validity?
- How uniform the test is for measuring a single concept
- Correlate sub-sections with the whole test score
- Item analysis
- How important is homogeneity?
What does Construct Validity Measure?
Construct Validity executes a comprehensive analysis of how scores on the test:
a. relate to other scores and measures
b. can be understood within some theoretical framework for understanding the construct that the test was designed to measure
What are the 13 factors which can negatively affect validity?
-Unclear directions
-Ambiguity in question terminology
-Inadequate time limits
-Inappropriate level of difficulty
-Poorly constructed test items
Test items are inappropriate for planned test outcomes
-Tests that are too short
-Improper arrangement of items
-Identifiable patterns of answers
-Administration and Scoring
-Nature of the Criterion
-Bias
-Fairness
Bias is one of the 13 factors which can negatively affect validity, what specific bias/biases are important?
- Test biased towards a certain population
- Implies a systematic variation in results
- Slope versus Intercept Bias
- Rating error - overcome by ranking
- Halo effect
What is a Criterion, and what essential properties does a criterion require?
A Criterion is the standard against which a test or a test score is evaluated.
*The Criterion needs to be relevant, valid and uncontaminated.
Define the characteristics of a criterion
- An adequate criterion is relevant, i.e. it is pertinent or applicable to the matter in hand.
- Evidence should exist that supports the validity of the criterion
- A criterion should be uncontaminated, so it should be based, at least in part, on predictor measures. If not, validation study cannot be taken seriously. There is no formal test for criterion contamination
Fairness is one of the 13 factors which can negatively affect validity, what specific aspects of fairness are important?
- Age, Culture, Gender
- Adjustment to scores - is this fair?
- Psychometric techniques for reducing adverse impact of unfairness to some groups
What method for quantifying content validity was put forward by C.H. Lawshe in 1975?
The method developed by Lawshe gauged agreement among raters/judges regarding how essential a particular item is.
i. e. is the skill or knowledge measured by this item:
- essential
- useful (but not essential)
- unnecessary
What Provides a measure of Concurrent Validity?
If test scores are obtained about the same time that the criterion measures are obtained, measures of the relationship between the test scores & the criterion provide evidence of concurrent validity.
What provides an indication of predictive validity?
Measures of the relationship between test scores & a criterion measure obtained at some future time provide an indication of the predictive validity of the test, that is, how accurately scores on the test predict some criterion measure
What statistical evidence is used to make judgements of criterion-related validity (either concurrent or predictive)?
Two types of statistical evidence is used:
- the validity coefficient and
- expectancy data
What does the validity coefficient provide?
The Validity coefficient is a correlation coefficient that provides a measure of the relationship between test scores and scores on the criterion measure.
*The Pearson Correlation coefficient is typically used to determine the validity between the 2 measures.
What variables negative influence the validity coefficient?
restriction or inflation of range, a key issue being whether the range of scores employed is appropriate to the objective of the correlational analysis
How high shoudl a validity coefficient be for a user or a test developer to infer that the test is valid?
Cronbach & Gleser cautioned against the establishment of such a rule, it simply should be high enough to result in the identification & differentiation of test-takers with regard to target ability.
What is incremental validity?
Incremental validity is the degree to which an additional predictor explains something about the criterion measure that is not explained by predictors already in use.
Each measure used as a predictor should have criterion-related predictive validity, possess incremental validity & only be included if they demonstrate something not covered by existing predictors
What type of information is provided by expectancy data and expectancy tables?
Expectancy data provides information that can be used in evaluating the criterion-related validity of a test.
*Expectancy tables illustrate the likelihood that the test-taker will score within some interval of scores on a criterion measure (e.g. pass/fail)
Name two renowned expectancy table developers
- Taylor-Russell (1939, 1973, 1974) - seven steps to an expectancy table
- Naylor-Shine Tables (1965)
List the strengths and weaknesses of the Taylor-Russell expectancy tables
- 7 step procedure was provided
- The table can assist in judging the utility of a test by determining the increase over current procedures
Limitation:
- The relationship between predictor and criterion must be linear
- It is difficult to identify the cut off for successful vs unsuccessful using the table
What are the strengths and limitations of the Naylor-Shine tables?
- No need for linear relationship as uses average criterion scores to compare
- Obtaining the difference between the means of the selected & unselected groups to derive an index of what the test is adding to already established procedures
- Identifies the utility of a test by determining the increase in average score on some criterion measure
What do the Taylor-Russell & Naylor-Shine tables have in common?
With both tables the validity coefficient used must be one obtained by Concurrent Validation procedures
What is the most often-cited application of statistical decision theory in the field of psycholgical testing, &, what are it’s 4 key points?
Cronbach & Gleser’s Psychological Tests & Personnel Decision (1957, 1965)
- a classification of decision problems
- various selection strategies ranging from single-stage to sequential analyses
- a quantitative analysis of the relationship between test utility, the selection ratio, costing of testing, expected value of outcomes &
- a recommendation that in some instances job requirements be tailored to the applicants abilities instead of the other way around
Define Construct Validity
Construct Validity is a judgment about the appropriateness of inferences drawn from test scores regarding individual standings on a variable called a construct.
*Constructs are observable, presupposed or underlying traits that a test developer may invoke to describe test behaviour or criterion performance
If a test is a valid measure of the construct (i.e. it has high construct validity) what results will the test developer observe?
High scorers and low scorers will behave as predicted by the theory
NB Construct validity has been viewed as the unifying concept for all validity evidence
What are some reasons results might behave contrary to those predicted?
- The test simply does not measure the construct
* The theory is sound, but the statistical procedures or their execution was flawed
List the 5 procedures which provide evidence for construct validity
- The test is homogeneous, measuring a single construct
- Test scores increase or decrease as a function of age, the passage of time, or an experimental manipulation as theoretically predicted
- Test scores obtained after some event or time (i.e. post-test scores) differ from pre-test scores as theoretically predicted
- Test scores obtained by people from distinct groups vary as predicted by the theory
- Test scores correlate with scores on other tests in accordance with what would be predicted from a theory that covers the manifestation of the construct in question
What is covergent evidence as it relates to construct validity?
Evidence for the construct validity of a particular test may converge from a number of sources, e.g. other tests designed to assess the same or similar construct. Thus, if scores on the test undergoing construct validation tend to correlate highly in the predicted direction with scores on older, more established, already validated tests, this is convergent evidence
What is Discriminant Evidence in relation to construct validity?
A validation coefficient showing little (i.e. a statistically insignificant) relationship between test scores &/or other variables with which scores on the test being construct-validated should NOT theoretically be correlated this provides Discriminant Evidence or discriminant validity
What Statistical method is employed to evidence convergent or discriminant construct validity?
Factor Analysis
What are the key points of Lawshe’s rating process, which he termed Content Validity Ratio (CVR)?
- If more than half the panelists indicate the item is essential, it has at least some content validity.
- Lawshe recommended that if the amount of agreement observed is more than 5% likely to occur by chance, then the item should be eliminated
1. Negative CVR: fewer than half the panelists indicate essential
2. zero CVR: exactly half the panelists indicate essential
3. Positive CVR: More than half, but not all the panelists indicate essential
What are unavoidable issues for all test developers?
That errors in measurement exist in all test.
- these errors affect both reliability and validity
- the test developers goal is to reduce / minimise error
what is intercept bias?
If a test systematically under predicts or over predicts the performance of a particular group with respect to a criterion, then it exhibits intercept bias.
Intercept bias is a term derived from the point where the regression line intersects the Y-Axis
What is slope bias?
If a test systematically yields significantly different validity coefficients for members of different groups, then it has a slope bias
Slope bias is named as the slope of one group’s regression line is different in a statistically significant way from the slope of another group’s regression line.
What is a rating error and what are the types of rating errors?
A rating error is a judgment resulting from the intentional or unintentional misuse of a rating scale.
- Leniency or generosity error (too generous)
- Severity error (too harsh)
- Central Tendency error (sticks to the middle)
- Halo Effect (high ratings in all things due to raters failure to discriminate)
What is the overall goal of test development?
To obtain consistent results that truly reflect the concepts we are trying to obtain.
Define fairness, as it applies to psychometric testing
Fairness, in a psychometric context is the extent to which a test is used in an impartial, just, and equitable way.
Name another variable that can influence all aspects of test construction, including test validitation?
The influence of Culture extends to judgements concerning validity of tests and test items
