Reliability and Validity Flashcards
Define
Alternative
Two forms of the same test developed; different items selected according to the same rules. Different distribution of scores (mean and variance may not be equal)
Define
Base rate
the proportion of individual in the population who show the behaviour of interest in a given psychological testing or assessment situation
Define
Classical test theory
a body of related psychometric theory that predicts outcomes of psychological testing such as the difficulty of items or the ability of test-takers
Define
Concurrent validity
a form of predictive validity in which the index of social behaviour is obtained close in time to score on the psychological test (or other assessment device)
Define
Construct underrepresentation
failure to capture important components of a contruct
Define
Construct validity
the meaning of a test score made possible by knowledge of the pattern of relationships it enters into with other variables and the theoretical interpretation of those relationships
Define
Constuct-irrelevant variance
measuring things other than the construct of interest
Define
Content validity
the extent to which items on a test represent the universe of behaviour the test was designed to measure
Define
Convergent and discriminant validity
the subjection of a multitrait-multimethod matric to a set of criteria that specify which correlations should be large and which small in terms of a psychological theory of the constructs
Define
Criterion-related validity
the extent to which a measure is related to an outcome (e.g. marks in Year 12 are used to predict performance at university)
Define
Cronbach’s alpha
an estimate of reliability that is based on the average intercorrelation of the items in a test
Define
Cutting point
the test score or point on a scale, in the case of another assessment device, that is used to split those being tested or assessed into two groups predicted to show or not show some behaviour of interest
Define
Domain-sampling model
a way of thinking about the composition of a psychological test that sees the test as a representative sample of the larger domain of possible items that could be included in the test
Define
Equivalent forms reliability
the estimate of reliability of a test obtained by comparing two forms of a test constructed to measure the same construct
Define
Errors of measurement
Factors that contribute to inconsistency - characteristics of tests taker, test or situation that have nothing to do with attribute being tested by effect scores
Define
Face validity
Does the test look like it measures the relevant construct?
Define
Factor analysis
a mathematical method of summarising a matric of values (such as the intercorrelation of test scores) in terms of a smaller number of values (factors) from which the original matric can be reproduced
Define
False negative decision
a decision that incorrectly allocates a test taker or person being assessed to the category of those predicted not to show some behaviour of interest on the basis of their score on a test or other assessment device
Define
False positive decision
a decision that incorrectly allocates a test taker or person being assessed to the category of those predicted to show some behaviour of interest on the basis of their score on a test or other assessment device
Define
Generalisability theory
a set of ideas and procedures that follow from the proposal that the consistency or precision of the output of a psychological assessment device depends on specifying the desired range of conditions over which this is to hold
Define
Incremental validity
the extent to which knowledge of score on a test (or other assessment device) adds to that obtained by another, pre-existing score or psychological characteristic
Define
Inter-rater reliability
the extent to which different raters agree in their assessments of the same sample of ratees
Define
Internal consistency
the extent to which a psychological test is homogenous or heterogeneous
Define
Kuder-Richardson 20 (KR20)
a particular case of Cronbach’s alpha for dichotomously scored items (i.e. scored as 0 or 1)
Define
Method variance
the variability among scores on a psychological test or other assessment device that arises because of the form as distinct from the content of the test
Define
Multitrait-multimethod matrix
the patterns of correlations resulting from testing all possible relationships among two or more methods of assessing two ro more constructs
Define
Parallel forms relaibility
Two forms of the same test developed; different items selected according to the same rules. Same distribution of scores (mean and variance equal)
Define
Predictive validity
the extent to which a score on a psychological test (or other assessment device) allows a statement about standing on a variable indexing important social behaviour independent of the test
Define
Reliability
the consistency with which a test measures what it purports to measure in any given set of circumstances
Define
Reliability coefficient
an index - often a Pearson product moment correlation coefficient - of the ratio of true score to error score variance in a test as used in a given set of circumstances
Define
Selection ratio
the proportion of those tested or assessed who can be allocated to the category of showing the behaviour of interest in a given psychological testing or assessment situation
Define
Social desirability bias
a form of method variance common in the construction of psychological tests of personality that arises when people respond to questions that place them in a favourable or unfavourable light
Define
Spearman-Brown formula
applied to estimate reliability if each half of the test was the same length as the test. I.e. allows you to estimate internal consistency if the test was longer or shorter
Define
Split-half reliability
the estimate of reliability obtained by correlating scores on the two halves of a test formed in some systematic way (e.g. odd versus even items)
Define
Stability over time
the extent to which test scores remain stable when a test is administered on more than one occasion
Define
Standard error of estimate
an index of the amount of error in predicting one variable from another
Define
Standard error of measurement
an expression of the precision of an individual test score as an estimate of the trait it purports to measure
Define
Test-Retest Reliability
the estimate of reliability obtained by correlating scores on the test constructor is seeking to measure and the conditions under which it will be used
Define
True scores
Factors that contribute to consistency - stable attributes under examination
Define
Valid negative decision
a decision that correctly allocates a test taker or person being assessed to the category of those predicted not to show some behaviour of interest on the basis of their score on a test or other assessment device
Define
Valid positive decision
a decision that correctly allocates a test taker or person being assessed to the category of those predicted to show some behaviour of interest on the basis of their score on a test or other assessment device
Define
Validity
the extent to which evidence supports the meaning and use of a psychological test (or other assessment device)
Definition
Two forms of the same test developed; different items selected according to the same rules. Different distribution of scores (mean and variance may not be equal)
Alternative
Definition
the proportion of individual in the population who show the behaviour of interest in a given psychological testing or assessment situation
Base rate
Definition
a body of related psychometric theory that predicts outcomes of psychological testing such as the difficulty of items or the ability of test-takers
Classical test theory
Definition
a form of predictive validity in which the index of social behaviour is obtained close in time to score on the psychological test (or other assessment device)
Concurrent validity
Definition
failure to capture important components of a contruct
Construct underrepresentation
Definition
the meaning of a test score made possible by knowledge of the pattern of relationships it enters into with other variables and the theoretical interpretation of those relationships
Construct validity
Definition
measuring things other than the construct of interest
Constuct-irrelevant variance
Definition
the extent to which items on a test represent the universe of behaviour the test was designed to measure
Content validity
Definition
the subjection of a multitrait-multimethod matric to a set of criteria that specify which correlations should be large and which small in terms of a psychological theory of the constructs
Convergent and discriminant validity
Definition
the extent to which a measure is related to an outcome (e.g. marks in Year 12 are used to predict performance at university)
Criterion-related validity
Definition
an estimate of reliability that is based on the average intercorrelation of the items in a test
Cronbach’s alpha
Definition
the test score or point on a scale, in the case of another assessment device, that is used to split those being tested or assessed into two groups predicted to show or not show some behaviour of interest
Cutting point
Definition
a way of thinking about the composition of a psychological test that sees the test as a representative sample of the larger domain of possible items that could be included in the test
Domain-sampling model
Definition
the estimate of reliability of a test obtained by comparing two forms of a test constructed to measure the same construct
Equivalent forms reliability
Definition
Factors that contribute to inconsistency - characteristics of tests taker, test or situation that have nothing to do with attribute being tested by effect scores
Errors of measurement
Definition
Does the test look like it measures the relevant construct?
Face validity
Definition
a mathematical method of summarising a matric of values (such as the intercorrelation of test scores) in terms of a smaller number of values (factors) from which the original matric can be reproduced
Factor analysis
Definition
a decision that incorrectly allocates a test taker or person being assessed to the category of those predicted not to show some behaviour of interest on the basis of their score on a test or other assessment device
False negative decision
Definition
a decision that incorrectly allocates a test taker or person being assessed to the category of those predicted to show some behaviour of interest on the basis of their score on a test or other assessment device
False positive decision
Definition
a set of ideas and procedures that follow from the proposal that the consistency or precision of the output of a psychological assessment device depends on specifying the desired range of conditions over which this is to hold
Generalisability theory
Definition
the extent to which knowledge of score on a test (or other assessment device) adds to that obtained by another, pre-existing score or psychological characteristic
Incremental validity
Definition
the extent to which different raters agree in their assessments of the same sample of ratees
Inter-rater reliability
Definition
the extent to which a psychological test is homogenous or heterogeneous
Internal consistency
Definition
a particular case of Cronbach’s alpha for dichotomously scored items (i.e. scored as 0 or 1)
Kuder-Richardson 20 (KR20)
Definition
the variability among scores on a psychological test or other assessment device that arises because of the form as distinct from the content of the test
Method variance
Definition
the patterns of correlations resulting from testing all possible relationships among two or more methods of assessing two ro more constructs
Multitrait-multimethod matrix
Definition
Two forms of the same test developed; different items selected according to the same rules. Same distribution of scores (mean and variance equal)
Parallel forms relaibility
Definition
the extent to which a score on a psychological test (or other assessment device) allows a statement about standing on a variable indexing important social behaviour independent of the test
Predictive validity
Definition
the consistency with which a test measures what it purports to measure in any given set of circumstances
Reliability
Definition
an index - often a Pearson product moment correlation coefficient - of the ratio of true score to error score variance in a test as used in a given set of circumstances
Reliability coefficient
Definition
the proportion of those tested or assessed who can be allocated to the category of showing the behaviour of interest in a given psychological testing or assessment situation
Selection ratio
Definition
a form of method variance common in the construction of psychological tests of personality that arises when people respond to questions that place them in a favourable or unfavourable light
Social desirability bias
Definition
applied to estimate reliability if each half of the test was the same length as the test. I.e. allows you to estimate internal consistency if the test was longer or shorter
Spearman-Brown formula
Definition
the estimate of reliability obtained by correlating scores on the two halves of a test formed in some systematic way (e.g. odd versus even items)
Split-half reliability
Definition
the extent to which test scores remain stable when a test is administered on more than one occasion
Stability over time
Definition
an index of the amount of error in predicting one variable from another
Standard error of estimate
Definition
an expression of the precision of an individual test score as an estimate of the trait it purports to measure
Standard error of measurement
Definition
the estimate of reliability obtained by correlating scores on the test constructor is seeking to measure and the conditions under which it will be used
Test-Retest Reliability
Definition
Factors that contribute to consistency - stable attributes under examination
True scores
Definition
a decision that correctly allocates a test taker or person being assessed to the category of those predicted not to show some behaviour of interest on the basis of their score on a test or other assessment device
Valid negative decision
Definition
a decision that correctly allocates a test taker or person being assessed to the category of those predicted to show some behaviour of interest on the basis of their score on a test or other assessment device
Valid positive decision
Definition
the extent to which evidence supports the meaning and use of a psychological test (or other assessment device)
Validity
What is reliability?
- The degree to which a test tool provides consistent results
- A test is considered reliable when it produces the same results again and again, when measuring the same thing
What is validity?
Validity can be broadly understood as the extent to which a test measures the constuct it is intended to measure
John is feeling unwell and visits his GP. The GP orders some blood tests. The results of the blood test indicate that John is iron deficient. The doctor prescribes iron supplements which John immediately start taking as prescribed. After a few weeks he returns to the doctor to repeat the blood tests. The results indicate that the iron levels have reduced!
What might be happening?
- The test may have poor validity (i.e. it is measuring some other variable)
- The test has poor reliability (i.e. when repeated, the test often shows different results)
Why is reliability and validity important?
- Diagnosis
- Assessment of ability
- Treatment
- Decisions around recommending treatment
- Monitoring treatment outcomes (e.g. reliability would be really important if you are repeating tests to see if the treatment is working)
- The conclusions you can draw rely on the reliability and validity of the tests/assessments you are using.
- Important clinically and in research
True or False:
Test can be reliable without being valid
True
True or False:
Test can be valid without being reliable
False
Tests cannot be valid without being reliable
According to Classical Test Theory, what are test scores the result of?
- Factors that contribute to consistency – stable attributes under examination (“True Scores”)
- Factors that contribute to inconsistency – characteristics of tests taker, test or situation that have nothing to do with attribute being tested but effect scores. (“Errors of Measurement”)
What are some commone sources of error on a test?
Item selection
Test administration
Test scoring
Systematic measurement error
How is item selection a potential source of error?
sample of items chosen may not be equally reflective of every individual’s true score.
How is test administration a potential source of error?
General environmental conditions e.g. temperature, lighting, noise; temporary “states” of test taker e.g. fatigue, anxiety, distraction.
E.g. completing an IQ test in a loud noisy room. Examiner providing non-standerdised instructions
Domain Sampling Theory considers the problem of using only a ________ of items to represent a construct
Domain Sampling Theory considers the problem of using only a sample of items to represent a construct
If the same test is administered to the same group twice at two different times, why might the scores not be the same?
More test-retest reliability
Practice effects
Maturation
Treatment effects or setting
Which of these would test-retest be approapriate for?
- State anxiety
- Weight of a baby
- Extraversion
- Intelligence
Which of these would test-retest be approapriate for?
- State anxiety
- Weight of a baby
- Extraversion
- Intelligence
How do you maximise test-retest reliability?
- Test a relatively stable construct
- No intervention in between testing
- Shorter time between testing
What is the difference between parallel and alternate forms reliability?
The both involve two forms of the same test developed; different items selected according to the same rules.
Parallel Forms: same distribution of scores (means and variance equal)
Alternate Forms: different distribution of scores (mean and variance may not be equal)
What is the split-half method?
- Test is split into halves (randomly, odd-even system, top vs bottom)
- Correlate the two halves
- Estimate of reliability based on split half is smaller due to smaller number of items
- Spearman-Brown formula is applied to estimate reliability if each half of the test was the same length as the test.
- i.e. Allows you to estimate internal consistency if the test if it was longer or shorter
What is the rationale for the split-half method?
if scores on 2 half tests from single administration are highly correlated, scores on 2 whole tests from separate administrations should also be highly correlated.
What is Cronbach’s alpha?
A generalised reliability coefficient for scoring systems that are graded for each item (i.e. agree to disagree)
- Mean of all possible split-half correlations, corrected by the Spearman-Brown formula
- Ranges from 0 (no similarity) to 1 (perfectly identical)
What are considered acceptable levels of reliability?
Depends on the purpose to some extent
- .70-.80 acceptable or good
- Greater than .90 may indicate redundancy in items
- High reliability is really important in clinical settings when making decisions for a person (e.g. decision making capacity assessment).
__________________: a particular case of Cronbach’s alpha for dichotomously scored items (i.e. scored as 0 or 1)
Kuder-Richardson 20 (KR20): a particular case of Cronbach’s alpha for dichotomously scored items (i.e. scored as 0 or 1)
The __________ the SEM, the less certain we are that the test score represents the true score.
The larger the SEM, the less certain we are that the test score represents the true score.
How do you maximise reliability?
- Clear administration and scoring instructions for test user
- Clear instructions for the test taker
- Unambiguous test items
- Standardised testing environment and procedure
- Reduced time between testing sessions
- Increase assessment length/items
- Test try-out and modification
- Discarding items that decrease reliability (item analysis)
- Maximise VALIDITY
Draw a diagram that demonstrate the different types of validity
What are the four main types of validity?
Face validity
Content validity
Criterion-related validity
Construct validity
What are some issues for content validity?
- Construct underrepresentation: failure to capture important components of a construct.
- e.g. A depression scale that assesses cognitive and emotional components of depression, but not behavioural components.
- Construct-irrelevant variance: measuring things other than the construct of interest.
- e.g. The wording of our depression scale may make it likely that people will respond in socially desirable ways.
What are some examples of criterion-related validity?
- e.g. marks in Year 12 are used to predict performance at university
- e.g. a marital satisfaction survey is used to predict divorce
- e.g. scores on an anxiety scale you developed are correlated with clinical observations.
What is an example of concurrent validity?
A test designed to measure anxiety may be issued in conjunction with a diagnostic interview by an experienced clinician using the DSM-5. The concurrent validity of the test represents the extent to which the test score corresponds with the clinician’s observations of the client’s anxiety levels.
What is an example of predictive evidence?
VCE marks or ATAR scores are used to predict performance at university
What is an example of convergent evidence?
e. g. Relationship between score on measure of psychopathy and low emotional arousal.
e. g. Relationship between low self-esteem and depression
What are some examples of discriminant evidence?
Scores on an anxiety measure should differ from scores on a depression measure, if each measure is assessing these individual constructs.
