Psychometrics in Neuropsychological Assessment (Strauss, 2006) Flashcards
1
Q
How do you calculate a z-score?
A
obtained score - sample mean/ sample SD
2
Q
What is the mean and SD of a Z-score?
A
Mean = 0 SD = 1
3
Q
What is a Z-score?
A
A type of standard score
4
Q
What does a Z-score quantify?
A
How many SDs a score is from the mean
5
Q
What is a t-score?
A
Another linear transformation of a raw score
6
Q
What is the mean and SD of a t-score?
A
Mean = 50 SD = 10
7
Q
Why do we use standard scores?
A
By virtue of conversion to a common metric, they facilitate the comparison of scores across measures
8
Q
What must the distribution of the tests be for us to use standardized scores?
A
Approximately normal
9
Q
What are two things that must be considered before comparing test scores?
A
- The reliability of the 2 measures
2. Their intercorrelation
10
Q
How can you calculate the prevalence value from a z-score?
A
- Look up the corresponding estimated frequency (e.g., -4) in a z-score table
- Divide 1 by that value (e.g., .00003/1 = 31,560)
Thus, the estimated prevalence of -4 is 1 in 31,560
11
Q
A test with a normal distribution in the general population may show extreme skew or other divergence from normality when administered to a population that differs considerably from the average individual. Give an example.
A
Vocab test being negatively skewed when administered to doctoral students in literature vs positively skewed when given to preschoolars who recently immigrated
12
Q
When a new test is constructed, how can non-normality be corrected?
A
By examining the distribution of scores on the prototype test, adjusting test properties and resampling until a normal distribution is reached
13
Q
What is another way of saying negatively skewed (in terms of testing)?
A
low ceiling
14
Q
What is another way of saying positively skewed (in terms of testing)?
A
high ceiling
15
Q
Will a large N correct for non-normality of an underlying population distribution?
A
No - a larger sample will only produce a more normal distribution if the underlying population distribution from which the sample was obtained is normal
16
Q
What factors may lead to non-normal test score distributions?
A
- The existence of discrete subpopulations within the general population with differing abilities
- Ceiling or floor effects
- Treatment effects that change the location of means, medians and modes and affect variability and distribution shape
17
Q
Small samples may yield non-normal distributions due to what?
A
Random sampling effects
18
Q
What is a formal measure of asymmetry?
A
Skewness
19
Q
What is the skew value of a true normal distribution?
A
0
20
Q
What will have a skew value NEAR 0?
A
A non-normal but symmetric distribution
21
Q
What do negative skew values indicate?
A
That the left tail of the distribution is heavier than the right
22
Q
What does skewness tell us about the mean and median?
A
If there is skewness then the mean and median are not identical because the mean will not be at midpoint in rank
Z-scores will not accurately translate into sample percentile rank values
23
Q
What increases as skew increases?
A
Error in mapping z-scores to sample percentile ranks
24
Q
What kind of distributions often have significant skew? Give an example.
A
Truncated distributions often have significant skew
Truncated distributions often occur when range is restricted: e.g., reaction time
25
Floor and ceiling effects may be defined as what?
The presence of truncated tails in the context of limitations in range of item difficulty
26
What does a high floor mean?
All items difficult
27
What does a low ceiling mean?
All items easy
28
What does multimodality refer to?
The presence of more than one peak in a frequency distribution
29
Non-normality has major implications for interpreting and comparing standard scores - elaborate.
Standardized scores derived by linear transformation will not correspond to sample percentiles and the degree of divergence can be quite high
30
Normalizing transformations introduce error - when is it acceptable to normalize scores?
1) If they come from a large and representative sample
| 2) If any deviation from normality arises from defects in the test rather than characteristics of the sample
31
Normalizing transformations introduce error, what is the preferable way to handle non-normality in scores?
Preferable to adjust score distributions prior to normalization by modifying test content rather than statistically transforming non-normal scores into a normal distribution
32
Define reliability
The consistency of measurement of a given test and can be defined in several ways, including consistency within itself (internal consistency reliability) and consistency over time (test-retest reliability)
33
What do indices of reliability show?
The degree to which a test is free from measurement error
34
Internal consistency reliability
The tests reliability with itself
35
Reliability coefficients are influenced by what two things?
1. Test characteristics
| Sample characteristics
36
Reliability coefficients are influenced by test characteristics - list some examples of test characteristics
Length, item type, item homogeneity, influence of guessing
37
Reliability coefficients are influenced by sample characteristics - list some examples of sample characteristics
Sample size, range, variability
38
Test clarity is closely related to reliability. What is meant by test clarity?
1. Clearly written
2. Easily understood instructions
3. Standardized administration conditions
4. Exploring scoring rules that minimize subjectivity
A process for training raters
39
What is internal reliability?
The extent to which items within a test measure the same cognitive domain or construct
A measure of the inter-correlation of items, internal reliability is an estimate of the correlation between randomly parallel test forms, and by extension, correlation between test scores and true scores.
40
Describe and provide examples of the major sources of measurement error
Time sampling error- is associated with the fluctuation in test scores obtained from repeated testing of the same individual.
Content-Sampling Error- is the term used to label the error that results from selecting test items that inadequately cover the content area that the test is supposed to evaluate.
Also: Quality of test items, test length, test-taker variables, and lastly the test administration.
41
Do longer or shorter tests yield higher reliability estimates
Longer
42
What is split half reliability
Correlating two halves of items from the same test - a type of internal reliability
43
What is another term for split half reliability?
Spearman-brown reliability coefficient
44
What is the error variance associated with split-half reliability?
Content sampling
45
What is the Kuder-Richardson reliability coefficient?
A type of internal reliability - used for yes/no items or heterogeneous tests where split half is necessary
46
What is the error variance associated with Kuder-Richardson?
A type of internal reliability - used for yes/no items or heterogeneous tests where split half is necessary
47
What is the coefficient alpha?
General estimate of reliability based on all the possible ways of splitting up test items - an internal reliability coefficient
48
What are the 3 internal reliability coefficients?
1. Split-half or Spearman-brown
2. Coefficient alpha
Kuder-Richardson
49
What is the type of error variance associated with test-retest reliability?
Time sampling
50
What are the advantages, in terms of reliability, of speed tests?
1. Very high internal reliability estimates
| Reliabilities can also be divided into specific time intervals
51
What is another term for test-retest reliability?
Temporal stability
52
What is test-retest reliability?
An estimate of the correlation between two test scores from the same test administered at 2 different times
53
What are you looking for in test-retest reliability?
Little change over time - this shows that there are no differential effects of prior exposure
54
What kind of tests will show lower test-retest reliability? What kind will show higher?
Tests measuring dynamic (changeable) abilities will show lower test-retest reliability than tests measuring more trait-like or stable abilities
55
What kind of situational variables can impact an individual's test score over time on the same test?
State, examiner state, examiner identity, environmental conditions
56
What are 3 sources of bias in test-retest situations?
1. Intervening variables (e.g., surgery)
2. Practice effects
3. Demographic considerations (Age, education etc.)
57
What are two types of statistical error associated with test-retest reliability?
Measurement error and regression to the mean
58
What is the type of coefficient associated with test-retest reliability?
stability coefficient
59
What is the purpose alternate forms reliability?
It attempts to eliminate confounding effects of practice when a test has to be administered more than once
60
Types of error associated with alternate forms reliability
Time sampling error and content sampling error
61
Why do alternate forms not necessarily eliminate effects of prior exposure?
Because exposure to stimuli and procedures can introduce carry over effects too
(these effects are less for tests assessing acquired knowledge)
62
What is inter-rater reliability?
Reliability of administration and scoring
63
How do you evaluate inter-rater reliability?
1. Percentage agreement
2. Kappa
3. Product-moment correlation
64
What is Generalizability/G-theory?
Reliability is evaluated by decomposing test score variance into between- and within-group variance using the general linear model
65
What is between groups variance considered to be an estimate of?
True score variance
66
What is within group variance considered to be an estimate of?
Error variance
67
What is the coefficient associated with G-theory?
Generalizability coefficient - the ratio of estimated true variance to the sum of the estimates of true variance and estimated error variance
68
Does high reliability indicate high validity?
No
69
Will reliability stay the same across populations?
No - it may vary
70
Can a test have a single level of reliability?
No - it is composed of several different kinds - the importance of each type depends on the test
71
What is more important - reliability or validity?
Both important, but usually preferable to choose a test of slightly lesser reliability if it can be demonstrated that the test is associated with a meaningfully higher level of validity
72
What does high internal consistency indicate?
That the test is measuring a single construct
73
What kind of reliability is considered essential?
High test-retest reliability is considered essential unless the test is measuring state variables that are expected to fluctuate
74
How do you interpret a reliability coefficient?
A reliability coefficient can be interpreted directly in terms of the percentage of score variance attributed to different scores
75
How do you estimate true score variance?
By knowing all the sources of variance (test-retest, alternate form, inter-rater) then you can estimate true score variance
76
What are the 5 categories of reliability coefficient magnitudes?
1. Very high (.90+)
2. High (.80-.89)
3. Adequate (.70-.79)
4. Marginal (.60-.69)
5. Low (
77
What is a low magnitude reliability coefficient?
78
What is a marginal reliability coefficient?
.60-.69
79
What is an adequate reliability coefficient?
.70-.79
80
What is a high reliability coefficient?
.80-.89
81
What is a very high reliability coefficient?
.90+
82
What is the important limitation of reliability coefficients?
They do not provide complete information on the reproducibility of individual test scores
83
What is a true score?
The score an examinee would obtain on a measure in the absence of any measurement error - these are never known, but rather estimated
Conceptually defined as the mean score an examinee would obtain across an infinite number of randomly parallel forms of a test
84
Obtained scores
The actual score yielded by the test
Sum of true scores and error
85
Estimated true scores
The sum of the mean score of the group and the deviation of the person's score from the normative mean weighted by test reliability
86
As test reliability approaches unity (r=1.0) estimated true scores approach X
Obtained scores
87
As test reliability is close to 0, estimated true scores approach X
the mean test scores
88
In the SEM formula what is the kind of reliability it is looking for?
reliability coefficient of the test (usually internal reliability)
89
What is validity?
the degree to which a test actually measures what it is intended to measure
90
What are the three main types of validity?
1. Content validity
2. Criterion validity
3. Construct validity
91
What are two sub-types of content validity?
1. Divergent
| 2. Convergent
92
What are two sub-types of criterion validity?
1. Concurrent
| 2. Predictive
93
How do you evaluate construct validity?
Correlation with other tests, factor analysis, internal consistency, convergent and discriminant validation, structural equation modelling
94
What is test sensitivity?
The proportion of COI+ examinees who are correctly identified as such by a test
(high sensitivity tests yields high levels of correct positives)
95
What is test specificity?
The proportion of COI- examinees who are correctly identified as such by a test
(high specificity test yields high levels of correct rejections)
96
What is positive predictive power?
The probability that someone with a positive test result has the COI
97
The likelihood that a profile of test scores will exceed criteria for abnormality increases as:
1. The number of tests in a battery increases
2. The z score cutoff used to classify a test score as abnormal decreases
3. The number of abnormal test scores required to reach criteria decreases
