lecture 5: standardized assessment and psychometrics (1) Flashcards

1
Q

what is an outcome?

A

the end result of clinical activity/intervention

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

what is an outcome measure?

A
  • instrument shown to measure desirable traits accurately
  • any measurement system used to uncover or identify health outcome of treatment
  • process by which changes measure over two or more points in time
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

why is it important to measure?

A
  • help determine status at the start of intervention
  • help determine is someone is actually improving during and at end of intervention
  • improves clinical decision-making, care and client outcomes
  • component of EBP
  • aid w/ objectivity –> concrete evidence
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

what is the instrument evaluation process (IEP)?

A
  • used to guide process of appraising outcome collection
  • if answer no to any question = need to select another instrument
  • IN SLIDES**
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

what is step 1 of the IEP?

A
  1. is the assessment clinically useful?
    - determine usefulness and usability for specific setting/purpose
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

what factors do you need to consider in step 1 of the IEP?

A
  • clinical applicability
  • specificity
  • availability
  • time/training demands
  • acceptability to clients
  • cost
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

what is step 2 of the IEP?

A
  1. is it standardized?
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

what are standardized assessments?

A
  • questions/methods/conditions for the administration, scoring, and interpretation of the results are consistent
  • allows for trustworthy comparison of scores (time to time, client to similar group)
  • any deviation from standardized procedures may = invalid conclusions about test performance (ex. modification of test instructions)

** want to stay as close to standardized instructions as possible - stray = lose standardization

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

what are the components of a standardized assessment?

A
  • assessment manual
  • instructions for administration
  • standardized equipment/questions
  • data on test construction, reliability, validity
  • normative data
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

what is step 3 of the IEP?

A
  1. what is the instruments purpose?
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

what are three possible purposes to standardized assessments?

A
  1. descriptive - describe the status of the person/group of interest
  2. predictive - predict the clients future status
  3. evaluative - evaluate the change in status of a client overtime
  • within each measure, need to look at construction, reliability, and validity
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

what are descriptive measures?

A
  • describe 1+ aspects of a person’s status at one moment in time
  • ex. occupational strengths/limitations, characteristics, behaviours
  • often be used to classify an individual via comparison with norms
  • information collected can be used to identify problems and to evaluate the need for intervention
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

what are predictive measures?

A
  • show/foretell 1+ aspects of the client’s future status
  • used to predict abut potential of client (ex. safety at home)
  • often have norms
  • used to screen individuals to determine eligibility for intervention/potential to benefit from a program
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

what are evaluative measures?

A
  • evaluate change in status of a client overtime
  • used at more than one point in time (beginning and end)
  • must be sensitive to change (i.e. responsive)
  • ex. COPM
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

what is test construction?

A
  • instrument development process
  • item inclusion/exclusion - does it include the questions you’d expect to see?
  • scaling/weighting - how to scoring work? how is it totaled?

descriptive construction = descriptive items
predictive construction = predictive items
evaluative construction = responsive items

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

what is a norm-referenced measure?

A
  • measures of the average or typical performance form the basis of how scores are interpreted
  • norm = reference point for test score
  • norm-referenced interpretation = comparing examinee’s test score to scores obtained by others in normative sample
  • IMPORTANT to consider…
    > characteristics of the sample in which the norms were developed
    > how they were obtained
    > how much that group is representative of the population the measure is intended for
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

what is reliabilty?

A
  • consistency
  • trustworthiness of a measure and its results
  • reliable measure yields dependable and consistent measurement of what you are trying to measure
  • degree to which the measure yields results that are free from measurement error (.e. works to dec measurement error)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

what is a measurement error?

A
  • difference between the true value of a phenomenon and its measured value
  • caused by factors that…
    > are irrelevant to what is being measured by the test > have an unpredictable effect on the test score
19
Q

what are sources of measurement error?

A
  1. test construction
  2. test administration
  3. test scoring
  4. test interpretation
20
Q

what are test construction errors?

A
  • test questions worded unclearly
  • test administration instructions unclear
  • scoring procedure not clear
21
Q

what are test administrations errors?

A
  • test environment issues
  • test-taker motivation and attention
  • examiner-related variables
22
Q

what are test scoring/interpretation errors?

A
  • hand vs. computer scoring
  • level of training
  • subjectivity
23
Q

what are ways to minimize measurement error?

A
  • choose assessment with strong psychometric properties
  • pilot-test assessments and instruments
  • follow standardized instructions
  • train interviewers or observers
  • make oberservation/measurement as unobtrusive as possible
  • keep test environment/equipment consistant with standardization
  • double-check data
24
Q

how is reliability usually scored?

A
  • reported as a value of 0-1
    > 1 = perfect reliability (no error)
  • closer to 1 = better reliability and less measurement error
  • if value…
    > 0.1 = poor reliability
    > 0.9 = high reliability
  • statistic measures…
    > Pearson product moment correlation coefficient (r)
    > intra-class correlations coefficient (ICC)
    > Spearman rank-order correlation (rho)
    > Kappa statistic (k)
    > Cronbach’s alpha
25
what are the ways to establish reliability?
1. test-retest 2. inter-rater 3. internal consistency
26
what is test-retest reliability?
- stability of the measure overtime - determined by calculating the agreement of scores at two different times for a characteristic that have NOT changed > don't use in client where statues to often variable - ICC > .70 = acceptable test-retest reliability (under = poor) - time interval can vary depending on what is being measured
27
whats is inter-rater reliablity?
- degree to which scores by different raters yield the same results - applies to assessments where test administrator assesses result - determined by having several raters measure the same phenomena - acceptable inter-rater = > .70 - descriptive, evaluative and predictive assessments should have high inter-rater reliability
28
what is internal consistency?
- degree of the relatedness among the items of an instrument - used to determine if items on test are consistent with one another - an estimate of the homogeneity of the structure of the test - high internal consistency = items closely related - measured with Cronbach's alpha - acceptable = 0.8 - 0.9 - if too high (0.97) = item redundancy
29
what are the most important reliabilities for each instrument purpose?
descriptive: - internal consistency - observer predictive: - test-retest - observer evaluative: - test-retest - observer
30
what is validity?
- accuracy - extent to which assessment measures what it is intended to measure (ex. fatigue, balance, OP)
31
what are the types of validity?
1. face 2. content 3. criterion - concurrent and predictive 4. construct - convergent, divergent, discriminative 5. responsiveness
32
what is face validity?
- assumption of validity based on a measure's appearance - subjective judgement - does test appear to measure what is says - least reliable validity - ONLY used as preliminary screening - if minimum requirement of face validity can not be established, its unlikely that it'll hold up against other validity measures
33
what is content validity?
- degree to which the instrument items are a comprehensive reflection of what the instrument reports to be measuring - does measure include ALL elements of a given concept - established based on theoretical frameworks, expert opinions, or literature review
34
what is criterion validity?
- extent to which scores of assessments relate to gold standard/valid external criterion - assessed by correlating the scores of a sample of individuals on the predictor with the scores on the criterion - test = predictor, gold standard = criterion
35
what are the two types of criterion validity?
1. concurrent: criterion data collected at the same time as data on predictor test > ex. TB test (skin test = predictor, chest x-ray = criterion) 2. predictive: criterion data collected after the predictors test was administered > scores on the MCAT (predictor) predict performance in medical school (criterion)
36
what is construct validity?
- degree to which scores of an instrument are consistent with a hypothesis about how they should perform - based on testing a measure against an idea based on theory - involves... 1. developing theoretical hypotheses relevant to the construct being assessed 2. investigating whether these hypotheses are upheld when the assessment is used - most difficult of all validities to establish
37
what are the types of construct validity?
1. convergent - degree to which scores are consistent with a hypothesis that the instrument WILL CORRELATE with another measurement 2. divergent - degree to which the scores of an instrument are consistent with a hypothesis that the instrument WILL NOT CORRELATE with another measurement 3. discriminative - degree to which the scores of an instrument are consistent with a hypothesis concerning DIFFERENCES BETWEEN GROUPS
38
what is responsiveness validity?
- ability of instrument to detect change over time in what it reports to be measuring - aka. sensitivity to change - evaluative assessments must have evidence of responsiveness - done by taking a group that does change and seeing if the measure picks up change - results expressed as effect size/standardized response mean
39
what are two characteristics of responsiveness validity?
- minimal detectable change (MDC): what amount of change, taking error into account, means that a change has actually occurred > change by 1 point = probably error > change by 3 points = change has occurred - minimal clinically important difference (MCID): what a patient would notice to be a meaningful change > grip strength increased by 3, MCID = 2 (can assume there was change) > grip strength increased by 3, MCID = 5 (yes, grip strength improved BUT it hasn't effected QOL)
40
what are the most important validities for each instrument purpose?
descriptive: - content - construct predictive: - content - criterion evaluative: - content - construct
41
what is cross-cultural validity?
- degree to which the performance of items on a translated or culturally adapted measure are a reflection of the performance of items of the original version
42
what is ecological validity?
- degree to which a measure reflects real life - ex. ability to memorize random words vs an address/name
43
what are tips for evaluating a measure?
1. obtain a copy of the measure - not always easy - detective work 2. refer to books that evaluate the measure 3. read literature (especially the first publication of a measure) 4. check wide range of literature 5. follow a template for evaluation and do your own evaluation
44
how do you find measures to use?
- search engine - textbooks - library database - measurement cupboard