lecture 5: standardized assessment and psychometrics (1) Flashcards
1
Q
what is an outcome?
A
the end result of clinical activity/intervention
2
Q
what is an outcome measure?
A
- instrument shown to measure desirable traits accurately
- any measurement system used to uncover or identify health outcome of treatment
- process by which changes measure over two or more points in time
3
Q
why is it important to measure?
A
- help determine status at the start of intervention
- help determine is someone is actually improving during and at end of intervention
- improves clinical decision-making, care and client outcomes
- component of EBP
- aid w/ objectivity –> concrete evidence
4
Q
what is the instrument evaluation process (IEP)?
A
- used to guide process of appraising outcome collection
- if answer no to any question = need to select another instrument
- IN SLIDES**
5
Q
what is step 1 of the IEP?
A
- is the assessment clinically useful?
- determine usefulness and usability for specific setting/purpose
6
Q
what factors do you need to consider in step 1 of the IEP?
A
- clinical applicability
- specificity
- availability
- time/training demands
- acceptability to clients
- cost
7
Q
what is step 2 of the IEP?
A
- is it standardized?
8
Q
what are standardized assessments?
A
- questions/methods/conditions for the administration, scoring, and interpretation of the results are consistent
- allows for trustworthy comparison of scores (time to time, client to similar group)
- any deviation from standardized procedures may = invalid conclusions about test performance (ex. modification of test instructions)
** want to stay as close to standardized instructions as possible - stray = lose standardization
9
Q
what are the components of a standardized assessment?
A
- assessment manual
- instructions for administration
- standardized equipment/questions
- data on test construction, reliability, validity
- normative data
10
Q
what is step 3 of the IEP?
A
- what is the instruments purpose?
11
Q
what are three possible purposes to standardized assessments?
A
- descriptive - describe the status of the person/group of interest
- predictive - predict the clients future status
- evaluative - evaluate the change in status of a client overtime
- within each measure, need to look at construction, reliability, and validity
12
Q
what are descriptive measures?
A
- describe 1+ aspects of a person’s status at one moment in time
- ex. occupational strengths/limitations, characteristics, behaviours
- often be used to classify an individual via comparison with norms
- information collected can be used to identify problems and to evaluate the need for intervention
13
Q
what are predictive measures?
A
- show/foretell 1+ aspects of the client’s future status
- used to predict abut potential of client (ex. safety at home)
- often have norms
- used to screen individuals to determine eligibility for intervention/potential to benefit from a program
14
Q
what are evaluative measures?
A
- evaluate change in status of a client overtime
- used at more than one point in time (beginning and end)
- must be sensitive to change (i.e. responsive)
- ex. COPM
15
Q
what is test construction?
A
- instrument development process
- item inclusion/exclusion - does it include the questions you’d expect to see?
- scaling/weighting - how to scoring work? how is it totaled?
descriptive construction = descriptive items
predictive construction = predictive items
evaluative construction = responsive items
16
Q
what is a norm-referenced measure?
A
- measures of the average or typical performance form the basis of how scores are interpreted
- norm = reference point for test score
- norm-referenced interpretation = comparing examinee’s test score to scores obtained by others in normative sample
- IMPORTANT to consider…
> characteristics of the sample in which the norms were developed
> how they were obtained
> how much that group is representative of the population the measure is intended for
17
Q
what is reliabilty?
A
- consistency
- trustworthiness of a measure and its results
- reliable measure yields dependable and consistent measurement of what you are trying to measure
- degree to which the measure yields results that are free from measurement error (.e. works to dec measurement error)
18
Q
what is a measurement error?
A
- difference between the true value of a phenomenon and its measured value
- caused by factors that…
> are irrelevant to what is being measured by the test > have an unpredictable effect on the test score
19
Q
what are sources of measurement error?
A
- test construction
- test administration
- test scoring
- test interpretation
20
Q
what are test construction errors?
A
- test questions worded unclearly
- test administration instructions unclear
- scoring procedure not clear
21
Q
what are test administrations errors?
A
- test environment issues
- test-taker motivation and attention
- examiner-related variables
22
Q
what are test scoring/interpretation errors?
A
- hand vs. computer scoring
- level of training
- subjectivity
23
Q
what are ways to minimize measurement error?
A
- choose assessment with strong psychometric properties
- pilot-test assessments and instruments
- follow standardized instructions
- train interviewers or observers
- make oberservation/measurement as unobtrusive as possible
- keep test environment/equipment consistant with standardization
- double-check data
24
Q
how is reliability usually scored?
A
- reported as a value of 0-1
> 1 = perfect reliability (no error) - closer to 1 = better reliability and less measurement error
- if value…
> 0.1 = poor reliability
> 0.9 = high reliability - statistic measures…
> Pearson product moment correlation coefficient (r)
> intra-class correlations coefficient (ICC)
> Spearman rank-order correlation (rho)
> Kappa statistic (k)
> Cronbach’s alpha
25
Q
what are the ways to establish reliability?
A
- test-retest
- inter-rater
- internal consistency
26
Q
what is test-retest reliability?
A
- stability of the measure overtime
- determined by calculating the agreement of scores at two different times for a characteristic that have NOT changed
> don’t use in client where statues to often variable - ICC > .70 = acceptable test-retest reliability (under = poor)
- time interval can vary depending on what is being measured
27
Q
whats is inter-rater reliablity?
A
- degree to which scores by different raters yield the same results
- applies to assessments where test administrator assesses result
- determined by having several raters measure the same phenomena
- acceptable inter-rater = > .70
- descriptive, evaluative and predictive assessments should have high inter-rater reliability
28
Q
what is internal consistency?
A
- degree of the relatedness among the items of an instrument
- used to determine if items on test are consistent with one another
- an estimate of the homogeneity of the structure of the test
- high internal consistency = items closely related
- measured with Cronbach’s alpha
- acceptable = 0.8 - 0.9
- if too high (0.97) = item redundancy
29
Q
what are the most important reliabilities for each instrument purpose?
A
descriptive:
- internal consistency
- observer
predictive:
- test-retest
- observer
evaluative:
- test-retest
- observer
30
Q
what is validity?
A
- accuracy
- extent to which assessment measures what it is intended to measure (ex. fatigue, balance, OP)
31
Q
what are the types of validity?
A
- face
- content
- criterion - concurrent and predictive
- construct - convergent, divergent, discriminative
- responsiveness
32
Q
what is face validity?
A
- assumption of validity based on a measure’s appearance
- subjective judgement - does test appear to measure what is says
- least reliable validity - ONLY used as preliminary screening
- if minimum requirement of face validity can not be established, its unlikely that it’ll hold up against other validity measures
33
Q
what is content validity?
A
- degree to which the instrument items are a comprehensive reflection of what the instrument reports to be measuring
- does measure include ALL elements of a given concept
- established based on theoretical frameworks, expert opinions, or literature review
34
Q
what is criterion validity?
A
- extent to which scores of assessments relate to gold standard/valid external criterion
- assessed by correlating the scores of a sample of individuals on the predictor with the scores on the criterion
- test = predictor, gold standard = criterion
35
Q
what are the two types of criterion validity?
A
- concurrent: criterion data collected at the same time as data on predictor test
> ex. TB test (skin test = predictor, chest x-ray = criterion) - predictive: criterion data collected after the predictors test was administered
> scores on the MCAT (predictor) predict performance in medical school (criterion)
36
Q
what is construct validity?
A
- degree to which scores of an instrument are consistent with a hypothesis about how they should perform
- based on testing a measure against an idea based on theory
- involves…
1. developing theoretical hypotheses relevant to the construct being assessed
2. investigating whether these hypotheses are upheld when the assessment is used - most difficult of all validities to establish
37
Q
what are the types of construct validity?
A
- convergent
- degree to which scores are consistent with a hypothesis that the instrument WILL CORRELATE with another measurement - divergent
- degree to which the scores of an instrument are consistent with a hypothesis that the instrument WILL NOT CORRELATE with another measurement - discriminative
- degree to which the scores of an instrument are consistent with a hypothesis concerning DIFFERENCES BETWEEN GROUPS
38
Q
what is responsiveness validity?
A
- ability of instrument to detect change over time in what it reports to be measuring
- aka. sensitivity to change
- evaluative assessments must have evidence of responsiveness
- done by taking a group that does change and seeing if the measure picks up change
- results expressed as effect size/standardized response mean
39
Q
what are two characteristics of responsiveness validity?
A
- minimal detectable change (MDC): what amount of change, taking error into account, means that a change has actually occurred
> change by 1 point = probably error
> change by 3 points = change has occurred - minimal clinically important difference (MCID): what a patient would notice to be a meaningful change
> grip strength increased by 3, MCID = 2 (can assume there was change)
> grip strength increased by 3, MCID = 5 (yes, grip strength improved BUT it hasn’t effected QOL)
40
Q
what are the most important validities for each instrument purpose?
A
descriptive:
- content
- construct
predictive:
- content
- criterion
evaluative:
- content
- construct
41
Q
what is cross-cultural validity?
A
- degree to which the performance of items on a translated or culturally adapted measure are a reflection of the performance of items of the original version
42
Q
what is ecological validity?
A
- degree to which a measure reflects real life
- ex. ability to memorize random words vs an address/name
43
Q
what are tips for evaluating a measure?
A
- obtain a copy of the measure
- not always easy
- detective work - refer to books that evaluate the measure
- read literature (especially the first publication of a measure)
- check wide range of literature
- follow a template for evaluation and do your own evaluation
44
Q
how do you find measures to use?
A
- search engine
- textbooks
- library database
- measurement cupboard
