Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

1141-Assessments > lecture 5: standardized assessment and psychometrics (1) > Flashcards

lecture 5: standardized assessment and psychometrics (1) Flashcards

1
Q

what is an outcome?

A

the end result of clinical activity/intervention

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

what is an outcome measure?

A
  • instrument shown to measure desirable traits accurately
  • any measurement system used to uncover or identify health outcome of treatment
  • process by which changes measure over two or more points in time
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

why is it important to measure?

A
  • help determine status at the start of intervention
  • help determine is someone is actually improving during and at end of intervention
  • improves clinical decision-making, care and client outcomes
  • component of EBP
  • aid w/ objectivity –> concrete evidence
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

what is the instrument evaluation process (IEP)?

A
  • used to guide process of appraising outcome collection
  • if answer no to any question = need to select another instrument
  • IN SLIDES**
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

what is step 1 of the IEP?

A
  1. is the assessment clinically useful?
    - determine usefulness and usability for specific setting/purpose
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

what factors do you need to consider in step 1 of the IEP?

A
  • clinical applicability
  • specificity
  • availability
  • time/training demands
  • acceptability to clients
  • cost
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

what is step 2 of the IEP?

A
  1. is it standardized?
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

what are standardized assessments?

A
  • questions/methods/conditions for the administration, scoring, and interpretation of the results are consistent
  • allows for trustworthy comparison of scores (time to time, client to similar group)
  • any deviation from standardized procedures may = invalid conclusions about test performance (ex. modification of test instructions)

** want to stay as close to standardized instructions as possible - stray = lose standardization

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

what are the components of a standardized assessment?

A
  • assessment manual
  • instructions for administration
  • standardized equipment/questions
  • data on test construction, reliability, validity
  • normative data
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

what is step 3 of the IEP?

A
  1. what is the instruments purpose?
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

what are three possible purposes to standardized assessments?

A
  1. descriptive - describe the status of the person/group of interest
  2. predictive - predict the clients future status
  3. evaluative - evaluate the change in status of a client overtime
  • within each measure, need to look at construction, reliability, and validity
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

what are descriptive measures?

A
  • describe 1+ aspects of a person’s status at one moment in time
  • ex. occupational strengths/limitations, characteristics, behaviours
  • often be used to classify an individual via comparison with norms
  • information collected can be used to identify problems and to evaluate the need for intervention
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

what are predictive measures?

A
  • show/foretell 1+ aspects of the client’s future status
  • used to predict abut potential of client (ex. safety at home)
  • often have norms
  • used to screen individuals to determine eligibility for intervention/potential to benefit from a program
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

what are evaluative measures?

A
  • evaluate change in status of a client overtime
  • used at more than one point in time (beginning and end)
  • must be sensitive to change (i.e. responsive)
  • ex. COPM
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

what is test construction?

A
  • instrument development process
  • item inclusion/exclusion - does it include the questions you’d expect to see?
  • scaling/weighting - how to scoring work? how is it totaled?

descriptive construction = descriptive items
predictive construction = predictive items
evaluative construction = responsive items

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

what is a norm-referenced measure?

A
  • measures of the average or typical performance form the basis of how scores are interpreted
  • norm = reference point for test score
  • norm-referenced interpretation = comparing examinee’s test score to scores obtained by others in normative sample
  • IMPORTANT to consider…
    > characteristics of the sample in which the norms were developed
    > how they were obtained
    > how much that group is representative of the population the measure is intended for
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

what is reliabilty?

A
  • consistency
  • trustworthiness of a measure and its results
  • reliable measure yields dependable and consistent measurement of what you are trying to measure
  • degree to which the measure yields results that are free from measurement error (.e. works to dec measurement error)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

what is a measurement error?

A
  • difference between the true value of a phenomenon and its measured value
  • caused by factors that…
    > are irrelevant to what is being measured by the test > have an unpredictable effect on the test score
19
Q

what are sources of measurement error?

A
  1. test construction
  2. test administration
  3. test scoring
  4. test interpretation
20
Q

what are test construction errors?

A
  • test questions worded unclearly
  • test administration instructions unclear
  • scoring procedure not clear
21
Q

what are test administrations errors?

A
  • test environment issues
  • test-taker motivation and attention
  • examiner-related variables
22
Q

what are test scoring/interpretation errors?

A
  • hand vs. computer scoring
  • level of training
  • subjectivity
23
Q

what are ways to minimize measurement error?

A
  • choose assessment with strong psychometric properties
  • pilot-test assessments and instruments
  • follow standardized instructions
  • train interviewers or observers
  • make oberservation/measurement as unobtrusive as possible
  • keep test environment/equipment consistant with standardization
  • double-check data
24
Q

how is reliability usually scored?

A
  • reported as a value of 0-1
    > 1 = perfect reliability (no error)
  • closer to 1 = better reliability and less measurement error
  • if value…
    > 0.1 = poor reliability
    > 0.9 = high reliability
  • statistic measures…
    > Pearson product moment correlation coefficient (r)
    > intra-class correlations coefficient (ICC)
    > Spearman rank-order correlation (rho)
    > Kappa statistic (k)
    > Cronbach’s alpha
25
Q

what are the ways to establish reliability?

A
  1. test-retest
  2. inter-rater
  3. internal consistency
26
Q

what is test-retest reliability?

A
  • stability of the measure overtime
  • determined by calculating the agreement of scores at two different times for a characteristic that have NOT changed
    > don’t use in client where statues to often variable
  • ICC > .70 = acceptable test-retest reliability (under = poor)
  • time interval can vary depending on what is being measured
27
Q

whats is inter-rater reliablity?

A
  • degree to which scores by different raters yield the same results
  • applies to assessments where test administrator assesses result
  • determined by having several raters measure the same phenomena
  • acceptable inter-rater = > .70
  • descriptive, evaluative and predictive assessments should have high inter-rater reliability
28
Q

what is internal consistency?

A
  • degree of the relatedness among the items of an instrument
  • used to determine if items on test are consistent with one another
  • an estimate of the homogeneity of the structure of the test
  • high internal consistency = items closely related
  • measured with Cronbach’s alpha
  • acceptable = 0.8 - 0.9
  • if too high (0.97) = item redundancy
29
Q

what are the most important reliabilities for each instrument purpose?

A

descriptive:
- internal consistency
- observer

predictive:
- test-retest
- observer

evaluative:
- test-retest
- observer

30
Q

what is validity?

A
  • accuracy
  • extent to which assessment measures what it is intended to measure (ex. fatigue, balance, OP)
31
Q

what are the types of validity?

A
  1. face
  2. content
  3. criterion - concurrent and predictive
  4. construct - convergent, divergent, discriminative
  5. responsiveness
32
Q

what is face validity?

A
  • assumption of validity based on a measure’s appearance
  • subjective judgement - does test appear to measure what is says
  • least reliable validity - ONLY used as preliminary screening
  • if minimum requirement of face validity can not be established, its unlikely that it’ll hold up against other validity measures
33
Q

what is content validity?

A
  • degree to which the instrument items are a comprehensive reflection of what the instrument reports to be measuring
  • does measure include ALL elements of a given concept
  • established based on theoretical frameworks, expert opinions, or literature review
34
Q

what is criterion validity?

A
  • extent to which scores of assessments relate to gold standard/valid external criterion
  • assessed by correlating the scores of a sample of individuals on the predictor with the scores on the criterion
  • test = predictor, gold standard = criterion
35
Q

what are the two types of criterion validity?

A
  1. concurrent: criterion data collected at the same time as data on predictor test
    > ex. TB test (skin test = predictor, chest x-ray = criterion)
  2. predictive: criterion data collected after the predictors test was administered
    > scores on the MCAT (predictor) predict performance in medical school (criterion)
36
Q

what is construct validity?

A
  • degree to which scores of an instrument are consistent with a hypothesis about how they should perform
  • based on testing a measure against an idea based on theory
  • involves…
    1. developing theoretical hypotheses relevant to the construct being assessed
    2. investigating whether these hypotheses are upheld when the assessment is used
  • most difficult of all validities to establish
37
Q

what are the types of construct validity?

A
  1. convergent
    - degree to which scores are consistent with a hypothesis that the instrument WILL CORRELATE with another measurement
  2. divergent
    - degree to which the scores of an instrument are consistent with a hypothesis that the instrument WILL NOT CORRELATE with another measurement
  3. discriminative
    - degree to which the scores of an instrument are consistent with a hypothesis concerning DIFFERENCES BETWEEN GROUPS
38
Q

what is responsiveness validity?

A
  • ability of instrument to detect change over time in what it reports to be measuring
  • aka. sensitivity to change
  • evaluative assessments must have evidence of responsiveness
  • done by taking a group that does change and seeing if the measure picks up change
  • results expressed as effect size/standardized response mean
39
Q

what are two characteristics of responsiveness validity?

A
  • minimal detectable change (MDC): what amount of change, taking error into account, means that a change has actually occurred
    > change by 1 point = probably error
    > change by 3 points = change has occurred
  • minimal clinically important difference (MCID): what a patient would notice to be a meaningful change
    > grip strength increased by 3, MCID = 2 (can assume there was change)
    > grip strength increased by 3, MCID = 5 (yes, grip strength improved BUT it hasn’t effected QOL)
40
Q

what are the most important validities for each instrument purpose?

A

descriptive:
- content
- construct

predictive:
- content
- criterion

evaluative:
- content
- construct

41
Q

what is cross-cultural validity?

A
  • degree to which the performance of items on a translated or culturally adapted measure are a reflection of the performance of items of the original version
42
Q

what is ecological validity?

A
  • degree to which a measure reflects real life
  • ex. ability to memorize random words vs an address/name
43
Q

what are tips for evaluating a measure?

A
  1. obtain a copy of the measure
    - not always easy
    - detective work
  2. refer to books that evaluate the measure
  3. read literature (especially the first publication of a measure)
  4. check wide range of literature
  5. follow a template for evaluation and do your own evaluation
44
Q

how do you find measures to use?

A
  • search engine
  • textbooks
  • library database
  • measurement cupboard

1141-Assessments (5 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions