lecture 5: standardized assessment and psychometrics (1) Flashcards
what is an outcome?
the end result of clinical activity/intervention
what is an outcome measure?
- instrument shown to measure desirable traits accurately
- any measurement system used to uncover or identify health outcome of treatment
- process by which changes measure over two or more points in time
why is it important to measure?
- help determine status at the start of intervention
- help determine is someone is actually improving during and at end of intervention
- improves clinical decision-making, care and client outcomes
- component of EBP
- aid w/ objectivity –> concrete evidence
what is the instrument evaluation process (IEP)?
- used to guide process of appraising outcome collection
- if answer no to any question = need to select another instrument
- IN SLIDES**
what is step 1 of the IEP?
- is the assessment clinically useful?
- determine usefulness and usability for specific setting/purpose
what factors do you need to consider in step 1 of the IEP?
- clinical applicability
- specificity
- availability
- time/training demands
- acceptability to clients
- cost
what is step 2 of the IEP?
- is it standardized?
what are standardized assessments?
- questions/methods/conditions for the administration, scoring, and interpretation of the results are consistent
- allows for trustworthy comparison of scores (time to time, client to similar group)
- any deviation from standardized procedures may = invalid conclusions about test performance (ex. modification of test instructions)
** want to stay as close to standardized instructions as possible - stray = lose standardization
what are the components of a standardized assessment?
- assessment manual
- instructions for administration
- standardized equipment/questions
- data on test construction, reliability, validity
- normative data
what is step 3 of the IEP?
- what is the instruments purpose?
what are three possible purposes to standardized assessments?
- descriptive - describe the status of the person/group of interest
- predictive - predict the clients future status
- evaluative - evaluate the change in status of a client overtime
- within each measure, need to look at construction, reliability, and validity
what are descriptive measures?
- describe 1+ aspects of a person’s status at one moment in time
- ex. occupational strengths/limitations, characteristics, behaviours
- often be used to classify an individual via comparison with norms
- information collected can be used to identify problems and to evaluate the need for intervention
what are predictive measures?
- show/foretell 1+ aspects of the client’s future status
- used to predict abut potential of client (ex. safety at home)
- often have norms
- used to screen individuals to determine eligibility for intervention/potential to benefit from a program
what are evaluative measures?
- evaluate change in status of a client overtime
- used at more than one point in time (beginning and end)
- must be sensitive to change (i.e. responsive)
- ex. COPM
what is test construction?
- instrument development process
- item inclusion/exclusion - does it include the questions you’d expect to see?
- scaling/weighting - how to scoring work? how is it totaled?
descriptive construction = descriptive items
predictive construction = predictive items
evaluative construction = responsive items
what is a norm-referenced measure?
- measures of the average or typical performance form the basis of how scores are interpreted
- norm = reference point for test score
- norm-referenced interpretation = comparing examinee’s test score to scores obtained by others in normative sample
- IMPORTANT to consider…
> characteristics of the sample in which the norms were developed
> how they were obtained
> how much that group is representative of the population the measure is intended for
what is reliabilty?
- consistency
- trustworthiness of a measure and its results
- reliable measure yields dependable and consistent measurement of what you are trying to measure
- degree to which the measure yields results that are free from measurement error (.e. works to dec measurement error)
what is a measurement error?
- difference between the true value of a phenomenon and its measured value
- caused by factors that…
> are irrelevant to what is being measured by the test > have an unpredictable effect on the test score
what are sources of measurement error?
- test construction
- test administration
- test scoring
- test interpretation
what are test construction errors?
- test questions worded unclearly
- test administration instructions unclear
- scoring procedure not clear
what are test administrations errors?
- test environment issues
- test-taker motivation and attention
- examiner-related variables
what are test scoring/interpretation errors?
- hand vs. computer scoring
- level of training
- subjectivity
what are ways to minimize measurement error?
- choose assessment with strong psychometric properties
- pilot-test assessments and instruments
- follow standardized instructions
- train interviewers or observers
- make oberservation/measurement as unobtrusive as possible
- keep test environment/equipment consistant with standardization
- double-check data
how is reliability usually scored?
- reported as a value of 0-1
> 1 = perfect reliability (no error) - closer to 1 = better reliability and less measurement error
- if value…
> 0.1 = poor reliability
> 0.9 = high reliability - statistic measures…
> Pearson product moment correlation coefficient (r)
> intra-class correlations coefficient (ICC)
> Spearman rank-order correlation (rho)
> Kappa statistic (k)
> Cronbach’s alpha