Ch. 10: Using Assessment & Evaluation Strategies Flashcards
Assessment
-Process where info is gathered to determine whether learning has occurred
-May result in making changes to improve learner outcomes (similar to formative eval)
Evaluation (Eval)
-Systematic & continuous process
-Info gathered to determine:
-worth & value of program,
-outcomes, & achievement of learner
=summative eval (judgment will be made about outcome)
What is the most important predictor of success?
College GPA before admission
Other excellent predictors of success
-High GPAs in natural sci courses
-Course grade in intro psych course
-HS rank
Eval is conducted to determine:
-Learner progress toward achieving program outcomes
-Effectiveness of edu process to foster learning
-Accomplishment of mission of institution to prep nurses for entry into practice
Assessment involves…
Setting standards & criteria for learning, gathering, analyzing, interpreting data.
Assessment determines…
How performance matches standards & criteria
What is info gathered from Assessment used for?
To improve learning, teaching, courses, programs
Examples of eval philosophies
1) Practice orientation
2) Service orientation
3) Judgment orientation
4) Constructivist orientation
1) Practice orientation (Eval philo)
-Practice goals = reflected in course outcomes
-Performance achieved at end of course, clinical EXP, or program = most important consideration
2) Service orientation (Eval philo)
-Based on values approach to eval
-Takes more global or holistic view of goals of edu process beyond course outcomes
-Instructors view eval process as means to identify learner strengths & weaknesses in diff components of performance
3) Judgment orientation (Eval philo)
-Focus on determination of what’s acceptable of learner’s performance & value/grade that should be assigned based on performance
Constructivist orientation (Eval philo)
-Heaviest consideration to stakeholders who’ll be affected by success/failure of program
-Eval process involves input from pts & staff who contribute to assignment of grade in clinical setting
Who are stakeholders in Constructivist orientation?
Employers of graduates & pts who will receive graduates’ nursing care
What are the purposes of Eval?
Promote learning
Dx of prob’s (i.e. learning needs or deficits)
Decision-making r/t assignment of grades, tenure, promotion
Judge effectiveness of learner goal achievement
Formative Eval
Eval of learning while it’s occurring
Identifies learner’s readiness to learn & his/her learning needs
May be used to improve performance before end of course
Assigns no final grade
Shares eval w/ learner informally thru-out learning process but formally presents in clinical courses at mid-semester
Summative Eval
-Refers to lvl of performance of learner at end of course/program/activity
-Judges whether or not standards & criteria have been met for event (course/clinical/program)
-Uses data collected thru-out learning EXP as basis for eval
-Assigns final grade (i.e. Pass/Fail)
-Evaluator determines eval & shares w/ learner being evaluated
Eval model
-Helps explain process on which variables, items are evaluated
-Provides systematic plan or framework for eval
-Model should be selected based on context, needs of stakeholders, & question to be evaluated
Logic model (more detail on p. 281)
-Useful in designing program evals
-Helps conceptualize, plan, communicate w/ others about program
-Uses flowcharts to help clarify key elements of program
Decision-oriented models: CIPP (Context, Input, Process, & Product) (more detail on p. 281)
-Gives info on which decisions can be made
-Measures strengths & weaknesses of program
-Identifies target pop’s & options
-Detects defects in design or implementation & satisfaction w/ EXP
Kirkpatrick’s four-lvl model
Uses variety of inputs (i.e. students, employers, faculty - to evaluate outcomes)
Four lvls of Kirkpatrick’s model
1) Reaction: student perspective
2) Learning: whether it did occur
3) Bx: assessment of skills & application
4) Results: students’ success in translating knowledge into practice
3 main orgs accredit nurs programs
ACEN (Accreditation Commission for Edu in Nurs)
CCNE (Commission on Collegiate Nurs Edu)
CNEA (Commission on Nurs Edu Accreditation)
After eval model has been chosen, need to…
-select method(s) of eval
-(To assess effectiveness of learning & if course and program outcomes were achieved)
-(Usually multiple methods chosen (b/c learning evaluated in 3 domains)
3 types of evaluative mechanisms used in nurs edu
1) Achievement tests
2) Norm-referenced tests
3) Criterion-referenced test
1) Achievement tests (evaluative mechanism)
(have sound psychometric reliability)
Ex: standardized tests made by outside vendor
*Should measure clearly defined learning outcomes
*Measured in terms of learner performance
Should be concerned w/ all intended learning outcomes
*Must incl outcomes based on knowledge, skill, understanding, application, complex learning
Instructors need to use systematic procedure to obtain representative sample of test items relevant to instruction
*Should be based on plans for using results
*Should provide consistent results (=reliable)
2) Norm-referenced tests
-(rank group of learners)
-Scores compared to “bell curve”
-Interpreted in relation to ranking of learners in class
-Compares scores to those of peers’
-Focus on broad range of learning tasks
-Focus on learner ranking
Example of Norm-referenced test
SATs, ATIs
3) Criterion-referenced test
*(Closely aligned to achievement test)
*Criteria = set (score is compared to criterion; NOT to other scores)
*Expressed in terms of specific knowledge & skills learner can demonstrate
-Focus on detailed learning domain
-Should be designed to measure ability of learner in specific area
-Most = teacher made & based on course objectives
*Scores reported as % correct based on preset standards
Example of Criterion-referenced test
100% of learners will correctly calculate med dosages on calculation exam
What is 1st step in Planning a Test?
Determine what type of test should be given (based on purpose of test)
What is 2nd step in Planning a Test?
Identify & define intended learning outcomes
-Cognitive domain: addresses areas r/t intellectual abilities
-Affective domain: concerned w/ values & attitudes
-Psychomotor domain: addresses motor skills
What is 3rd step in Planning a Test?
Decide on types of test questions to be included (based on objectives & outcomes for test (should incl variety)
What domain is achievement testing mainly focused on?
Focused mainly on cognitive domain (w/ 6 main areas to consider)
What are the 6 main areas to consider that are focused mainly on Cognitive Domain (in achievement testing)?
1) Knowledge
2) Translation
3) Application
4) Analysis
5) Synthesis
6) Evaluation
1) Knowledge (cognitive domain)
-Intellectual abilities & skills
-Comprehension (grasping meaning of material)
2) Translation (cognitive domain)
(converting from 1 form to another)
Interpretation (explaining material)
Extrapolation (extending meaning beyond data)
3) Application (cognitive domain)
Using info in concrete situation
4) Analysis (cognitive domain)
(breaking down material into its parts)
Identify parts, relationships, organization
5) Synthesis (cognitive domain)
(putting parts together to create a whole)
Uniqueness
Abstract relations
6) Evaluation (cognitive domain)
-(judging value for given purpose using specific criteria)
-Judgment in terms of internal evidence
-Judgment in terms of external criteria
Pt needs = basis of test plan (incl four major areas)
1) Safe, effective care environ
2) Health promotion & maintenance
3) Psychosocial integrity
4) Physiological integrity
NGN
-Iterative process
-Uses nurses’ knowledge to observe & access presenting situation
-identify prioritized pt concern -generate best possible evidence-based soln to deliver safe pt care
Clinical judgment (NGN)
“Observed outcome of crit thinking & decision-making”
Clinical Judgment Model
-Recognizing & analyzing cues
-Prioritizing hypothesis
-Generating soln’s
-Taking action
-Evaluating outcomes
Approaches for testing validity:
- Content-related evidence
- Criterion-related evidence
- What was the correlation coefficient (r)?
- Construct-related evidence
Content-related evidence (Approach for testing validity)
How well does test measure intended learning outcomes?
Criterion-related evidence (Approach for testing validity)
How accurately does test performance predict future performance?
What was the correlation coefficient (r)? (Approach for testing validity)
-Pos (+) relationship - high or low scores on 1 measure are accompanied by high or low scores on another
-Neg (-) relationship - high scores on 1 measure are accompanied by low scores on another measure
Construct-related evidence (Approach for testing validity)
How well can test performance be explained in terms of psychological characteristics
What are influencing factors of Reliability?
Insufficient test length & group variability
What is Reliability measured by?
Correlation coefficient
KR-20 (formula)
-Estimates reliability of test scores from single administration of test
-Reflects accuracy or power of discrimination of test
What 3 types of info are required to determine KR-20 formula?
of items in test
The mean
Standard deviation
Standardized achievement tests should have reliability (KR) of what?
.90 or better
What reliability (KR) should classroom tests have?
KR of .50 and .80 (1.0 is maximum)
Factors that lower reliability of test scores
-Too few items on test
-Excessive #’s of very easy or very hard questions
-Inadequate testing conditions
-Items = poorly written (do not discriminate)
-Scoring = subjective (should prepare rubric)
Item analysis
Done to determine whether tests have separated learners from non-learners (discrimination)
Key concepts of Item analysis
-Item difficulty (p-value)
-% of group that answered item correctly
-p = .5 (50% correct) (is a good discrimination index)
-Upper limit = 1.00 (100% of learners answered question correctly)
-Lower limit depends on # of possible responses & probability of guessing correctly
-if there are four options, then p = .25 is lower limit or
probability of guessing
Item Discrimination
Measured by point-biserial correlation (measures each learner’s item performance w/ each learner’s overall test performance)
Questions that discriminate well have point-biserial correlations that are…
highly pos for correct answer (& neg for distractors)
What indices are good & very good?
Indices > .3 (good)
Indices > .4 (very good!)
Item difficulty of p = .5 shows…
-Discrimination index that is maximized
(if index is too high or too low, index is attenuated & item is a poor discriminator)
Distractors should appeal to who?
Nonlearners
What does it mean if distractors have point biserial of zero?
Learners did not select them (should be revised/replaced); learner may have chosen correct answer by guessing
When does neg discriminating power occur?
When more learners in lower group than upper group choose correct answer (these items need to be revised/replaced)
-Review p. 294
Multiple-choice questions are discriminating when…
they are answered correctly by upper 1/3 of class and answered incorrectly by lower 1/3 of class
Why do some evaluators like to purposely incl 1 or 2 easy questions at start of exam?
To decr learner’s anxiety
When should Item Revision be done?
After item analysis!
Revise items w/ following traits:
p - values that are too high/low
Correct answers w/ low pos or neg point biserials
Distractors w/ highly pos point-biserial correlations
Items that correlate w/ < .15 w/ total test scores should…
be restructured (may be confusing or misleading to exam takers)
What type of distractors should be replaced?
-Distractors not chosen by any learner (prevents discriminating good learners from poor learners)
-Items that all test takers get right (do not discriminate among learners and should be replaced by more difficult ones)
Important considerations when developing collaborative (group) learning assignments & eval of learning
-Assignments must be meaningful and should be designed for small groups whenever possible
-Educator must ensure members of group understand role of each person in group
-Groups should be structured heterogeneously (w/ regard to gender, ethnicity, ability, EXP, etc) to incr learning
Factors that affect learning environ after 1st semester:
Mid-program, progression criteria, program outcomes
Factor that affects students in 1st semester
Admission criteria
What does a neg (-) point-biserial indicate?
More learners who did not score well on exam picked the right answer (compared to learners who scored higher) (and vice versa)
