10. Evaluation (people/projects) Flashcards
personal evaluations
(individual level)
Recognise people’s achievements
Encourage future performance
program evaluations
(systems level)
Identify how a policy or intervention is working
Identify areas for improvement/refinement
Decisions to continue/abandon program
what is the process of person evaluation?
career planning (describe career and work goals) -> planning proposal (create goals expected outcomes) -> performance agreement (agree on performance indicators, targets, timeframe) -> performance evaluation (compare indicators with targets) -> feedback discussion (revise and update career goals) -> back to career planning
Why do we evaluate people?
Assign rewards (e.g., bonuses) Identify where help/training is needed Identify when people can take on greater challenges and responsibilities (e.g., promotion) Reinforce good behaviours Extinguish bad behaviours
what do we evaluate?
performance
effectiveness
prodctivity
what do we evaluate in performance?
task-related behaviour
what do we evaluate in effectiveness?
evaluation of standard of performance
what do we evaluate in productivity?
cost of achieving level of effectiveness
time, money, burnout
what is Bartram’s GREAT EIGHT competency model?
Leading/deciding supporting/cooperating interaction/presenting analysing / interpreting creating/conceptualising organising/executing adapting/coping enterprising/performing
who evaluates?
supervisor other peers clients subordinates team members
goes in a circle - 360 degrees feedback
how do we evaluate?
with objective measures and judgemental measures
what are the objective measures of evaluation?
Quantitative measure of production, e.g., sales, outputs
Academics:
- Research (papers published; article views; number of citations, h-index, i-10 index…)
- Teaching (unit ratings; number of units taught)
what are the judgemental measures of evaluation?
allows consideration of context factors not captured by objective measures
e.g., supervisor’s overall impression
rating compared to other employees, influenced by perceived difficulty of job or other contextual information
Why do we abandon annual performance reviews?
due to measurement and psychological factors
measurement factors for abandoning annual performance reviews
annual reviews are disconnected from timescales of work multiple factors (and people) contribute to one’s performance
psycholigical factors for abandoning annual performance reviews
Competition v. collaboration -
Performance rankings set up competitive mindsets
Doesn’t satisfy needs for learning and growth-
Fostered by more immediate feedback
More frequent communication = more informative feedback
Shifts from debating performance ratings to discussing development opportunities
what should person evaluations be?
Developmental
Tied to organisational objectives
Specific
Sufficiently frequent
how should person evaluations be sufficiently frequent?
by evaluating according to:
according to a person’s experience (less experience, more frequent)
task timeline (shorter task timelines, more frequent)
timely for decision-making
types of evaluation
proactive clarificative interactive monitoring impact
proactive evaluation
Assessing need for a program
If program required, assessing how it has been done elsewhere
clarificative evaluation
Understand rationale and plausibility of a proposed program prior to commitment
interactive evaluation
Inform stakeholders about issues and concerns with a planned program
Use knowledge and expertise to improve a planned program
monitoring evaluation
Assessing program processes
Is process of implementing program on track?
impact evaluatin
Identify the program/intervention outcomes
Inform decisions about future programs, e.g., whether to use approach in other settings
the steps to program evaluation
engage stakeholders (understand key issues) -> describe program (goal & purpose expected effects) -> design evaluation (Method, users, agreements) -> gather evidence (indicators, sources, quality, quantity, practicality) -> justify conclusions (analyse, interpretation, recommendations) -> tweak or revise program (implement lessons) -> back to engage stakeholders
What are the general standards of program evaluations
utility
feasibility
proprietary
accuracy
utility standard for program evaluations
Information will serve user needs
Measures are directly related to program goals
It is clear what the findings can show
feasibility standard for program evaluations
Information is practical to collect
Use of resources in proportion to the program/issue
Methods require skills within organisation’s capacities
proprietary standard for program evaluations
Information is obtained ethically
Respect the rights and welfare of those involved
Adheres to relevant laws and workplace policies
accuracy standard for program evaluations
Information is accurate
Reliability: indicators produce consistent responses
Validity: Indicators fairly represent the construct/idea
How does one describe the program?
What is the main purpose of your specific program/intervention? For example…
- change behaviours directly
- change attitudes
- change social norms
- change public awareness
- Change knowledge
Evaluation needs to be linked to your purpose
what are the ideal features for evaluation measures?
- Derived from the theory driving the intervention
- Includes measures of…
- the intervention itself (whether it was implemented correctly)
- its impact (consequences) - Include both immediate and longer-term consequences
- Valid
-measures are clearly related to the theoretical construct - Reliable
e.g., survey items are rated consistently – highly correlated with each other - Uses multiple methods
to make up for weaknesses of any single type of measure
what are the types of measures for gathering evidence?
Behaviours Behavioural intentions Attitudes Perceptions - Knowledge - Social norms -> Descriptive: reports of observed behaviours (what people actually do) -> Injunctive: shared expectations about what people should/should not do -Awareness
what is the hierarchy for selecting intention, attitude and norm measures in gathering evidence?
- USE existing scales when suitable
- ADAPT existing scales when possible
- DEVELOP new scales when there are no suitable/adaptable existing scales
what are the issues in creating items for gathering evidence?
leading or biased questions
eg. “how much do you dislike getting up early in the morning”
double-barrelled questions (unless topics very closely aligned)
eg. “please rate the extent to which you enjoy lectures and tutorials”
use scales likely to match the detail people are likely to use
eg. 5-, 7-, 9-point scales; percentages
what are the direct measures for measuring norms in the theory of planned behaviour?
descriptive
injunctive
descriptive measures of measuring norms
(what others do)
many people who are important to me do…
people in my life whose opinions I value do…
injunctive measures of measuring norms
(what others expect of you)
many people who are important to me think that I should…
it is expected of me that I will…
people in my life whose opinions I value would approve of…
what are the belief-based measures for measuring norms in the theory of planned behaviour?
strength
motivation to comply
strength measures of measuring norms
(measurement similar to injunctive)
My family thinks that I should…
usually tied to distinct reference groups
motivation to comply measures of measuring norms
E.g., generally, how much do you want to do what your family thinks you should do?
what is a common multi-method form of research?
Field studies
what rare field studies?
a general term for methods emphasising observing and interacting with people in natural contexts
what do field studies include?
observation
inquiry
breach
what is observation in field studies?
observe/record
could be observing behaviour directly, or records of behaviour (eg. transcripts of conversations)
what is inquiry in field studies?
interview people in relevant situations
gain knowledge about context as well as people
what is breach in field studies?
record reactions to unexpected actions
e.g., ethnomethodology – reactions to norm-breaking
