lecture 13

Question 1

Social intervention

Answer 1

An action taken – within a
particular social context – with the goal of producing
an intended outcome.

Question 2

Evaluation research

Answer 2

Research undertaken for the
purpose of determining the impact of some social
intervention.

Question 3

mission of the second research

Answer 3

To develop effective
leaders for the Canadian
agri-food industry.

Question 4

CALL objectives

Answer 4

1. Develop a pool of excellent leaders to serve national
   and regional organizations of importance to
   Canadian agriculture;
2. Create a network of leaders able to link diverse
   sectors and regions within the Canadian agri-food
   industry;
3. Create a network of leaders who will provide vision,
   advocacy, and leadership for the Canadian agri-food
   industry; and
4. Create a network of leaders who will, through their
   impact as educators, mentors and role models,
   improve production and farm business
   management practices across Canada

Question 5

Who took part?

Answer 5

16 men and 14 women.
 From all ten Canadian provinces.
 From just under thirty to just over fifty years of age.
 Two-thirds were farmers.
One-third employed in non-farm agri-businesses and
non-governmental organizations.
 Wealth of experience with agricultural leadership in
farms, agri-businesses, non-governmental
organizations and rural communities.

Question 6

The Kirkpatrick framework

Answer 6

Reaction
— Learning
— Behavior
— Results

Question 7

Level 1: Reaction

Answer 7

Frequently referred to as happy
face evaluation, this level measures participant
reaction to, and satisfaction with, the program and
the learning environment

Question 8

Level 2: Learning

Answer 8

Changes in knowledge, skills,
and/or attitudes constitute learning in the Kirkpatrick
model.

Question 9

Level 3: Behavior

Answer 9

this level determines whether
changes in behavior have occurred as a result of the
program

Question 10

Level 4: Results

Answer 10

Level 4 looks at the final results that
occurred because the participants attended the
program. Results can be thought of as “the bottom
line,“ or the impact of the program.

Question 11

Seminar evaluation instruments

Answer 11

1. Participants were asked to identify the greatest
   strength(s) of the seminar, taking into consideration
   its overall structure and content.
2. Participants were asked to rate the
   accomplishment of each of the learning objectives
   of the seminar on a scale from one (poor) to ten
   (outstanding).
3. Participants were asked to rate, on a scale from one
   (poor) to ten (outstanding), four aspects of the
   seminar:
   a. preparation via computer conferencing;
   b. overall content;
   c. organization and logistics;
   d. accommodation, meals and meeting spaces.
4. Participants were asked to provide any comments
   they may have had regarding the seminar, with a
   view to improving future seminars.

Question 12

Participants were asked to rate, on a scale from one
(poor) to ten (outstanding), four aspects of the
seminar:

Answer 12

a. preparation via computer conferencing;
b. overall content;
c. organization and logistics;
d. accommodation, meals and meeting spaces.

Question 13

seminar evaluation summary

Answer 13

Primarily aimed at gauging the satisfaction of
participants with leadership development seminars.
 Provided a self-assessment of learning achieved.
 Participants were not examined

Question 14

detailed questionnaire

Answer 14

open and closed questions

Question 15

Standardized leadership development instrument

Answer 15

Developed by James Kouzes and Barry Posner (1997)
as a companion to their 1995 textbook, The
Leadership Challenge.
 This text was required reading for CALL participants.

Question 16

Leadership Practices Inventory: purposes

Answer 16

Pedagogy: a means of encouraging participants to
reflect about their leadership practices.
 Peer assessment: one way to move beyond purely
self-reported assessments in our research.
 Pre-test / post-test structure: an indicator of
behavior change among participants over the course
of the program (Level 3).

Question 17

Leadership Practices Inventory: method

Answer 17

The LPI-Self asks respondents to rate (on a scale from
one to ten) the extent to which they typically engage
in thirty different behaviors. The LPI-Observer asks
those who have an opportunity to observe the
individual being rated to rate that person on the
same scale for the same thirty behaviors

Question 19

Leadership Practices Inventory: results

Answer 19

Pedagogically useful.
 Did not measure behavior change (ratings very high
at pre-test – little room for improvement).
 Either no change took place, or the instrument was
not sensitive enough to measure change

Question 20

Observers’ survey: outcomes

Answer 20

Evidence gathered from the observers was used to
provide some modest external validation of
participants’ self-reported behavior change and
impact.
 There was overwhelmingly positive response to the
program from both participants and the observers

Question 21

Basic results: reaction

Answer 21

We assembled a tremendous amount of data to
document that a great majority of participants were
satisfied with their experience of the program.
 With very few exceptions, participants expressed
high levels of satisfaction with the overall program.
 Specific activities within the program were not
always highly rated, and this led the ongoing
adaptation of the program to better suit its
participants.

Question 22

Basic results: reaction

Answer 22

A large majority of participants believe they
developed their knowledge, skills and networks
through participating in the program.
 Most were able to provide specific examples of their
learning and its application to their leadership
practices.
 Evaluation of learning was largely restricted to self-
reports (no pre-tests / post-tests).

Question 23

Basic results: behavior

Answer 23

Many CALL participants claimed to have changed
their leadership practices as a result of taking part.
 Most were able to provide examples of specific
changes they have made, and of concrete ways in
which they have applied their learning from CALL to
agricultural leadership work.
 These claims were supported by the modest survey
of peers that was conducted.
 The LPI did not measure behavior change.

Question 24

Basic results: impact

Answer 24

Many participants claimed that the CALL program
had an impact on their leadership practices, and as a
result on businesses and organizations of importance
to Canadian agriculture.
 Many were able to articulate specific examples.
 Given the tremendous difficulty of creating control-
group conditions, the attribution of behavior change
and impact to the specific interventions of the
program is very difficult
- it is hard to say if it was the program that made them better leaders or if they just matured

Question 25

Key outcomes

Answer 25

 Formative evaluation influenced the design of
learning activities during the program.
 Summative evaluation gathered evidence for the
effectiveness of the program.
 CALL was funded by the CFBMC for a second cohort.

Question 26

Evaluation research

Answer 26

Expensive and time-consuming.
 Kirkpatrick framework made sense to stakeholders.
 The impossibility of isolating the impact of the
intervention from maturation and everything else
that happened over the 18 months of the program
(impracticality of control groups) meant that Levels 3
and 4 were difficult to evaluate truly

Question 27

quasi experiment