Chapter 8

Question 1

evaluating effectiveness of training and development

Answer 1

is a set of planned, information gathering and analytical activities undertaken by ETD practitioners to provide those responsible for the management of the strategic HRD effort with an assessment of ETD interventions’ quality and impact

Evaluations of the effectiveness of learning programmes are usually done to provide information and influence a decision that has to
be made about the HRD strategy, practices and procedures

Evaluation provides diagnostic
information that shows where remedial actions should be undertaken and whether the
L&D intervention should be continued

Question 2

validity vs reliability

Answer 2

Validity is the extent to which the measuring instrument reflects the concept it is intended to measure.

Reliability is the extent to which scores obtained on a measure are reproducible in repeated
administrations under similar measurement conditions

Question 3

different types of evaluation

Answer 3

Similar to assessment evaluation can take place at different times: 1. diagnostic evaluation - before a training (learning) intervention
2. formative evaluation - during a training intervention
3. summative evaluation - at the conclusion of a learning programme
4. longitudinal evaluation - after a learning programme

Question 4

comparing assessment and evaluation

Answer 4

Assessment focuses on evaluating collected
evidence of learners’ achievements against a set standard.

Evaluation, on the other hand, makes judgements about the quality and added value of learning programmes and whether changes and/or improvements in learners’ performance in the workplace occurred as a result of the learning programme

Question 5

advantages of L&D training evaluation

Answer 5

• measures the difference what was required and what has been achieved
• justifies HRD budget
• improves design and delivery of learning programmes
• improves transfer of learner
• identifies unnecessary or ineffective programs
• improves the credibility of HRD
• meets the needs of management and gains their support
• shows the financial return on training

Question 6

timing of evaluation and its related purpose

Answer 6

1. diagnostic evaluation
   - design of the program
   - existing skills level as part of needs analysis
2. formative evaluation
   - quality of delivery process
   - adequacy of learning material
   - appropriateness of delivery methods
3. summative evaluation
   - satisfaction of learners with program
   - achievement of outcomes
   - overall effectiveness
4. longitudinal evaluation
   - transfer and application of learning in workplace
   - support for new knowledge, skills and attitudes in workplace

Question 7

two types of evaluation

Answer 7

1. compliance
2. value-added

Question 8

compliance evaluation

Answer 8

• in terms of national standards = compulsory
• organisation’s compliance with international and national quality standards for outcomes-based L&D practices

Question 9

steps of compliance evaluation

Answer 9

1. L&D providers first conduct a self-evaluation of their L&D practices, procedures and processes against QCTO or professional body
2. second step = arrange a peer evaluation by external quality reviewers
3. prepare for and undergo a formal quality audit by QCTO or relevant standards body

Question 10

activities included in compliance evaluation step 3

Answer 10

• clarifying and describing customer expectations and needs
• ensure required resources are available for design, delivery and management
• ensure providers, practitioners, assessors and moderators have necessary skills, knowledge and motivation
• ensure quality assurance evaluation systems are in place to monitor design, delivery, management and evaluation of qualifications
• ensure that the independent auditing and monitoring of quality and feedback systems by providers of learning and learning programs as well as other stakeholders

Question 11

value added evaluation

Answer 11

• concerned with organisation’s bottom line
• measure cost-effectiveness of L&D interventions
• involve continuing, expanding or eliminating learning program

Question 12

evaluation criteria used in value added evaluation

Answer 12

• learning programme design
• training delivery
• competence
• transfer of learning
• impact on the performance of the organisation

Question 13

• learning programme design

Answer 13

design evaluated in terms of
content, training methods
and the physical design of the
curriculum and learning
materials

does not indicate if training method is effective

if content is valid and if training program is conducted by qualified proffesional - learning programme = succesful

Question 14

learning intervention delivery

Answer 14

evaluation of the quality

based on learner satisfaction and sound learning facilitation process

evaluation indicated the administrative and support process related to program

Question 15

competence

Answer 15

relates to the quality of the assessment process, methods and instruments

to what extent is outcomes achieved

main criterion is if learner demonstrated mastery of learning outcome

Question 16

transfer of learning

Answer 16

extent to which acquired competencies are transferred to the workplace is evaluated

does the workplace support the transfer of competence acquired during learning process

Question 17

impact on the performance of the organisation

Answer 17

how learning programme and accompany transfer of K,S,A,B affect performance of organisation

Question 18

measuring value added in L&D

Answer 18

Measurement focuses staff on important issues. A measurement system is a management decision-making tool that helps to prioritise tasks. Measurement also shows L&D
professionals the return on investment in a learning programme

Measurement clarifies expectations. Once the objectives for L&D interventions are set in terms of cost, time, quality, quantity and stakeholder satisfaction, HRD staff and
L&D professionals understand what is expected of them

Measurement involves, encourages and fosters creativity. Once a measurement system is in place, staff tend to compete to meet or exceed the objectives

Measurement brings the HRD function closer to departments. The L&D measurement
system should include factors that relate to quality, productivity, services and
profitability within the organisation.

Measurement improves HRD management and control. If L&D professionals and
managers measure the value added by L&D initiatives, they can manage it. If they can
manage it, they can improve it

Question 19

criteria for value-added
(three measures of training)

Answer 19

• cost
• change
• impact

Question 20

learning intervention evaluaition

Answer 20

Process to determine whether the learning intervention has achieved its goals in the most effective & efficient manner possible

Information to influence decisions about learning interventions & performance solutions to improve the organisation as a whole

Continuous process

Systematic process of making judgements ito the quality of the programme wrt 1)effectiveness & 2)efficiency of the T&D intervention

Measurement tools need to supply
1) Valid data ( the accuracy of a measure)
2) Reliable data ( the consistency of a measure)

Question 21

advantages of learning evaluation

Answer 21

ROI

Feedback system to improve the design

Improves transfer of learning

Improves the credibility of learning programmes

Measures difference between training need & what has been achieved

Question 22

predictive evaluation

Answer 22

an alternate and integrated approach to evaluation that
provides data to management which focuses on predicting the success of training/learning
intervention in three areas;

Question 23

three areas of predictive evaluation

Answer 23

(1) intention – examining whether the participants’
goals and beliefs are aligned with anticipated goals upon completion of the programme;

(2) adoption of behaviours – examining the degree to which the training has been implemented on the job and successfully incorporated into the work behaviours of the participants; and

(3) impact on business results – measuring to see if success has been achieved.

Question 24

steps followed in predictive evaluation

Answer 24

1. Choose the course to be evaluated, either a new course design or an existing course.
2. Review this comprehensively: understand what it is, what it is supposed to achieve, processes followed to deliver the content, the business issues it addresses, who attends, how this takes place, nature of pre-course preparations, type of post-course support mechanisms (if any), the course sponsors, and what they consider to be the purpose of the course.
3. Constitute a committee to predict the value of the training, create an impact matrix
   (the key features the programme is expected to impact) and present these to the key decision-makers;
4. Evaluate intention by monitoring the course during the pilot and following sessions,
   and make adjustments where necessary.
5. Evaluate adoption, that is, the degree to which participants have adopted the changes
   in behaviours promulgated by the course.
6. Evaluate impact by determining the degree to which the programme has impacted on
   the business results of the organisation

Question 25

training evaluation process

Answer 25

formal, professional evaluation of learning program - is systematic

relies on skills:
- planning skills
- conducting evaluation
- communicating outcomes

Question 26

krikpatricks model

Answer 26

level 1: reaction

level 2: learning

level 3: transfer

level 4: results

Question 27

why organisations avoid doing evaluations

Answer 27

• Barriers to evaluation and problems experienced with the evaluation process.
• Top management may not emphasise the importance of evaluation and may simply
  accept that training is valuable and effective.
• Managers of HRD departments may not have the necessary skills to conduct evaluations;
  they avoid this aspect of the HRD process.
• There may be uncertainty about what exactly should be evaluated. Owing to the wide
  range of models, each with a different emphasis, it is often difficult to decide on what
  to evaluate.
• Evaluation may be viewed as risky and expensive. It is often felt that the costs outweigh
  the benefits. There may also be a fear that an evaluation may bring negative attention to
  the HRD department

Question 28

number of issues that cause organisations to do no evaluations of superficial evaluations

Answer 28

• Too many models and theories. There are many theories and models for evaluation. This is confusing, as the different models and theories focus on a wide variety of issues.
• The complexity of models. Models and theories tend to be complex and contain many
  variables. This makes it difficult for the average L&D professional to use them.
• A general lack of understanding about evaluation.
• The lack of research skills. Effective evaluation involves a research process. This implies
  that L&D professionals should have research skills; this is not always the case.
• There is difficulty in identifying the impact of training on specific variables. It is often difficult to identify specific variables or the specific impact that a training intervention has on an organisation.
• Evaluation is considered to be a post-programme activity. Most evaluation focuses on the
  end results of training programmes, rather than on the process.
• Managers do not see the long-term advantages. Evaluation is often aimed at individual
  programmes and interventions, rather than at the overall training and development
  functions.
• There is little support from the main stakeholders. Managers and other stakeholders often
  see evaluation as a ‘nice-to-have’ rather than a ‘must-have’.
• Evaluation is not focused on management needs. Evaluation data often focus on the
  learners’ needs rather than on those of management.
• Evaluation data are not used appropriately. Evaluation results are not used at all, do not
  reach the appropriate stakeholders, are not used to bring about improvements, or are used for political or disciplinary purposes.
• Inconsistent use. Evaluation will not be taken seriously if it is used in an inconsistent way
  across different learning programmes.
• No clear standards. No consistent standards exist for evaluation in terms of the process, methods and techniques.
• Lack of sustainability. Evaluations tend to be short-term processes aimed at specific
  goals, rather than at strategic long-term processes