Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

Exam Prep > Evaluating Impact > Flashcards

Evaluating Impact Flashcards

1
Q

What does a TD professional’s efficacy depend on?

A

Demonstrating the value of the TD investment

Evaluation is one way to document whether the investment in TD achieved its desired outcomes

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

What changes can be made to a training program based on evaluation data?

A

Program additions, changes, or deletions

Prioritize needs at the org level

Decisions to shift financial and other resources

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

What are 7 purposes of evaluation?

A
  • determining business impact, cost-benefit ratio, and the ROI for the solution
  • determining whether the objectives of the solution were met and how well
  • assessing the effectiveness and appropriateness of the content and instructional strategies
  • reinforcing learning concepts by using a test or similar performance assessment
  • providing feedback to the facilitator
  • providing feedback to participants about what they learned
  • assessing the on-the-job environment to support learning retention.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

When will evaluation be most beneficial?

A

When TD professionals incorporate them into the design of the solution from the beginning and relate them to LOs

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

What are some key benefits of evaluation to the TD function?

A
  • secures client support and confidence
  • monitors whether program results are consistent with org analysis and needs assessment
  • validate performance gap and learner needs
  • help determine whether training is the solution to a performance gap
    helps mgt team meet objective and TD to be a true partner in the org’s success
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

What is the first step/input to designing and developing a solution, creating an eval plan, and initiating measurement?

A

Needs assessment

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

Ralph Tyler’s Goal-Attainment Method

A

a process for evaluating the success of a curriculum that first identifies objectives, “the ends of instruction.”

The content is then identified to address the objectives, and the various instructional elements and delivery mechanisms are then designed to assist learners in obtaining the objectives.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

What are the 4 questions that must be answered in the Goal-Attainment Method? Hint: They are progressive through the instructional design process.

A
  • What objectives should learners achieve?
  • What learning activities will assist learners to achieve these objectives?
  • How should the curriculum be organized?
  • How should learner achievement be evaluated?
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

Measure

A

a standard used to evaluate the degree or quality of the results of a solution

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Measurement

A

the process of quantifying assessment data and providing the necessary information to make sound decisions about an issue or situation

Measurements define or quantify specific attributes of an observation.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

What is the role of the assessment process in evaluation planning?

A

Using assessment data, TD professionals should identify the desired outcomes before designing an evaluation plan.

To do this they gather, summarize, and interpret the data generated by the assessment process to determine the root cause, which could be the process, a lack of resources, a lack of information, a lack of motivation, poor health, or a need for knowledge, skills, or affective (attitude) learning (KSAs).

Based on these results, TD professionals will determine the best solutions and write objectives.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

What are three factors to consider when selecting the best measurement process?

A
  • the nature of the solution
  • the characteristics of the learners
  • the outcome focus
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

Similar to Ralph Tyler’s method, most evaluation process models follow the same process. What are the 5 major steps in the process?

A
  • Use assessment data to identify evaluation outcomes and goals
  • Develop an evaluation design and strategy
  • Select and construct measurement tools
  • Analyze data
  • Report data
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

What is a program?

A

a set of resources and activities directed toward one or more common goals, typically under the direction of a single manager or management team

It may be limited to one set, or complex across multiple sites/levels, and it may be developed by in-house, public, nonprofit, or private sectors

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

Program evaluation

A

the systematic assessment of program results and, if possible, the assessment of how the program caused them

Includes ongoing monitoring of programs as well as one-time studies of program processes or effects

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

What are 2 categories of program evaluation?

A

Program evaluation: assesses the effect of a learning program

Learning transfer evaluation: measures the learner’s ability to use what they’ve learned on the job

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

Formative Evaluation

A
  • occurs throughout the design of the TD solution
  • improves the draft learning program to better achieve results
  • use the results immediately to revise the program and make it more effective
  • goal during formative eval: program should be understandable, accurate, current, and functional
  • includes activities like pilot testing, beta testing, technical reviews with SMEs, production reviews, stakeholder reviews
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

Summative Evaluation

A
  • occurs after the TD solution has been delivered
  • focuses on the results or impact of the solution to provide evidence about the program’s value
  • could measure participant reactions, the effect on organizational goals, the initiative’s costs, and stakeholder expectations
  • includes activities like standardized tests, participant reaction forms, stakeholder satisfaction surveys, and final return on investment
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

Kirkpatrick’s 4 Levels of Evaluation

A

Reaction:
Knowledge: learner’s mastery of content; knowledge/skill acquisition
Behavior Change on the Job: comprehensive, continuous monitoring of performance
Business Impact: targeted outcomes or program’s org effect;

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

Measurement methods for Level 1?

A
  • Questionnaire/surveys
  • Individual responses in class
  • Follow-up interviews
  • Observation checklists

Could also include word of mouth feedback

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

Measurement methods for Level 2?

A
  • Knowledge tests, which may be paper-and-pencil tests; oral questions and answers
  • Performance tests, role plays, case studies with evaluation or feedback sheets
  • Monitored skill demonstrations
  • Checklists
  • Product tests
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

Measurement methods for Level 3?

A
  • Performance records
  • Performance contracts
  • Action plans
  • Interviews
  • Surveys or questionnaires
  • Direct observation with checklists
  • Supervisor interviews
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

Measurement methods for Level 4?

A
  • Action plans
  • Interviews
  • Questionnaires
  • Focus groups
  • Performance contracts
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

When is Level 1 measured?

A

During the program (end of day)
Right after program completion

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
25
Q

When is Level 2 measured?

A

Before, during, and after the program (measure before to create a baseline and during the program as a knowledge/progress check)

26
Q

When is Level 3 measured?

A

A few weeks to three months after the program

27
Q

When is Level 4 measured?

A

Three months to one year after the program

28
Q

What are sample measures for Level 1?

A
  • Reactions
  • Pace and sequence
  • Relevance (content)
  • Instrument strategies
  • Interaction
  • Facilitator’s style
  • Level of discussion
  • Objectives met
  • Environment
  • Knowledge of facilitator
  • Participant interaction
  • Registration process
29
Q

What are sample measures for Level 4?

A
  • Productivity and production measures
  • Cost or expense measures
  • Financial or direct output measures
  • Employee turnover
  • Engagement ratings
30
Q

How is ROI calculated?

A

By converting business impact measures to monetary value and comparing them with program costs, which requires placing a value on each unit of data connected with the program

31
Q

6 Data Types of Phillips ROI Method

A

Level 1: reaction and planned action
Level 2: learning
Level 3: application and implementation
Level 4: business impact
Level 5: return on investment
Intangible measures

32
Q

Data sources/inputs for ROI

A
  • standard values, which are those that are already accepted in the organization
  • historical costs, which represent what the measures being converted have cost the organization in the past (such as the cost of an unexpected absence) - extant data
  • links to other measures that have already been converted to monetary value; for example, placing value on customer and employee satisfaction
  • input from internal or external experts on a particular measure
  • participant estimates
  • supervisor and manager estimates
  • TD staff estimates
33
Q

How does Phillips ROI Method address the need to isolate the effects of training, answering the question, “How do we know our training caused the results?”

A

Using a control group, which is not always possible

Trend-line analysis, forecasting methods, use of experts

34
Q

Brinkerhoff Success Case Method (SCM)

A

identifies the most and least successful cases in a program and examines them in detail

a useful approach for documenting stories of impact that can then be shared with stakeholders and used to develop an understanding of the factors that enhanced or impeded program success

35
Q

5 Steps of the Success Case Method

A
  1. plan a successful case study
  2. create an “impact model” that defines what success should look like
  3. design and implement a survey to search for best and worst cases
  4. interview and document success cases
  5. communicate findings, conclusions, and recommendations
36
Q

Balanced Scorecard Approach

A

A way for orgs to evaluate effectiveness with more than financial measures

37
Q

What are the 4 perspectives of the Balanced Scorecard Approach?

A
  • The customer perspective: Did the solution, initiative, or practice meet the customer’s need or expectation?
  • The innovation and learning perspective: Did users gain the needed skills or knowledge?
  • The internal business perspective: Did the solution, initiative, or practice have an effect back on the job?
  • The financial perspective: Did the solution, initiative, or practice have a financial payoff?
38
Q

What a meta-evaluation?

A

An evaluation of an evaluation

39
Q

What are other examples of evaluation methods? (10)

A
  • Cost-benefit analysis: measures monetary gains and losses
  • Culturally responsive evaluation: a holistic framework for centering evaluation in culture
  • Developmental evaluation: useful in complex or uncertain environments such as innovation, radical program redesign, or crises
  • HPT evaluation model: an analysis of present and desired levels of performance, identifies the causes for the gap, and offers a range of solutions (human performance technology)
  • Lean six sigma: combines eliminating waste with lean practices and quality improvement from six sigma
  • Predictive learning analytics: a systematic methodology for predicting learner outcomes and actions
  • Return on expectations: Kirkpatrick approach in which stakeholders identify the TD value in terms of how it contributes to their goals
  • Robinson’s training for impact: helps achieve organizational goals, gives people the skills and knowledge required, and produces measurable results that can be traced on the job.
  • Six sigma: a disciplined, data approach to eliminate defects in processes
  • Total quality management (TQM): focuses on improving quality and productivity
40
Q

What are the steps in an evaluation process? (8)

A
  1. Conduct a needs assessment to measure business, performance, learning, and learner needs. Utilize data collection methods for the needs assessment.
  2. Analyze needs assessment data to identify solution objectives and develop proposed outcomes.
  3. Present the results of the needs assessment. This includes organizing information and discussing interpretations from the analysis, which will show stakeholders how the proposed objectives/outcomes will solve the identified problem and meet management’s request/need.
  4. Determine the proposed solution. This should be tied to the business need and use business metrics to measure the ultimate objective. May not be training.
  5. Determine which results to measure and how to measure them to show whether outcomes were achieved and objectives met. (Identify measures and methods.)
  6. Design and develop evaluation strategy/methodology. (Select and construct measurement tools.)
  7. Conduct evaluation and engage in data analysis.
  8. Report data.
41
Q

What 4 things does a needs assessment identify?

How does an eval strategy convert these into eval measures?

A

Business, performance, learning, and learner needs

Business needs = examining results of training initiatives (Level 4)
Performance needs = observing learner behavior and evaluating transfer of learning on the job (Level 3)
Learning needs = Assessing knowledge and skill mastery immediately after the training (Level 2)
Learner needs = collecting data on learner reaction (Level 1)

42
Q

Examples of data collection methods for a needs assessment

A
  • Surveys and questionnaires
  • Analytics from technology platforms
  • Examinations, assessments, and tests
  • Self-evaluation
  • Simulations and observations
  • Archival or extant data
43
Q

What is included in the final report of the needs assessment?

A
  • defines and documents the findings
  • summarizes the problem statement, analyses used to determine need, and proposed module design
  • discusses how findings relate to overall org strategy/goals
  • shows how proposed change/plan will benefit the org

Results should:
- explain current state, outline org goals, and show how proposal will reach goals
- provide baseline data to determine gap and measure whether progress is made
- envision future state by describing vision and gathering hard metrics/data to measure
- explain gap between current and future states by including root causes
- identify issues that will support or deter success
- describe most appropriate solutions for closing the gap

44
Q

Business drivers are also an input into establishing the evaluation measures. What are business drivers? What are internal and external business drivers?

A

Business drivers are the internal and external forces that direct an organization’s strategy, goals, business needs, and performance needs

Internal business drivers are generated by internal decisions and can include technology; a change in system, process, or key policy; shareholder or financial drivers; and new product generation.

External business drivers can include economics, human resources, government, public perception, and market or customer drivers.

45
Q

What is the value in a TD professional having knowledge in the analysis of data?

A

allows TD professionals to explore data comprehensively, inform decision making, and determine the value of the outcome metrics

46
Q

When might measuring business results be especially important?

A

When there is a high cost or investment involved, such as when designing large or expensive programs or deciding whether an expansive company-wide initiative should be implemented

47
Q

Defining the business need during the assessment and planning process determines what 2 things?

A
  • What measures should be quantified/measured
  • What questions need to be answered during level 4 evaluation
48
Q

Reporting ROI analysis should be paired with WHAT in order to improve programs and identify tangible benefits?

A

ROI analysis should be reported alongside HOW the ROI was achieved.

49
Q

What is the Chain of Impact?

A

A Chain of Impact occurs as people are involved in programs and projects.

The Chain of Impact introduces a step in the ROI Methodology between measuring application/behavior outcomes and measuring business impact where analysis isolates program impacts in order to connect the program outcomes to business results.

Reaction -> Learning -> Behavior/Application -> Isolate Program Impacts -> Business Impacts -> ROI -> Intangible Benefits, which is linked to Business Impacts

50
Q

What is benefit-cost ratio? (BCR)

A

the ratio of the benefits of a project or proposal (expressed in monetary terms) relative to its costs (also expressed in monetary terms); summarize the overall monetary value of a project or proposal

51
Q

What is the difference between BCR and ROI?

A

BCR measures actual results, while the ROI is often forecasted before investing in a program or project

How it’s displayed is also different:
- BCR of 1.0 = every dollar invested, a dollar is returned
- 0% ROI = the program broke even

  • BCR of 2.0 = every dollar invested, two dollars are returned
  • 100% ROI = the return doubles the investment, meaning the return is 100% = the investment
52
Q

Cost-benefit analysis

A

a framework for considering a range of costs and benefits in monetary terms; provides several analytical tools to quantify and monetize various hard data and soft data measures to reflect the value of learning

53
Q

What costs should be included in cost-benefit analysis?

A

all costs:
- including time away from work, travel, materials, and overhead costs
- lost opportunity costs, such as failure rates, downtime, re-work, and accident costs

benefits-solution savings: things such as increased productivity, new accounts generated, reduced overtime, items sold, and on-time shipments

54
Q

How is cost-benefit analysis helpful?

A
  • showing overall benefits
  • comparing relative magnitude of specific costs and benefits
  • prioritizing alternatives
55
Q

Utility Analysis

A

Utility analysis measures a program’s economic contribution according to how effective the program was in identifying and modifying behavior and, therefore, the future service contribution of employees

56
Q

Difference between ROI and utility analysis

A

utility analysis places a value on new skills, disregarding the actual effect of those skills in the workplace

57
Q

Brogden utility estimation equation

A

U = N × T × dt × SDy – c

Where:
U = total change in utility in dollars after implementing a learning solution
N = number of employees developed
T = duration, in number of years, of a program’s effect on performance (e.g., tenure of trained employee; number of years trained employees will use new system or approach)
dt = true difference in job performance between the average trained and average untrained employees in units of standard deviation (ie, net gain of job performance improvement)
SDy = standard deviation of job performance of the untrained group in dollars (ie, monetary value of job performance improvement)
c = cost of training per employee

58
Q

How might forecasting be used to define outcome metrics?

A

a process by which TD professionals can make a prediction of the future based on current and past data, usually by analyzing trends

derived from trend data plus additional influences that may impact the results to create a statement about the future

59
Q

When would forecasting be challenging?

A

if more than one additional influence needs to be considered, because that may require more sophisticated statistical techniques and software for multiple variable analyses

60
Q

ROI doesn’t need to be calculated for all projects. When is ROI most useful?

A

For programs with a long lifecycle, programs that are important to meeting operational goals, and programs closely linked to strategic initiatives

Exam Prep (39 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions