Unit 2 Assessments and interventions Flashcards
Not common in OBM literature. May be due to :
• interventions are effective without Assessment,
* Rule-governed behavior * OBM is concerned with “increasing”behavior vs. decreasing behavior
Functional Assessment
Records review
Informant assessments—interviews
with staff and managers
Descriptive assessment (e.g., ABC
analysis)
Experimental analysis (structural,
functional analysis
Types of Functional Assessments
Planning documents
Organization chart
Industry productivity standards
Industry conditions
Competitors
Customer list
Product/services
Employee handbook
Job descriptions
Records Review: Historical Data -OBM
An analysis of events that precede
and follow a pinpoint
Conduct for both the undesired and
desired pinpoint
Two basic formats
The data sheet can be open-ended
or contain antecedents and
consequences common to the
performance.
Open ended requires more skilled
data collectors
A-B-C Analysis
Must observe behavior Occurring- problematic if major concern is a behavior that Rarely occurs.
Time consuming
targets Results
What makes A-B-C difficult?
Very Little in the literature.
Therrein,systematically varied five Antecedents in a restaurant to assess for influence on customer greeting;
Two of the five that occasioned high levels of greeting were combined into an intervention that increased customer greeting. Note: To achieve a satisfactory increase a final feedback condition was necessary.
For experimental analysis ) in OBM
Accurate
Advantage Experimental analysis – OBM
- time consuming
- requires Frequently occurring behavior,
- requires expertise.
Some disadvantages of experimental analysis are that:
do not rely on the behavior Occurring to determine the function, but they are likely the Least accurate.
interviewing staff members,
Two types of Interviews
Structured- follows a Script
Unstructured-interviewer develops a list of topics ahead of time, but informant answers determine which direction the interview takes
We will Use a structured interview.
Make sure the staff member is Relaxed. (start with ‘grand tour question’),
Reinforce their responding
Don’t allow for Attacks
Don’t use behavioral Jargon
Give them time to answer, general to specific,
End with: ‘Is there anything else you think is important?’
Some good interviewing techniques to
An interview assessment that covers Four areas that could contribute to performance problems.
One of the most Common assessment tool in the OBM literature.
Performance Diagnostic Checklist (PDC)
- Is there a written job Description
- Have the employees received adequate Instruction about what to do?
• Are the employees aware of the Rational of the behavior as it relates to the organization?
-If yes can they Explain the rationale?
• Are there job aides visible while completing the job?
-If yes, what are they?
Are there frequently updated, challenging, and attainable Goals that employees are comfortable with?
-If yes, do employees feel these goals are Fair? If employees say that the goals are NOT fair, what reasons have they stated?
PDC-Antecedents
If equipment is required, is it reliable?
Is it in good working order?
Is it ergonomically correct?
If no, what is?
Is the equipment and environment optimally arranged in a physical sense
Are larger processes suffering from certain
incomplete tasks along the way (process
disconnects)
Are these processes arranged in a logical
manner, without unnecessary repetition
Are these processes maximally efficient?
Are there any other obstacles that are keeping the employees from completing the task?
PDC Equipment and Processes
Can the employees tell you what they are supposed to be doing and how to do it?
If yes have they mastered the task?
If fluency is necessary, are they fluent?
Can the employees physically/verbally
demonstrate completion of the task?
If yes, have they mastered the task?
If fluency is necessary, are they fluent?
Do the employees have the capacity to learn
how to complete the task
PDC-Knowledge and Skills
Are they delivered contingent on the task?
If so, provide examples of the consequences.
Are they Positive or negative
Are they Immediate , Frequent
Consistency/probability?
Are there Premack Reinforcers?
- If yes, what reinforcers are used?
Do the employees see the effects of their performance?
- if yes, How? Are they natural Arranged?
Supervisors should deliver them. Best if they are available.
- Direct or indirect Feedback
Is there performance monitoring?
- If yes, self-supervisor direct (verbal)
supervisor indirect (written)
Is there a RESPONSE EFFORT associated with the performance?
Are there other behaviors competing with the
desired performance?
PDC-Consequences
Sample uses from the literature
Franchise restaurants
Ski shops
Coffee shops
Implementation generally delivered to managers and
employees
In larger organizations, or in a business with multiple sites it can be delivered to relevant managers and
sample staff
PDC
If the PDC shows Decrements in
Antecedents…
-Keep in mind:
• role of antecedents. They don’t function independently of the consequences.
• Few examples of effective antecedent only interventions
An antecedent intervention consists of manipulations before the behavior occurs.
broadly defined as
antecedents that encourage
engaging in a task
Examples:
Smiley faces
Have you washed your hands?
Remember to clock out for breaks
Job Aides—
“Are there Job Aides Visible While
Completing the Job?”
Highly detailed set
of instructions on what is expected
Can be:
A memo
Checklist
Meeting/workshop
Task Clarification:
PDC Antecedent Interventions
-A list of activities in sequential order that need to be completed
.
-Conduct a task analysis of a task and place the tasks in order of occurrence
Found to be successful with simple and complex tasks. Perhaps especially useful for complex jobs:
Pilots
Physicians
Antecedent Interventions
Checklists
Clearly identify when to use it
Simple wording
Not too lengthy
One page
Test them out first
The Checklist Manifesto
An antecedent that describes an expected level of performance. That is, what is required by the
company
Sometimes include timelines
Employees need to know what is
expected
In a PM-designed business, it signals when reinforcement can be obtained—not that punishment is available for failing to make the goal
Goal
Difficult , yet obtainable
• Stay away from stretch goals
Under the control of the employee
Good goals
Create an ultimate goal and sub goals
Sub-goals allow for the shaping of behavior so that employees can contact reinforcement
Small reinforcers for the sub-goals
Large reinforcer for the ultimate goal
Lots of verbal praise for increasing performance
if the goal is much higher than current baseline levels
Too easy, too hard?
Watch for new problems arising from an increase in behavior (out og paper)?
quality of the performance? Consider adding qualifying statement
Increase requirements when met for several
consecutive sessions
Make sure employees have contacted reinforcement before increasing
Do not reinforce lower ones once a new, higher one has been set
Keeping an eye on your goals
Patterns will likely be different if
introducing new behavior or setting a
goal for a long-standing behavior:
- Slow and steady - Seep increase followed by leveling off - Slow increase followed by steep increase
Patterns of Increasing Behavior
Can the employees tell you what they are supposed to be doing and how to do it?
-If yes have they mastered the task?
If fluency is necessary, are they fluent?
Can the employees physically/verbally
demonstrate completion of the task?
-If yes, have they mastered the task?
If fluency is necessary, are they fluent?
Do the employees have the capacity to learn how to complete the task?
PDC-Knowledge and Skills
Training (and is a common solution to many organizational problems. Many times the problems are due to an environment without properly aligned antecedents and consequences
Training is an antecedent intervention and shouldn’t be expected to solve these types of problems.
Very expensive and time consuming
Classroom-An instructor teaches a
group of individuals typically by lectures, worksheets, tests, and occasionally role-play/rehearsal
Behavioral Skills Training (BST)-An instructor teaches skills using verbal antecedents, modeling, rehearsal, and feedback
Select the training that makes the most sense in terms of:
Number of staff to be trained
Is opportunity to practice in real-life
essential (e.g., CPR)
How often will they use the information
Types of Training
Advantages:
-Many people can be taught at once
Disadvantages
-: Questionable retention/generalization, limited ability to practice skills
Useful for:
-Large number of employees (initial training)
- A lot of information to teach (be careful)
Classroom Training- OBM
Procedures: Set clear learning objectives Deliver a pre-test Provide instruction Evaluate
Typically delivery methods includes:
Lecture
Watching videos
Internet
Conferences
Classroom
Avoid passive attendance by audience
members (i.e., pure lecture/video
provided by the instructor)
Generally, provide the rationale, and
specific aspects of the job skill from the
learning objectives
Deliver material in a succinct and
interesting manner
Consider classroom management
strategies for some groups
Consider the use of Active Student
Responding (ASR)
Instruction in Classrooms
Promoting responding in the audience during instruction
-allows the instructor to receive ongoing feedback
Types:
Response cards
Guided notes
Choral responding
ASR In the classroom
A key component appears to classroom instruction
Can be somewhat difficult to achieve in a classroom setting
Role-playing or
Evaluation of video models may allow for this
Performance Feedback-Classroom instruction
Some skills can be acted out
Have employees complete a skill as
taught and provide feedback either during or immediately after the performance
Consider using mastery criteria
Role-play
Evaluating video models performing
behavior (Williams & Gallinat, 2011)
Employees evaluate correct and incorrect elements of a performance
May be especially useful for skills that do not occur often, or skills that are difficult to role-play
While watching, employees evaluate performance with assessments tools
Typical tools include checklists, or other
evaluation tools used on the job
Provide feedback on the accuracy of recording
Vary the scenarios: different employees, situations, tools, etc.
Video Modeling
Advantage
Can be done on the job and perhaps
facilitate generalization
Disadvantage
Time-consuming
Useful for:
Employees can’t demonstrate skills
Increase fluency
BST
- Provide instruction-verbal or written
- Model the performance for employee
- Have employee engage in activity and provide immediate feedback on correct and incorrect
elements of performance - Set mastery criteria—this will depend on the behavior but consider 3 consecutive demonstrations with 100% accuracy
- Repeat modeling and rehearsal until criteria are reached
BST Procedure
A rate of accurate performance
Some jobs require it
Typically helpful with most job duties
Assists with retention (Bucklin)
Methods:
Additional practice with a time component after a skills has been learned
- time drills
-SAFMEDS
Fluency
Experts train supervisors who then
train staff
Page, Iwata, & Reid (1982)
- supervisors were trained on
evaluating teaching behaviors These supervisors trained 45 staff members . Replicated using peers to
train peers
Caregivers were trained to respond to
emergency situations (e.g., fire response)
These caregivers then taught other caregivers to respond to emergency situations
- Benefit: Trainer maintenance
This approach may be very effective, but make sure there is reinforcement programmed in for the staff trainers (i.e., it is not just extra work)
Pyramidal Training
Provide feedback!
If a behavior is not used frequently consider making opportunities available to engage in the behavior:
Ie: video tapes, simulated drills, etc
After Training
New staff members in a residential
setting with a three week pre-service
Skills required by the state were taught in a classroom
Fluency measures incorporated after mastery criteria were met
BST follow-ups were completed on the floor
On-going feedback
Pyramidal training through supervisors when new material was introduced
Created opportunities for low-occurrence behavior
Data collection
A Sample Training Program
for new Staff
They are Set by:
Looking at internal performers
Input from managers and employees
Customer requirements
Baseline levels of performance
External competitors-careful
Industry standards-careful
Goals
Are there reminders to Prompt the task at the correct time/duration?
Is the supervisor present during task completion?
PDC – Antecedents
Can the employee learn to
complete the task in a time that is
feasible in the current setting
If not, consider changing the job (can
anyone learn this) or reassign to
another position
Employee training
PDC-Equipment and Processes
If equipment is required, is it reliable?
Is it in good working order?
Is it ergonomically correct?
If no, what is?
Is the equipment and environment optimally arranged in a physical sense
PDC – equipment and processes
Examining and designing the interaction with equipment and workplaces to fit employees A person sitting at a desk typing on a computer Problems with equipment can cause long-term problems
Ergonomics
Are larger processes suffering from
certain incomplete tasks along the way
(process disconnects)
Are these processes arranged in a logical manner, without unnecessary repetition
Are these processes maximally efficient?
Are there any other obstacles that are keeping the employees from completing the task?
PDC-Equipment and Processes
As defined by Rummler and Brach
1995):
“A series of steps designed to produce a product or service”
Results in products/services for customers, products for the business to operate, or actions that support the other processes.
Many flow through Departments. They are not Departments (Production department).
Should have a goal and formal structure, but is not
always the case, Otherwise problems like repetition and disconnects are possible.
**This is the REALM of behavioral systems analysis
Process
A change in the environment that follows a response that maintains or increases responding in the future under similar environmental conditions
This is NOT what we will tell managers
- A better definition: An event that follows a behavior that makes it more likely that the behavior will occur again - Focus on how important \_\_\_\_\_\_is, and how nothing else will work long-term without it
Reinforcement
The general concept typically rings
true with a few basic examples but reinforcers are individualized
Strange things may reinforce behavior
***We at the EFFECT, NOT the stimulus
Reinforcement in Detail
When we enforcement is used in organizations it is typically in the form of…
Negative reinforcement.
“Making the opportunity to engage in
a behavior that occurs at a relatively
high free operant rate contingent upon the occurrence of low-frequency behavior will function as reinforcement for the low-frequency
behavior”Cooper,
The Premack Principle
In contrast to reinforcement, a _______ is something that is delivered long after a behavior has occurred. Many managers get reinforcement confused with non-contingent stimulus delivery.
Attempts at reinforcement are actually just NCR because they are not contingent upon behavior (bonus systems)
NCR will suppress behavior
Reward
Insincere
Too thin
Assumptions of value
Too delayed
Too general
Non-contingent
Reaction from employee
Why “Reinforcement” Fails
Reinforcement Effectiveness
Reinforcement has the greatest effect when it meets three general requirements:
- Quickly follows behavior
- Delivered frequently when behavior is
completed - Consistently delivered
Chase and Smith (1994)
- It’s an actual Reinforcer
- Immediate
- Frequent
- Contingent
Reinforcer Effectiveness
Shaping
Shape desired behavior through
reinforcement of______
Reserve reserve large reinforcers for ultimate outcomes
Be careful about reinforcing an old
approximation
Remember, no freebies (you really
want to work for things)
..,sub-goals
Praise
Small Tangible (paired with praise)
Public Recognition
Leaving early for a day
These all mean so much more when personalized
Sample reinforcement for meeting
sub-goals
Large group events to recognize the
achievement of the group:
Celebrations!
Sample reinforcement for meeting
ULTIMATE goals:
Allow opportunities for management to
hear from the employees
Management should not talk and congratulate, but encourage employees to describe how they arrived here (experience accomplishments)
Tangibles used only to cement the experience
From Daniels (2000) Celebrations should:
If you ask more of employees, you
should increase reinforcement:
Short-term initiatives
Temporary change in schedules New regulations Troubleshooting
Be Sensitive to Times of
Deprivation
“Information about behavior or performance that allows a person to change his/her behavior
Natural in the environment…but, many employees work for months on end without any information about their performance
Being told they are not doing well may come as a shock
Feedback
Braksick (2007):
Positive feedback: encourage desired behavior
Constructive feedback: increase appropriate behavior while discouraging unproductive behavior
NOT A PUNISHMENT PROCEDURE PER SE
Don’t mix the two types in one conversation
Deliver at least four times as much positive feedback than constructive feedback
Types of Feedback
Specific to the performance
Delivered as soon after the behavior as possible
individualized
Delivered by a person in-charge
Easily understood
Graphed
Positive Feedback Characteristics
likely to have the
greatest impact when:
You have been paired with the delivery of reinforcement
Delivered fairly and equally
Based on data
Positive Feedback
May need to give the first instance of
feedback in private (or maybe more)
Be sincere and enthusiastic: we
should be excited to see
improvements in our employees
Be sure to congratulate
improvements (never stop shaping)
Positive Feedback
Relied upon too heavily in many
organizations
Use sparingly to decrease inefficient
behavior
Lots of positive feedback will pair
managers with reinforcers and allow
for easier implementation of
constructive feedback
Constructive Feedback
Done in private
Soon after the behavior
Describe the desired performance
Talk specifically about behavior, nothing else
Use ‘I statements’
Don’t do it when angry
Braksick,
Characteristics of good constructive
feedback:
Describe both the undesirable and
desirable behaviors
An example of constructive
“I would appreciate if you would use the safety guard on the saw. Without the guard the saw is much more dangerous. Please allow the guard to fall in-place when you start it.” “I’m worried that by talking about our clients financial numbers out loud in meetings the information could leak out. Please remember our privacy policy states that we should not use names when discussing financial information in open forums. In the future use a numeric code to represent the client
Constructive Feedback
The employee can summarize what they should be doing instead
The employee should acknowledge that what they were doing was a problem
Agree with you on a solution
should prompt desired responding on the next
occasion
You must watch and deliver positive feedback
Maintain the dense schedule of positive feedback
If you have done constructive
feedback well:
Medication administration and the
supervisor yelling
This is likely punishment and may evoke countercontrol, emotional responding or other punishment side-effects
“Please explain what you did wrong
Clocking in late and the end of the pay period feedback
Non-examples of Feedback
Don’t assume a quick explanation
will solve the problem
Some key variables typically missing when first learning:
being present
variety
enthusiasm
evaluating the employee’s behavior
Teaching Others to Reinforce
Work process consequences (Braksick, 2007)
Look for opportunities to decrease response effort required in a job
Make it easier, have employees earn supervisor giving them a break
Other Reinforcement Procedures
The process of collecting data on one’s own behavior
Can change behavior in the desired direction
Bracket , Reid, & Green (2007) found it increased adequate performance in employees and results sustained over time. Stated, REACTIVITY was
caused by the measurement system itself
Self-Monitoring
Improvements were modest but the
intervention may prove easy to deliver
Authors ( Brackett) cautioned these results
may not be durable
Augment with reinforcement of data collection, but be careful with tying self-collected data with reinforcement
Self-Monitoring
Created a measure from 0 to 100
based upon three pinpoints: Time to complete repairs, Quality of repair, Check-ins
Baseline data is collected and goals
are set
Four Line Mechanics
PDC is implemented with the supervisor and all mechanics
Direct observation of all mechanics is conducted
PDC demonstrates no clear goals for check-ins, very little feedback for any target
Direct Observation suggests a more efficient method used by the exemplar
Assessment
BST for three of four mechanics
Goals set with mechanic’s input
Publicly posted group data
Feedback delivered by supervisor
Group celebration set for reaching ultimate goal
Intervention
Extra line was added without the need to hire or replace mechanics
Mechanics are satisfied with the new
management strategy
Downtime of existing lines decreased
Impact
From the basic literature, so be careful with employees
Fixed ratio-FR-(all or none-post reinforcement pause)
Variable ratio-VR-(steady high rates
Fixed Interval-FI-(scallops)
Variable Interval-VI-(steady rate of responding-low to moderate)
Little research on complex schedules in humans and work environments
Basic Schedule Review
Time Based Schedule-FT, VT-Usually
decreases behavior or maintains very low rates of behavior
Unfortunately, time-based schedules dominate most work environments paychecks, bonuses, traditional business programs)
Basic Schedule Review
Schedule to teach new behavior or when behavior has been placed on extinction
FR1
schedules allow for stretching and may be good for maintenance
Ratio
In general these two schedules increase performance above hourly pay (Dickinson & Poling, 1996)
After the goal has been reached VR schedules may be useful to maintain responding
, FR and VR
Will likely produce higher rates of responding
VR
As in clinical application, do it slowly
after behavior has been established
Watch the data
Watch for signs of extinction
Thinning schedules
Be careful with interval schedules
Watch the data when you set the time interval and remember VR schedules will likely produce higher rates of responding
Applying Schedules (continued
a verbal description of a contingency
Rule:
Behavior controlled by the rule rather than the contingency it describes
Bottom line we do not have to experience the contingencies.
Rule-governed behavior
Most contingencies in office settings are too REMOTE to directly change behavior
Rules in the Office
vary in their effectiveness in controlling behavior
The varying effectiveness may be due to:
-
-Learner history of rule-following
- Characteristics of the rules
Rules
Pelaez and Moreno (1998)
vary in whether they are:
simple or complex
accurate or inaccurate
vary in terms of source
Implicitly or explicitly stated
Rules
Pelaez Compared rules delivered by rumor (implicit) to a memo (explicit) in a store setting:
Implicit condition: a non-specific statement concerning performance was delivered by a manager to one employee
Explicit -A memo from the CEO with a
clear contingency
Both types of rules were effective in changing behavior (e.g., customer greeting)
Explicit rules produced greater change that also lasted longer than the change produced by the implicit rule
Implicit versus Explicit
Squires Examined the effects of rules on ‘sharps’ compliance. Two rules in relation to goals were compared:
- Praise: can be obtained the following day for meeting a daily goal
2, Reprimand: management will be angry if daily goal is not met
Both were effective in increasing compliance with two participants when compared to baseline conditions (no rules)
Delayed contingencies were delivered on the following day, but it is unclear if this influenced behavior
Praise goals were preferred by both participants
Rules-OBM
An area ripe for research
Rules can quickly change behavior—
compare to direct-acting contingencies
Rule-governed behavior does not appear to sustain over time without contingencies
Positive rules are preferred
Indirect rules can change behavior.
that means:
Use explicit rules (make it very clear what is expected of employees)
be careful of rumors
Summary of Rules
“Continuing of performance after it
was first established”
(Malott & Suarez,
Levels of responding reached during
intervention that maintain after the intervention has been terminated
(Boyce & Geller, 2001)
AKA, Maintenance
Duration of the interventions vary
considerably in the literature (Sigurdsson & Austin, 2006)
Once a consultant/researcher withdraws, the gains may not last
Institutionalization
3 Barriers to Lasting Change
Involvement in design
-Involving employees in selection of
intervention(s)
-Training of internal staff
-Training staff to implement parts of the
intervention
Formal data collection system
- Employee (typically management)
become responsible for collecting data
- sometimes self-monitoring
Formal system of dispensing consequences
-Employees (typically managers) become
responsible for delivering consequences
Use naturally occurring contingencies
Relationship building on the front end
Don’t rely on a single person
Consequences at all levels
Lasting Change Strategies
From Sigurdsson & Austin (2006)
Be careful of reaching your goals)
Train management before recommending thinning
Thin the schedule of reinforcement slowly (but don’t stop)
Schedules of Reinforcement
Revisited
Punishment from a behavior analytic
perspective.
Collective feedback increase the behavior we want and decreases behavior we don’t want.
Punishment only decreases behavior.
Problem behavior – punishment in organizations
Many problems in OBM are teaching or
acceleration problems
What if there is unwanted behavior?
Examples:
-Wearing safety gear improperly
-Fighting with supervisors
-Yelling at a customer
We should have an operational definition
and data collection
Problem behavior in organizations
Skipping meetings Negative statements Leaving early Spreading rumors Smoking on grounds Behavior targeted by corrective feedback that does not respond
Nuisance Behaviors
Welding without eye protection
Smoking next to flammable materials
Threatening a colleague
Stealing from the business
Engaging in discrimination
Severe Behaviors
Double check the contingencies
Replace the behavior with something socially appropriate
Start with reinforcement based procedures for appropriate behavior
Limit reinforcement for inappropriate behavior
If this doesn’t work consider punishment procedures(?)
Nuisance Behaviors -OBM
A lack of behavior (i.e., nuisance
behavior and the employee is not
engaging in any appropriate behavior
to reinforce) then consider,,,,
Once behavior begins to occur reinforce it
Fade out negative reinforcement and maintain with positive reinforcement
Monitor data
Negative reinforcement to evoke behavior (e.g., you will be suspended on __ if you do not complete ___ )
Implement punishment procedures: Industry and job specific (observe labor laws)
Examples: loss of earned benefits, demotions, suspension, verbal/written reprimand
Make sure no matter what you
choose, you are collecting data
Severe
Don’t threaten punishment just
implement
Punish the behavior not the person (leave emotions out)
Punish immediately
Punish every time
Make it clear what is expected and reinforce the occurrence
Continue to deliver reinforcement for appropriate behavior
Punish in private
Be consistent
Don’t mix punishment and reinforcement
Use an intense punisher
Guidelines
From (Daniels and Daniels, 2004)
Avoidance
Emotional responding
Counter control
Become a signal for Punishment
Side-effects of Punishment
Termination
Consider re-assignment/termination for behavior that resists multiple change efforts—you are not able to compete with other reinforcers
really severe behavior
Since there are few direct-acting contingencies, most of what controls behavior are…..
RULES
We need to explain this- office
When rules do not control behavior….
Interventions are required
Rules tend to fail when they are too…
.DELAYED , inaccurate, or cumulatively significant (Malott, 1992)
Our interventions are often rules in and of themselves and they are often to create ….
clear, accurate, important contingencies
Additional practice with a time component after a skills has been learned
- time drills - SAFMEDS
Fluency
condition: a non-specific statement concerning performance was delivered by a manager to one employee
Implicit Rule
-A memo from the CEO with a
clear contingency
Explicit
Anything that is occurring regularly is being ____
Many things that we do are being maintained by natural contingencies, this is true in and out of organizations
Reinforced
Being present
Variety
Enthusiasm
Evaluating the employees behavior
Some key variables typically missed one for us learning how to reinforce
Work process consequences ( Naturally occurring consequences)
Look for opportunities to decrease the response effort required in a job. Make it easier and have employees on their supervisor giving them a break
Other reinforcement procedures
Too delayed
Inaccurate
Cumulatively significant
Rules tend to fail when they are…
Our interventions are often rules in and out themselves and their often to create…
Clear, accurate, important contingencies.
Duration of the Interventions varies considerably
Once a consultant/researcher with draws, gains are lost
Institutionalization
Barriers to lasting change
Teaching or acceleration problems
Problems in OBM
