Instructor's notes Flashcards
Targeting a student’s attitude in class would be an example of a(n) OBJECTIVE or SUBJECTIVE measure.
-Subjective
Which would be an objective measure to replace the student’s “attitude” in class?
A. How happy the student is in class
B. How many times the student makes comments
like “I hate this”
C. The student’s tone of voice and agitation level
B. How many times the student makes comments
like “I hate this”
Behavior-analytic assessments will never include subjective measures.
A. True
B. False
B. False
- Behavior-analytic assessments will always include objective measures but may supplement with subjective, particularly in measures of social validity
- Not sufficient, but necessary (Baer et al., 1987)
When behavior analysts attempt to identify if the participants, caregivers, and other consumers consider the intervention acceptable, we are assessing the social validity of the:
A. Goals
B. Procedures
C. Results
B. Procedures
Tucker is assessing the social validity of the goals selected for his client. He decides it is important to include the teacher, his client’s parent, and a sibling. Which step for conducting a social validity assessment does this describe?
A. Collect and summarize the data
B. Identify what to assess
C. Identify who will be conducting the assessment
D. Identify who will be evaluating
D. Identify who will be evaluating
Social Validity OF Goals
- Broad social goals - E.g. Make Friends
- Behavioral categories - Ex Skills for conversation
- Discrete responses - EX. Asking appropriate Q
John is a 45-year-old man who resides in a group home. The employees are concerned because John curses and refuses to cooperate when they direct him to shower. Which of the following would be a socially valid goal for John?
John will:
A. Follow staff directions 100% of the time
B. Independently initiate and take a shower
C. Reduce cursing to zero incidents a month
B. Independently initiate and take a shower
Methods for Assessing Social Validity
- Indirect Technique: Interviews, Questionnaires, Rating Scales
- Direct Techniques Normative comparison and Norm-ref’d assessment
Which of the following is a direct method for assessing the social validity of goals?
A. Normative comparison B. Norm-referenced assessment C. Rating scales D. All of the above E. A and B but not C
E. A and B but not C
You are working with a young child in a classroom, attempting to decrease call-outs. When selecting a target level of behavior, you observe her same- age peers and collect data on the rate of call-outs. This is a(n) ___ technique for meas. the SV of ___.
A. Direct; goals
B. Direct; procedures
C. Indirect; goals
D. Indirect; procedures
A. Direct; goals
*What direct method? Normative comparison
Assessing Risks & Benefits
- Reconcile with Key parties
- General Risk factors of Tx
- Benefits of Tx
- Risks for each procedure
You work with a student in her classroom. You are concerned about the fact that the student’s teacher has stated he does not agree with delivering reinforcers because of the negative effects of extrinsic motivation. Which risk factor?
A. Client and stakeholder buy-in B. Liability to the behavior analyst C. Nature of the behavior D. Risk to others E. Sufficient personnel
A. Client and stakeholder buy-in
Ongoing Assessments Including Treatment Integrity
Social Validity of Procedures
- Comprehensive Tx program: Ex. Reducing prob Bx
- Components of the program Ex. Fx comm. training
- Specific Bx procedures Ex. Using Extinction
Lily is a 5-yr-old who does not stay in her seat and runs around her classroom. She screams when her teachers attempt to physically guide her to do work. Which would be a socially valid procedure?
A. Assist Lily with completing a task using hand-over- hand prompting before allowing her to run around
B. Place preferred items at the table and attending to Lily when she comes and engages with them
C. Seat Lily in a high-chair with a seat-belt for the entire school day
B. Place preferred items at the table and attending to Lily when she comes and engages with them
Methods for Assessing Social Validity
- Indirect Techniques: Interviews, Questionnaires, Rating scales
- Direct Techniques: Participant preference, Sustained use
Before implementing a selected intervention, you ask your client’s parent to answer questions like, “How acceptable do you find the suggested procedures?” This is a(n) ___ meas. of SV of ___.
A. Direct; goals
B. Direct; procedures
C. Indirect ; goals
D. Indirect; procedures
D. Indirect; procedures
Can have a team member not involved in implementation conduct the assessment
Assessing Treatment Integrity
- Select which Tx
- Define steps of Tx
- Determine type of data (global or specific)
- Meas. implementation
- Compare data to criteria (train?)
Bella has created a data sheet that includes all the necessary steps to correctly implement the teaching procedure and plans to calculate a global score as the supervisee is proficient in the protocol. What is the next step Bella should do?
A. Compare data to criteria B. Define the steps of the treatment C. Determine the type of data to use D. Measure implementation E. Select which treatment to measure
D. Measure implementation
Bella’s supervisee, Charlie, is using a FR1 to teach the client beginner intraverbal skills. Charlie forgets to deliver the reinforcer on trial 3 after his client’s correct response. This is an example of which type of treatment integrity error?
A. Error of commission
B. Error of omission
C. Neither (not an error)
B. Intraverbal
Bella has collected data on her supervisee’s adherence to the teaching procedure and is analyzing the graph of % correct implementation. What step for assessing Tx integrity is she doing?
A. Compare data to criteria B. Define the steps of the treatment C. Determine the type of data to use D. Measure implementation E. Select which treatment to measure
A. Compare data to criteria
*What if there is low accuracy? Need to re-train
Assessments Conducted as Goals Are Reached
Social Validity of Results
- Proximal (Short-term) Ex. Reduce Prob Bx
- Intermediate: Ex. Able to go to park
- Distal (Long-term): Ex. Makes friends
Leslie is a 15-yr-old student with an intellectual disability. She has been receiving behavioral services through a social skills program so that she can expand and deepen her peer relations. Which would be a socially valid result? If Leslie:
A. Accurately answered all the therapist’s questions about how to be a good friend
B. Earned an A in the social skills program
C. Went to the roller-rink and exchanged phone numbers with a new friend
C. Went to the roller-rink and exchanged phone numbers with a new friend
Which of the following is an indirect method for assessing the social validity of results (outcomes)?
A. Questionnaires B. Rating scales C. Real-world test D. All of the above E. A and B but not C
E. A and B but not C
After providing services you ask your client’s teacher to fill out form indicating if she dis/agrees with statements like, ”The student is engaged in academic work tasks, sitting appropriately, and attending.” You are using a(n) ___ technique for measuring the social validity of ___.
A. Direct; goals
B. Direct; results
C. Indirect; goals
D. Indirect ; results
D. Indirect; results
After providing services you collect data on your client’s rate of responding to teacher’s questions and compare this to the rate that other students in the class are answering questions. You are using a(n) ___ technique for meas. the SV of ___.
A. Direct; goals
B. Direct; results
C. Indirect; goals
D. Indirect ; results
B. Direct; results
What type of technique? Normative comparison
In behavior analysis, we most often use OBJECTIVE or SUBJECTIVE measures.
*Which of the following is true of subjective measures:
A. Can change over time B. Counter to the natural science approach C. Can identify functional relations D. All of the above E. A and B but not C
*Objective
E. A and B but not C
Assessing consumer choice between treatment options cannot be accomplished if the client has limited expressive language.
A. True
B. False
B. False
The process of identifying both risks and benefits associated with a specific action and making a decision based on these factors
A. Formal reassessment B. Indices of happiness C. Objective measure D. Procedural fidelity E. Risk-benefit analysis F. Social validity G. Social validity assessment H. Subjective measure
E. Risk-benefit analysis
Which of the following is a characteristic of behavioral approaches to assessment?
A. Leads to broad “level” or global score
B. Leads to specific targets
C. Subjective self-report may be only measure
D. All of the above
E. A and B but not C
B. Leads to specific targets
Behavioral assessments require direct (objective) observation of behavior
Which of the following is a direct method for assessing the social validity of results (outcomes)?
A. Expert evaluation B. Normative comparison C. Norm-referenced assessment D. All of the above E. A and B but not C
D. All of the above
Lola’s mom is only supposed to give Lola a popsicle when she says, “popsie.” Lola bangs her hands on the freezer door and her mom opens the freezer and gives her a popsicle. This is an example of which type of treatment integrity error?
A. Error of commission
B. Error of omission
C. Neither (this is not an error)
A. Error of commission
Adherence to all conditions of research design (baseline and treatments)
A. Formal reassessment B. Indices of happiness C. Objective measure D. Procedural fidelity E. Risk-benefit analysis F. Social validity G. Social validity assessment H. Subjective measure
D. Procedural fidelity
Behavior-analytic assessments will always include objective measures.
A. True
B. False
A. True
An outside observer reviews a recorded pre- intervention performance and compares to post- intervention performance. This is a(n) ___ technique for measuring the social validity of ___.
A. Direct; goals
B. Direct; results
C. Indirect; goals
D. Indirect ; results
B. Direct; results
*What type of direct method? Expert evaluation
The degree to which the consumer finds professional involvement acceptable from beginning to end
A. Formal reassessment B. Indices of happiness C. Objective measure D. Procedural fidelity E. Risk-benefit analysis F. Social validity G. Social validity assessment H. Subjective measure
F. Social Validity
The purpose of a behavior-analytic assessment is to identify a diagnosis through measures that lead to a global score or grade level equivalent.
A. True
B. False
B. False
Leslie compares Ron’s red meat consumption to the Am. Heart Association’s recommendation to determine how meaningful a reduction in Ron’s red meat consumption will be to his health. This is a(n) ___ technique for meas. the SV of ___.
A. Direct; goals
B. Direct; procedures
C. Indirect; goals
D. Indirect; procedures
A. Direct; Goals
*What method? Norm-referenced assessment
Which of the following is a behavior-analytic assessment?
A. Assessment of IOA B. Grade placement test C. IQ test D. All of the above E. A and B but not C
A. Assessment of IOA
Kaya is evaluating social validity for a manding program. She has just finished collecting data and is currently summarizing. What is the next step Kaya should complete for conducting a social validity assessment?
A. Collect and summarize the data B. Identify what to assess C. Identify who will be conducting the assessment D. Identify who will be evaluating E. Reconcile any low scores
E. Reconcile any low scores
Measures that are influenced by personal opinions and can only be reported by the individual
A. Formal reassessment B. Indices of happiness C. Objective measure D. Procedural fidelity E. Risk-benefit analysis F. Social validity G. Social validity assessment H. Subjective measure
H. Subjective measure
When behavior analysts attempt to identify if the the participants, caregivers, and other consumers are satisfied the outcomes of services, we are assessing the social validity of the:
A. Goals B. Procedures C. Results D. All of the above E. A and B but not C
C. Results
When behavior analysts attempt to identify if whether the behavioral targets are important to the participants, caregivers, and other consumers, we are assessing the social validity of the:
A. Goals B. Procedures C. Results D. All of the above E. A and B but not C
A. Goals
You work with a student in her classroom. You are concerned about how the fact that this teacher’s class already has more students than it should will impact the teacher’s ability to implement your behavior plan as written. Which risk factor?
A. Client and stakeholder buy-in B. Experience of behavior analyst C. Liability to the behavior analyst D. Prior training of the mediator E. Sufficient personnel
E. Sufficient personnel
Periodic comprehensive assessment to identify progress toward goal, or to reevaluate existing problem behavior
A. Formal reassessment B. Indices of happiness C. Objective measure D. Procedural fidelity E. Risk-benefit analysis F. Social validity G. Social validity assessment H. Subjective measure
A. Formal reassessment
Which of the following is a characteristic of behavioral approaches to assessment?
A. Ongoing B. Requires direct (objective) observation of behavior C. Separates out each operant D. All of the above E. A and B but not C
D. All of the above
*Educational/psychological assessment often combine operants in a “snapshot”
Before designing a behavior plan, you ask your client’s parent to rate the importance of some potential social skills to determine a hierarchy which will guide treatment . You are using a(n) ___ technique for meas. the SV of ___.
A. Direct; goals
B. Direct; procedures
C. Indirect; goals
D. Indirect ; procedures
C. Indirect; goals
What type of indirect method? Rating scale
Which of the following is an indirect method for assessing the social validity of procedures?
A. Interviews B. Questionnaires C. Rating scales D. All of the above E. A and B but not C
D. All of the above
Measures that are impartial and independently verifiable
A. Formal reassessment B. Indices of happiness C. Objective measure D. Procedural fidelity E. Risk-benefit analysis F. Social validity G. Social validity assessment H. Subjective measure
c. Objective measure
The purpose of a behavior-analytic assessment is to gain specific information about individual performance within a specific environment.
A. True
B. False
A. True
A behavior analyst identifies the typical performance and compares this to a dimension of the client’s performance. This is a(n) ___ technique for measuring the social validity of ___.
A. Direct; procedures
B. Direct; results
C. Indirect; procedures
D. Indirect ; results
B. Direct; results
What type of technique? Normative comparison
You go back to the school you consulted with last month to see if the teachers are still using the token economy you designed and implemented. This is a(n) ___ measure of social validity of ___.
A. Direct; goals
B. Direct; procedures
C. Indirect ; goals
D. Indirect; procedures
B. Direct; procedures
What direct method? Sustained use
A way to measure behavior that is correlated with pleasant opinions and feelings
A. Formal reassessment B. Indices of happiness C. Objective measure D. Procedural fidelity E. Risk-benefit analysis F. Social validity G. Social validity assessment H. Subjective measure
B. Indices of happiness
Which of the following is a behavior-analytic assessment?
A. Narrative recording B. Reinforcer assessment C. Psychological inventory D. All of the above E. A and B but not C
E. A and B but not C
After providing services you ask your client’s teacher to fill out form indicating if she dis/agrees with statements like, ” The student is engaged in challenging behavior and not attending.”You are using a(n) ___ technique for meas. the SV of ___.
A. Direct; procedures
B. Direct; results
C. Indirect; procedures
D. Indirect ; results
D. Indirect ; results
What type of indirect method? Rating scale
Before implementing a selected intervention you ask your client’s parent to fill out a form where he answers questions like, “How well do you understand the suggested procedures?”You are using a(n) ___ technique for meas. the SV of ___.
A. Direct; procedures
B. Direct; results
C. Indirect; procedures
D. Indirect ; results
C. Indirect; procedures
*What type of indirect method? Rating scale
Examination of the acceptability, viability, and appropriateness of a client goal, programmed intervention, and/or outcome
A. Formal reassessment B. Indices of happiness C. Objective measure D. Procedural fidelity E. Risk-benefit analysis F. Social validity G. Social validity assessment H. Subjective measure
G. Social validity assessment
Annie has been asked to work with a new client who is having difficulty in the classroom setting. After careful consideration, Annie has identified an appropriate survey to use. What should have already been done?
A. Determine when to assess B. Identify what to assess C. Select methods and tools D. All of the above E. A and B but not C
E. A and B but not C
Which of the following is an indirect method for assessing the social validity of goals?
A. Interviews B. Normative comparison C. Norm-referenced assessment D. All of the above E. A and B but not C
A. Interviews
Bella is creating a task analysis for the teaching procedure to design a data collection sheet for observing and recording. What step for assessing treatment integrity is Bella completing?
A. Compare data to criteria B. Define the steps of the treatment C. Determine the type of data to use D. Measure implementation E. Select which treatment to measure
B. Define the steps of the treatment
Before designing a behavior plan, you ask your client’s parent to answer questions like, “If you could change one thing today that would make your child’s life better, what would it be?” You are using a(n) ___ technique for meas. the SV of ___.
A. Direct; goals
B. Direct; results
C. Indirect; goals
D. Indirect ; results
C. Indirect; goals
*What type of indirect method? Interview
Which of the following is a direct method for assessing the social validity of procedures?
A. Expert evaluation B. Participant preference C. Real-world test D. All of the above E. A and B but not C
B. Participant preference
What is another direct method for assessing the social validity of procedures? Sustained use
Johan observes an RBT running a backward chaining procedure for washing hands. He checks off each step on his sheet as it is completed.
Is this a behavior-analytic assessment?
A. Yes
B. No
A. Yes
After talking to a client’s parents and having them complete a form, Nisha concludes that their child is not ready for a classroom setting based on their assessment score
Is this a behavior-analytic assessment?
A. Yes
B. No
B. No
The ranking of a learner’s academic skills as “needs improvement”, “meeting expectations”, or “exceeding expectations”.
A. Objective measure
B. Subjective measure
B. Subjective measure
The number of times a student vocally states the correct answer to a multiplication problem.
A. Objective measure
B. Subjective measure
A. Objective measure
What is Social Validity?
-The degree of consumer acceptability of services
How is social validity assessed?
*A variety of both objective and subjective means
- ID who to conduct assessment with 2. ID when to assess
- ID what to assess
- ID how to assess (i.e., select methods) 5. ID who will conduct the assessment 6.Collect and summarize data
- Reconcile low scores
Joel has been asked to work with a new client who is having difficulty in the classroom setting. After careful consideration, Joel has identified an appropriate survey to use. What steps for conducting a social validity assessment should Joel have already completed?
A. Identify what to assess
B. Reconcile any low scores
C. Select methods and tools
D. Determine when to assess
E. Collect and summarize the data
F. Identify who to conduct assessment with
G. Identify who will be conducting the assessment
D. Determine when to assess
F. Identify who to conduct assessment with
Joel has been asked to work with a new client who is having difficulty in the classroom setting. After careful consideration, Joel has identified an appropriate survey to use. What steps for conducting a social validity assessment should Joel complete next?
A. Identify what to assess
B. Reconcile any low scores
C. Select methods and tools
D. Determine when to assess
E. Collect and summarize the data
F. Identify who to conduct assessment with
G. Identify who will be conducting the assessment
G. Identify who will be conducting the assessment
What to Measure…
- Goals
*Goals:
Levels: Broad social, Behavioral categories discrete responses
*Meaningful & appropriate to client & society?
What to Measure…..
- Procedures
- Levels: Comprehensive plan, components, specific procedures
- Are these acceptable to relevant parties?
What to Measure….
- Results
- Levels: Short-term, intermediate, long-term
- Are consumers satisfied with both intended & unintended results?
Indirect Methods
- Interview/ Questionnaires: Helps prioritize goals identifies expectations & values
- Rating Scales: May assist when there is a disagreement of priority
Direct Methods
- Normative Comparison: Compare current client data to sample to inform client goals
- Norm-referenced assessment or curricula: Use of a curriculum/ standard to identify valid goals
Which of the following is likely the most socially valid goal for a 7-year-old child who is behind on some academic skills and screams when given a difficult task?
A. Complete all seat work at 100% accuracy
B. Follow all instructions provided by an adult
C. Raise hand to request assistance when given difficult seat work
C. Raise hand to request assistance when given difficult seat work
Leslie compares Ron’s red meat consumption to the American Heart Association’s recommendation to determine how great a reduction in Ron’s red meat consumption will be appropriate/meaningful to his health. Which method for assessing social validity is being used?
A. Behavioral interview B. Rating scales C. Normative comparison Leslie compares Ron’s red meat consumption to the American Heart Association's recommendation to determine how great a reduction in Ron’s red meat consumption will be appropriate/meaningful to his health. Which method for assessing social validity is being used? A. Behavioral interview B. Rating scales C. Normative comparison D. Norm-referenced assessment or curricula
D. Norm-referenced assessment or curricula
Weighing Risks and Benefits
- Nature of the Bx
- Is there sufficient personnel?
- Is the mediator well-trained?
- Is the setting appropriate?
- Is the Bx analyst experienced?
- Risk to others
- Buy-in
- Personal liability
Izabella has identified that the classroom may be an inappropriate place to implement the treatment component of extinction for attention-maintained behavior as it is likely not possible to control for the attention delivered by peers.
A. Nature of the behavior B. Sufficient personnel C. If the mediator is well trained D. Appropriateness of the setting E. Experience of the behavior analyst F. Risk to others G. Buy-in H. Personal liability
D. Appropriateness of the setting
How to Assess: Procedures Indirect
Interview/ Questionnaires)
Assesses stakeholder opinion on the feasibility/ appropriateness of procedures
How to Assess: Procedures Indirect
Rating Scales
-May also include acquiring information on indices of happiness
How to Assess: Procedures
Direct
Participant preference
-Allow individual to indicate treatment preference(of options they have experienced). Commonly uses concurrent chains arrangement
How to Assess: Procedures
Direct
Sustained Use
-Observe if procedures continue to be used by implementers in the setting
Amar plans to train the parents of his client to delivery reinforcement paired with praise when the client engages in appropriate peer play with their sibling.
Is this likely to be a socially valid procedure?
A. Yes
B. No
A. Yes
Amar also plans to train the parents to use an escape extinction procedure when transitioning through a bedtime routine, this may include full physical guidance despite the parents expressing concerns about the use of manually forcing the child through actions.
Is this likely to be a socially valid procedure?
A. Yes
B. No
B. No
Simeon goes back to the school he consulted with last month to see if they are still implementing the token economy he had instituted. Which method for assessing social validity is being used?
A. Behavioral interview
B. Questionnaire/Rating scales C. Participant preference D.Sustained use
D. Sustained use
Procedural fidelity is how well _____ are adhered to.
Treatment integrity is how well _____ is implemented as designed.
- Conditions of the research design
* The Treatment Condition(s)
Types of Treatment Integrity Errors:
- Omission: A step missing (or omitted) that should have occurred
- Commission: A step was added (committed) that should NOT have occurred
When working with Sunflower the dog I am supposed to approach and provide a treat if she sits or lies down when I enter the room. I am to turn-away and leave if she jumps. When I enter the room, Sunflower runs toward me and I say “good girl” and give a treat.
A. Omission error
B. Commission error
B. Commission error
When working with Sunflower the dog I am supposed to approach and provide a treat if she sits or lies down when I enter the room. I am to turn-away and leave if she jumps. After entering the room, I forget to turn-away and leave when she jumps.
A. Omission error
B. Commission error
A. Omission error
Assessing Treatment Integrity
- Select the treatment to measure
- Define the steps of treatment/procedure
- Determine the type of data to use (global or specific)
- Measure implementation via observation
- Compare to selected criteria
- Train if needed
Robin’s supervisor has collected data on Robin’s adherence to the teaching procedure being used to teach hand-washing to a client. The supervisor compares the data collected to the set criterion for implementation of the procedure and identifies that Robin is inappropriately deviating from the procedures as-written. Which step for assessing treatment integrity does this describe?
A. Train implementer if needed
B. Select which treatment to measure
C. Define the steps of the selected treatment
D. Compare data collected to predetermined criteria
E. Determine the type of treatment integrity data to use
F. Measure the implementation through direct observation
D. Compare data collected to predetermined criteria
Robin’s supervisor has collected data on Robin’s adherence to the teaching procedure being used to teach hand-washing to a client. The supervisor compares the data collected to the set criterion for implementation of the procedure and identifies that Robin is inappropriately deviating from the procedures as-written. Which step should Robin’s supervisor complete next?
A. Train implementer if needed
B. Select which treatment to measure
C. Define the steps of the selected treatment
D. Compare data collected to predetermined criteria
E. Determine the type of treatment integrity data to use
F. Measure the implementation through direct observation
A. Train implementer if needed
How to Assess: Results Indirect
Interview/ Questionnaires
Rating Scales
-Assesses stakeholder opinion on the obtained results post-intervention
How to Assess: Results, Direct
Expert Evaluation
-Outside expert reviews pre- and post-intervention performance and compares
How to Assess: Results, Direct
Normative Comparison
-Compare client data post- intervention to sample
How to Assess: Results, Direct
Norm-referenced assessment
-Compare client score on standardized assessment to typical population
Hot to Assess: Results Direct
Test in Natural Environment
-Assess is behavior change maintains over time and in real-world contexts
Torrey is an adult who speaks in stereotypic phrases, and otherwise has limited verbal skills. In his workshop he stuffs colored paper into envelopes. He often runs out of the shop into the breakroom where he sits and plays an old poorly tuned piano with considerable skill. Prior to treatment, anytime he was redirected back to work from the piano, he would bite his hand (causing some mild tissue damage). Now Torrey is able to play the piano and sing some of the songs, but still bites his hands when asked to return to work. Is this a socially valid result?
A. Yes
B. No
B. No
Chris asks an independent SLP and pediatrician to review video before and after their client, James was provided services for food refusal and over selectivity. Which method for assessing social validity is being used?
A. Behavioral interview B. Rating scales C. Expert evaluation D. Normative comparison E. Norm-referenced assessment F. Test in the natural environment
C. Expert Evaluation
Lacie sits with each of a sampling of staff that have been using the level system Lacie designed for a group home. She asks them questions about the difficulty of the system, how appropriate it seems, and how well it is understood among staff. Which method for assessing social validity is being used?
A. Behavioral interview
B. Rating scales
C. Participant preference
D. Sustained use
A. Behavioral Interview
Amara, a supervisor, has taught her trainee/ supervisee how to use the BST model to teach a skill. Amara now watches her trainee/ supervisee train a new employee in implementing a program. Which method for assessing social validity is being used?
A. Behavioral interview B. Rating scales C. Expert evaluation D. Normative comparison E. Norm-referenced assessment F. Test in the natural environment
F. Test in the natural environment
Michael takes data for a week on the rate of Eleanor’s tantrums, defined as any combination of 2 or more of: flopping, kicking, crying, screaming, hitting inanimate objects. He then takes data for a week on the rate of this behavior in children of a similar age as Eleanor. Michael compares the two to identify what an appropriate and meaningful change in this behavior would be. Which method for assessing social validity is being used?
A. Behavioral interview
B. Rating scales
C. Normative comparison
D. Norm-referenced assessment or curricula
C. Normative comparison
Behavioral Assessment
•This is how we collect information to make data-based decisions regarding behavior and the environment
- Behavior
- Environment
Lucas runs through a bunch of targets, writing how his client does after each question, and at the end labels him with a diagnosis of ADHD. Is this behavioral assessment?
A) Yes
B) No
B) No
•Narrative recording.•Is this behavioral assessment?
A) Yes
B) No
A. Yes
Objective
•Measures are impartial and independently verifiable
•Two people observing bx can see and record the same thing
-This is What we are striving for with our response measures
Subjective
•Measures reflect person bias and opinions and can only be reported by the individual
-Problematic on its own –we don’t use but we can supplement objective measures with subjective
•Which of the following is objective?
A) The sound of the doorbell ringing
B) Hiring a baby sitting with a love for children
C) ADHD
D) Excitement for your upcoming birthday
A) The sound of the doorbell ringing
Megan dropped her car off for an oil change. After she picked it up, she immediately got a survey via email asking her questions about her experience. The questions asked her to rate her satisfaction with the experience on a scale of 1-5. This is an example of a(n) ____ measure.
A) Objective
B) Subjective
B. Subjective
Which of the following is the MOST objective measure?
A) A peer selecting who had the loudest roar out of 3 kids in a roaring contest
B) Playing loud on a piano versus soft and having someone judge if the last song was louder or softer than the previous song
C) Using a sound level meter to determine how many decibels a scream is
D) Seeing if someone can hear you scream their name from 10 feet away
C) Using a sound level meter to determine how many decibels a scream is
Social Validity
*We are opening ourselves up to subjective measures here and it’s ok. But why is it ok?
- The degree to which the consumer finds out involvement acceptable from beginning to end (Wolf, 1978)
- All PARTs –we should be open to feedback
- Do the consumers like…
- What we’re doing?
- How we’re doing it?
- How progress is being made?
SOCIAL VALIDITY IS SUPER IMPORTANT
- Consumer is more likely to stick to and carry out treatment in the absence of behavior analyst
- Better outcomes because of adherence
- Encourages others to seek help/treatment: more positive word of mouth
Social Validity Assessment
•Assessment that you will conduct to determine if the client goals, programmed intervention an/or outcome are acceptable, viable, and appropriate
•Assessing if the behavior change is of clinical or applied importance
•We can measure social validity in three areas
-Goals
-Procedures
-Results
Social validity is really an “all-or-nothing” thing?
A) True
B) False
A) True
Socially valid goals
•Directly applies to the APPLIED dimension of ABA
•Hence, goals are based on their social importance
•Not determined by the topography. Determined by how much it will help the individual
-Example: teaching someone type a resume
-Non-example: teaching someone to type but they can’t read
Is making eye contact 100% of the time during a conversation a socially valid goal?
A) Yes
B) No
B) No
Is brushing teeth twice a day a socially valid goal?
A) Yes
B) No
A) Yes
How to Assess
•Broad social goals -Ultimate outcomes •Behavioral categories -Intermediate outcomes •Discrete responses -Specific target behavior
Which one is a level for assessing the social validity of goals?
A) Broad social goals
B) Ultimate responses
C) Emotional flexibility
D)Deep understanding
A. Broad Social goals
General Process for evaluating social validity
- Identify WHO will be evaluating
- Determine WHEN to assess
- Identify WHAT to assess
- Select methods and tools
- Identify who will conduct the assessment
- Collect and summarize the data
- Reconcile any low scores
Sarah is in the process of getting ready to evaluate social validity. She identified her client’s parents as important individuals to conduct the assessment with. She will do this pre-intervention. Now that she knows who will be evaluating and when it will take place, what should she do next?
A) Identify what to assess
B) Collect and summarize the data
C) Reconcile any low scores
D) Select methods and tools
A) Identify what to assess
Janelle is evaluating social validity for an imitation program. Janelle just finished collecting data and is currently summarizing. What is the next step Janelle should complete for the social validity assessment?
A) Identify who will be evaluating
B) Identify what to assess
C) Select methods and tools
D) Reconcile any low scores
D) Reconcile any low scores
Which of the following is NOT an area to measure social validity?
A) Goals
B) Procedures
C) Results
D) Rapport
D) Rapport
Indirect Techniques
•Interview
•Rating Scale
-List potential targets/goal and have team members rate them
•Questionnaires
Direct Techniques
-Normative comparisons
- Normative comparisons
- Collect data on TB of both client & normative sample
- Compare data between client and sample
- Set client goals
Direct Techniques
Norm-referenced assessment and/or curricula
•These exist already such as VBMAPP [broken down by age], Essentials for Living [based on function]
Risk-benefit analysis
•When the bx is dangerous, or the procedures are more invasive and so we would want to take additional steps to make sure we are being safe and using the most effective procedures
General Risk Factors
•Nature of the behavior• Sufficient personnel •Meditator training •Appropriate setting •Experience of behavior analyst •Risk to others •Buy-in •Personal liability
Mariah is weighing the risks and benefits associated with a particular intervention for the client she is working with. Right now, she is unsure if the benefits outweigh the risks because the client spends most of his time at school and teachers are not on board with the protocol. Which general risk factor is Mariah considering?
A) Nature of the behavior
B) Experience of the behavior analyst
C) Buy-in
D) Personal liability
C) Buy-in
Social Validity of procedures
- Acceptability of treatment procedures
- Three levels (from very broad to very narrow)
•Comprehensive treatment programs
-Scope of treatment (clear description of the full program), setting, stakeholders
•Components of a program/package
-General components or teaching approach used
•Specific procedures
-Planned treatment component [what are we doing exactly]
Ashley is a BCBA & working with a 3-year-old client, Mike.Mike drops to the ground & screams & cries when he is denied access to mom’s phone. At the center, the team is using a protocol that mom consented to. The procedure involves ignoring the screaming and crying and turning away. Mom just informed Ashley that she is not totally comfortable doing this in public should Mike drop to the ground and start screaming and crying. What should Ashley do?
A) Continue using the current procedure as it is effective and approved
B) Consider a new procedure to reduce the tantrums
C) Beg mom to atleast try it for a week in public & see what happens
D) Ask mom if she can try to not take Mike out in public until they squash this target behavior
B) Consider a new procedure to reduce the tantrums
Indices of happiness
- A measure of opinions/feelings
- These are indirect measures
- Basically, one type of behavior is being measured as an indication of something else
Treatment integrity versus procedural fidelity
•DATA collection set up and follow through looks the same for both. The only difference is what you are collecting data on.
- Treatment integrity versus procedural fidelity
- Fidelity and integrity …ok …interchangeable
- Treatment and procedure is where it differs…
- Treatment…is the treatment being implemented as intended
- Procedure…are we doing what were supposed to be doing across all conditions (e.g., baseline and treatment)
I want to ensure that mom is implementing the treatment the way I wrote it. I should collect…
A) Treatment integrity
B) Procedural fidelity
A) Treatment integrity
To ensure all parts of my thesis are being conducted as written I need to collect some data on proper implementation of all phases of my study (e.g., baseline, treatment, generalization, follow-up). I should collect…
A)Treatment integrity
B)Procedural fidelity
B) Procedural Fidelity
•Treatment integrity is SO important for many reasons one of which is…..
A) To guarantee social validity is high
B) To make sure DV is reliable
C) To ensure all therapists are using the correct intervention and doing it correctly
D) All of the above
C) To ensure all therapists are using the correct intervention and doing it correctly
Problem with poor treatment integrity
•Potential issues with acquisition
-Learning at a slower rate than expected
•Continue engaging in problem behavior for longer than expected
•Implementers may abandon effective treatment and use more restrictive treatment
Steps to assessing treatment integrity
- Select which treatment to measure
- Define the steps of the selected treatment
- Determine the type of treatment integrity data to use (global or specific)
- Measure implementation through direct observation
- Compare data collected to predetermined criteria
- Train implementer if needed
Ralph is measuring treatment integrity of the echoics program the therapists are implementing with his client. He has a data sheet that includes all the necessary steps to correctly implement it. He plans to calculate a global score as the technician is proficient in the protocol. What should Ralph do next?
A) Train implementer
B) Select with program to measure
C) Measure implementation through direct observation
D) Define the exact steps in the protocol
C)Measure implementation through direct observation
Romy is collecting treatment integrity data. She is currently conducting a booster session due to low integrity scores. Which step did Romy just complete prior to conducting the booster session.
A) Write out each step in the treatment
B) Graph the collected data
C) Measure through direct observation
D)Select which treatment to measure
B) Graph the collected data
Donna is noting the steps specific to a procedure in the sequence in which they should occur to help design a data collection sheet. What steps for assessing treatment integrity would Donna have already completed?
A) Select which treatment to measure
B) Determine the type of integrity data to use
C) Measure the implementation through direct observation
D)Train the implementer if needed
A) Select which treatment to measure
Omission
Leaving out part of treatment that was supposed to be applied
Think “OMIT”
Commission
Did something (added an action) that you were not supposed to Think “COMMITTED”
Reasons for Low Integrity
1) Complex Protocol
- Complex protocol (e.g., it’s too hard, or doing too many things at once)
- Simplify and have clear descriptions
Reasons for Low integrity
2) Lack of Generalization
*Lack of generalization (e.g., not running protocol as written in other settings)•Train across settings with sufficient exemplars
Reasons for Low Integrity
3) Treatment Drift
Treatment drift (e.g., txis applied different by accident simply due to passage of time) •Careful monitoring, booster training
Reasons for Low Integrity
4) Competing contingencies
•Competing contingencies (e.g., managing too many tasks at once)
-Prioritize, train, feedback
•Lack of training (e.g., inadequate or incomplete training)
-Competency-based training across exemplars and context
Alexander is currently using a FR1 to teach his client beginner intraverbal skills. Alexander forgets to deliver the reinforcer on trial 3 after his client’s correct response. Which type of treatment integrity error?
A) Omission
B) Commission
A) Omission
Darcy’s mom is only supposed to give Darcy a popsicle when she says, “popsie.” Darcy bangs her hands on the freezer door and her mom opens upthe freezer and gives her a popsicle. Which type of treatment integrity error?
A) Omission
B) Commission
B) Commission
Formal Reassessment
•Periodic comprehensive assessment to identify progress toward goal, or to reevaluate existing problem behavior •Can do this for many reasons: •Client progress• Required by funding sources •Organizational policy •Curriculum guidelines
Socially valid results
•Can look at results across three different levels
-Proximal [short-term/immediate]
-Has target behavior changed as desired?
•Intermediate[larger category of skill] -Example: Social skills, increased access to reinforcement
•Distal[long-term, bigger picture stuff]
-Meeting intermediate & ultimate outcomes
Indirect
Assess stakeholder opinion about results of service
- Interview
- Questionnaires
- Rating Scales
Direct
Evaluate behavior of client compared to criteria of behavior change
- Expert evaluation
- Normative comparison
- Norm-referenced (standardized) assessment
- Test in natural environment (real-world test)
The BCBA invites someone well versed in verbal behavior to review a video of the communication skills of the client both before and after treatment to identify if improvements were meaningful. Which method for assessing social validity is being used? A) Behavioral interview B) Rating scale C) Expert evaluation D) Test in the natural environment
C) Expert Evaluation
Outline
- Assessments/Social Validity
- Initial
- Ongoing
- Results
Behavior-Analytic Assessments
Gathering information to make treatment decisions
–Not a diagnosis
–Not “level”
–Not a snapshot
Behavior-Analytic Assessments
Examples
- Functional Behavior Assessments
- Skill Assessments
- Preference Assessments
- Social Validity Assessments
Behavior-Analytic Assessments
NON Examples
- Diagnostic Assessments
- Grade Level Assessments
- IQ tests
- Personality tests
Social Validity
•The degree of acceptability of our involvement
•3 Common areas to assess:
–Goals
–Procedures
–Results
Social Validity Assessment
- Identify (who) will evaluate
- Determine (when)
- Identify (what) to assess
- Select Methods (How)- #5. Identify who will conduct
- Collect and summarize
- Reconcile
Socially Valid Goals
*Should be Socially relevant/ Appropriate
- Behavioral targets (Discrete responses)
- Intermediate goals (Behavioral categories)
- Larger goals (Broad social goals)
General Risk Factors
•Nature of behavior •Sufficient personnel •Mediator’s level of training •Appropriate-ness of setting •Experience of behavior analyst •Risk to others •Buy in• Personal liability
Socially Valid Procedures
- Comprehensive program
- Components of program
- Specific procedures
- Balancing effectiveness and acceptability
Methods for Assessing Procedures
- Interviews
- Questionnaires
- Rating scales
- Participant Preference
- Sustained use
Participant Preference
- Expose the client to each treatment option
- Allow them to choose
- Collect and compare data (effectiveness as well as preference)
Treatment Integrity
- How accurately the treatment is implemented as intended
* Often used along with IOA in evaluating technician performance
Assessing Tx Integrity
- Select treatment to measure
- Define the steps
- Determine data type (Global vs Specific)
- Measure
- Compare data
- Train (if needed)
Socially Valid Results
•All intended and unintended results matter
- Proximal
- Intermediate
- Distal
Assessing Results
- Interviews/Questionnaires
- Rating Scales
- Expert Evaluation
- Normative Comparison
- Norm-referenced assessment
- Real-world test
Expert Evaluation
- Try to make “blind” when possible
- Use recordings/permanent products
- Evaluators score each without knowing if it is pre- or post- treatment sample
Trini is observing a child’s math skills and scoring performance based on an assessment tool. In the end the assessment will provide a range of grade level equivalent performance.
a. Behavioral assessment
b. Not a behavioral assessment
b. Not a behavioral assessment
Which is a behavioral assessment?
a. Scoring language skills one time for a broad score
b. Scoring specific language skills ongoing basis
c. Scoring self-report for diagnosis
b. Scoring specific language skills ongoing basis
Which are objective measures?
a. Likert scales
b. Brightness measured in lumens
c. Weight in grams
d. Asking someone “is this too loud?”
e. Asking someone “is this heavy?”
f. Measuring speed with a stopwatch
b. Brightness measured in lumens
c. Weight in grams
f. Measuring speed with a stopwatch
Stephanie is planning a social validity assessment for her toilet training program. She has decided an interview will be the most appropriate method. What should she do next?
a. Conduct the interview
b. Identify who is best to conduct the interview
c. Determine who to interview
b. Identify who is best to conduct the interview
Stephanie is planning a social validity assessment for her toilet training program. She has decided an interview will be the most appropriate method. What should she have already done?
-Decided who to interview; At what point to conduct the interview; Identified what to ask about
Leslie compares Ron’s red meat consumption to the American Heart Association’s recommendation to determine how great a reduction in Ron’s red meat consumption will be appropriate/meaningful to his health. Which method for assessing social validity is being used?
a. Behavioral interview
b. Rating scales
c. Normative comparison
d. Norm-referenced assessment or curricula
d. Norm-referenced assessment or curricula
Which would be a normative comparison for identifying socially valid goals?
a. Observing a typical preschool classroom to identify the instructions to follow
b. Using the VB-MAPP curriculum to identify complexity of skills
c. Asking a parent what they consider
a. Observing a typical preschool classroom to identify the instructions to follow
Gary is worried about the intensity of self-injury demonstrated by a potential client. What risk factor is he examining?
a. Nature of behavior
b. Mediator training
c. Experience of behavior analyst
d. Buy in
a. Nature of Bx
Gina is worried about her client’s rate of aggression toward peers. She considers their safety while determining the best behavior reduction procedures. She is accounting for which risk factor?
a. Mediator training
b. Experience of behavior analyst
c. Risk to others
d. Appropriate setting
c. Risk to others
The parents are divided on use of time-out. The behavior analyst suggests that time-out may reduce problem behavior more rapidly. Alternatively, the analyst has designed a differential reinforcement procedure that is likely to reduce behavior, but not as rapidly. Which procedure is more socially valid in this case?
a. Time out
b. Differential reinforcement
b. Differential RX
The analyst identified a need to reduce aggression towards their client’s peers. To identify termination criteria, they watch a classroom of same age individuals and count episodes of aggression. What is the analyst assessing?
a. Social validity of goals
b. Social validity of procedures
a. Social validity of goals
What method did they use?
*Normative Comparison
Kay has tested out 2 different error correction procedures. She set up sessions in which she used one type exclusively and correlated with colored card. She then allowed the client to select the condition before session. How is the analyst assessing social validity?
a. Normative comparison
b. Client preference
c. Sustained use
b. Client preference
Leann is tracking how well a technician is following the designed teaching procedures. She has determined that she wants data the reflect overall performance, will all steps included. What should she have already done?
a. Measured the technician’s performance
b. Conduct follow-up training
c. Compared the data
d. Defined the steps in the teaching
d. Defined the steps in the teaching
What should next step be?
-Measure
Judy compares the rates of school absences for her client after treatment against the national average for their age group. What is Judy assessing?
a. Social validity of goals
b. Social validity of results
b. Social validity of results
Judy compares the rates of school absences for her client after treatment against the national average for their age group. What is Judy assessing?
a. Social validity of goals
b. Social validity of results
b. Social validity of results
What methods
-Norm-referenced
Bill shows teachers examples of essays written by his client before and after his writing intervention. The teachers score each paper. How is Bill assessing the social validity of the results of his intervention?
a. Norm-referencing
b. Real-world use
c. Expert evaluation
d. Interview
C. Expert evaluation
Behavioral Assessment
*A systematic gathering of information to make data-based decisions, regarding behavior and the environment
–To determine if a problem exists
–Describe behavior & the environment
–Determine/describe functional relations
Behavioral Assessment
Vs Psycho/ Educational Assessments
-Behavioral
- Leads to specific targets
- Separates out each operant (function)
- Frequent, ongoing assessment
- Requires direct observations
Behavioral Assessment
Vs Psycho/ Educational Assessments
Psychological
- Lead to “Level” or global score
- Combine operants
- Pre/ post- treatment snapshots
- May rely solely on subjective measures, such as self or caregiver report
Targeting a student’s attitude in class would be an example of a(n):
A. Objective measure
B. Subjective measure
B. Subjective Measure
Which would be a objective measure to replace the student’s “attitude” in class.
A. The student’s tone of voice
B. The student’s agitation level
C. How happy the student is in class
D. How many times the student makes comments such as “this is stupid” or “I hate this”
D. How many times the student makes comments such as “this is stupid” or “I hate this”
Social Validity
Degree to which the consumer finds professional involvement acceptable from beginning to end •The “so what?” factor •Were the procedures acceptable? •Did the results/outcomes matter? •Would they continue?
Assessing Social Validity
Examining the acceptability, viability and appropriateness of a goal, intervention, or outcome
–Are the goals, procedures, and results acceptable to the client and stakeholders?
–Is the intervention DO-able?
–Are the goals and procedures reasonable and appropriate?
Assessing Social Validity•
•Areas: –Goals •Is this really what the client and society want? –Procedures •Are methods acceptable? –Results• Are ALL the results satisfactory?
Assessing Social Validity
•Social validity by nature is subjective, but supports the applied dimension of ABA•
Should assess social validity throughout treatment, aim for high validity, and
•Be responsive to measures!
Socially Valid Goals
- Is this really want the client, stakeholders, and society really want?
- “If I could change your (child’s) life by teaching one skill today, what would that be?”
- “Where will this person be going, what skills will they need to be successful there, and where are they in terms of those skills now?”
Goals: Three Levels
- Broad Social Goals–Ultimate outcomes
- Behavioral Categories–Intermediate Outcomes
- Discrete Responses–Target behaviors or short-term objectives
Assessing Goals
- Methods: Ask Client, Stakeholders
- Compare: Successful individuals, typically developed peers
- Standards
Aaron is a passionate behavior analyst who wants to provide quality care and change the lives of his clients. Before doing an intake for a 5-year-old boy with autism, Aaron determines that the boy’s parents will be the best determiners of social validity, and that he wants to assess validity prior to, during, and following treatment. Aaron’s next step would be to:
A.Identify who will conduct the assessment
B.Identify what to assess
C.Collect and summarize the data
D.Select methods and tools
B. Identify what to assess
Angelique is a 6-year-old girl who does not engage in any vocal verbal behavior. Which of these would be a socially valid goal?
A. Identify community helpers
B. Make simple requests using sign or single words
C. State her parent’s names
D. Identify the letters of the alphabet receptively
B. Make simple requests using sign or single words
Asking Angelique’s parents what their biggest concern is would be an example of which method of assessing the social validity of a goal? A. Interviews/questionnaires B. Rating scales C. Normative comparison D. Norm-referenced assessment/curricula
A. Interviews/ questionnaires
John is a 45-year-old man who resides in a group home. John curses at the staff when they direct him to shower and shave and has cursed at his housemates as well, if they make comments to him. Which of these would be a socially valid goal?
A. John will reduce cursing to zero incidents a month for 6 months.
B. John will follow a picture schedule daily with 100% accuracy.
C. John will follow staff directions 85% of the time.
D. John will learn to shave and shower independently
D. John will learn to shave and shower independently
Determining that reducing cursing to zero is inappropriate as a goal for John because many adults curse from time to time would be an example of which method of assessing the social validity of a goal?
A. Interviews/questionnaires
B. Rating scales
C. Normative comparison
D. Norm-referenced assessment/curricula
C. Normative comparison
Demetrius is another adult living in the group home with John. Demetrius loves shopping for his personal items and other preferred household items. Which would be a socially valid goal in this case?
A. Learning to identify money: coins and bills.
B. Learning to balance a checkbook
C. Learning to make a shopping list and find the items in the store.
D. Learning to Identify items as “need” or “want”
C. Learning to make a shopping list and find the items in the store
Risk Benefits Analysis
- Comparing risks to related benefits, and making decisions to act accordingly
- Asssess general risk factors and benefits of behavioral treatment
- Assess the risk factors associated with each individual procedure
- Reconcile risks and benefits with stakeholders
Procedures: 3 Levels
- Comprehensive treatment programs–Scope of treatment, setting, included stakeholders
- Components of program/ package–Programming for acquisition targets–Interventions for deceleration targets–General approaches
- Specific procedures–Extinction, punishment, prompting, shaping, etc
•Treatment Integrity
–Extend to which procedure is carried out as intended
–Did you implement the plan correctly?
•Procedural Fidelity
–Adherence to all conditions of research baseline design–Did you implement the design correctly?
Discontinuing the extinction procedure due to lack of parent implementation would be an example of which method of assessing the social validity of a procedure? A. Interviews/questionnaires B. Rating scales C. Participant preference D. Sustained use
D. Sustained use
Kelly has trained the classroom teacher and paraprofessionals on the use of a crisis management system that includes the use of physical restraint procedures to protect the student and others in emergencies. She has created a checklist of the steps for a specific procedure and plans to collect data on each individual step. What is Kelly’s next step in assessing treatment integrity?
A. Compare data collected to predetermined criteria
B. Measure implementation through direct observation
C. Define the steps of the selected treatment
D. Determine the type of treatment integrity data to use (global or specific)
B. Measure implementation through direct observation
During her observation, Kelly sees a paraprofessional place her hand on the back of a child’s neck, which is NOT an approved procedure. This represents an error of:
A. Omission
B. Commission
B. Commission
Kelly also observes the teacher fail to place their open hand on the child’s upper arm, just above the elbow. This represents an error of:
A. Omission
B. Commission
A. Omission
- Formal Reassessment
- •Periodic, comprehensive assessment to identify progress, or reassess function of behavior
•Initiated due to:
–Client progress, or lack thereof
–New behavior, or changes in effectiveness
–Funding source or organizational requirements
–Curriculum guidelines
Socially Valid Results
•Acceptability of effects
•Are clients/stakeholders satisfied with ALL the results?
–Intended AND unintended
Malik is a 7 year-old who attended an ABA clinic. One of his targets was initiating play with peers, which he learned via role play, and was observed to demonstrate during social skills groups at the clinic. However, his parents reported that when they picked him up from after-school care recently, Malik was sitting on the playground by himself, playing with mulch. This indicates a failure of social validity at which level?
A.Short-term (proximal)
B.Intermediate
C.Long-term (distal)
B. Intermediate
Which of these is an example of a socially valid outcome for a high-school-aged student with an intellectual disability?
A. Received a certificate of attendance
B. Attended prom with a self-selected group of peers, both with and without disabilities
C. Attended an end-of the year-classroom party
D. Received an A in social skills class.
B.Attended prom with a self-selected group of peers, both with and without disabilities
The fact that Loukia’s son no longer met the diagnostic criteria for autism represents which method for assessing the validity of outcomes?
A.Interviews/questionnaires
B.Rating scales
C.Expert evaluation/standardized assessment
D.Test in natural environment
C.Expert evaluation/ standardized assessment
