Reliability, Treatment Integrity, Graphing Flashcards
Reliability
Consistency (repeatability) of measurement by a given “instrument” determined via level of agreement between 2 observers recording the same bx at the same time
Level
Where the line is on a graph (higher or lower)
Variability
How much fluctuation on the pattern of performance
Trend
Overall direction taken by a data path
Procedural integrity/treatment fidelity
Extent to which procedures are implemented as specified or intended
What are the sources of reliability problems (Kazdin 1977)? What are some solutions to these problems?
- Inadequate instrument-have clear definitions, simplify procedures
- Inadequately trained observers-train via BST
- Instrument decay-know that consequences can influence reliability
- behavioral characteristics- know that rate affects the stringency of calculations
What is the process for collecting reliability?
- Periodic checks throughout evaluation
- Observers record independently
- Unobtrusive reliability checks if possible
- Acceptable level of agreement depends on variety of factors, as certain methods more stringent than others
What are the strengths and limitations for calculating total agreement?
- Pros-easy
- Cons-Does not establish agreement for any instance of Bx. Least stringent method-not adequate for research
What are the strengths and limitations of calculating exact agreement?
Pros-More conservative than total
Cons-overly stringent, especially for high rate behavior
What are the strengths and limitations of calculating proportional agreement?
Pros-compromise between total and exact
Cons-also influenced by rate
What are the strengths and limitations of calculating total interval IOA?
Pro-
Con-Influenced by rate (overestimates low or high rate B)
What are the strengths and limitations of calculating occurrence interval agreement?
Pro-More stringent for low rate
Con- influenced by rate of Bx (less stringent for high rate)
What are the strengths and limitations of calculating nonoccurrence interval agreement?
Pro-more stringent for high rate
Con-influenced by rate
What are the strengths and limitations of calculating trial-based agreement?
Pro-
Con-Influenced by number of opportunities
What are the 3 practical uses of reliability and procedural integrity monitoring?
- Positive and corrective feedback
- Provide delayed and cumulative performance feedback to data collectors and procedure implementers
- Aids in clinical decision-making
What is the process for evaluating and calculating procedural integrity?
- Periodic checks throughout evaluation
- Develop recording method
- Unobtrusive checks if possible
- Acceptable level of integrity depends on variety of factors
Describe the sources of integrity problems
- Inadequate training
- Complex procedures
- Procedural drift
- Consequences
What are some ways to misrepresent data using graphing strategies?
- Summary bars
- Plotting large blocks of time to reduce variability
- Overly big or small axis range
What is the importance of monitoring and reporting treatment data?
To inform clinical decision-making
What are the current trends in the reporting of independent variables?
95%
What are the current trends in the reporting of treatment integrity data?
Reporting of treatment data has been low and stable over the years
