quality of assessments

Question 1

what are the conditions of an assessment?

Answer 1

• must be flexible (have a broad range of options beyond selecting frequency, interval, duration, and measurement strategies)
• must be rigorous (observation need to be made consistently)

Question 2

what are natural vs. contrived trades, activities, and settings?

Answer 2

• performance is observed without intervening or structuring the situation for the client
• often not possible as who knows how long you’ll have to watch them for?
• best to select contrived tasks as it is easier to do than naturalistic

Question 3

what is the natural environment vs. laboratory settings?

Answer 3

• naturalistic settings are real-life
• ideally, direct observations are made in a natural setting where clients normally are
• however, natural settings are often expensive
• usually real-life situations are used but something about them is contrived to allow observations to be made

Question 4

what are obtrusive vs. unobtrusive assessments?

Answer 4

• obtrusive (clients are aware of the measurement process and that their behavior is being assessed)
• obtrusive could be reactive, meaning it could influence the performance of the client
• unobtrusive (clients are not aware of the measurement process)
• often unobtrusive assessment is not possible due to clients having to be informed of the intervention

Question 5

what are human observers vs. automated recording?

Answer 5

• in most cases, humans assess behaviors
• with automated recording (recommended the most), humans are only involved when it comes to calibration of the machine
• often automated recording is better as it is not biased, can assess behavior more accurately, etc.

Question 6

what are point-by-point agreement ratios?

Answer 6

• assesses whether there is an agreement of each instance of observed behavior
• uses discrete opportunities (number of trials, intervals, or correct answers)
• most useful in interval assessment

Question 7

what is plotting agreement data?

Answer 7

• may be one observer (primary) observing the client each day and perhaps a second observer (secondary) also occurring the behavior
• usually only the data from the primary observer is plotted but data from a secondary observer may also be plotted to see similarities between them
• provides useful information about how closely the observers usually agreed in their totals for occurrences

Question 8

what are conducing checks for agreement?

Answer 8

• agreement has to achieve an acceptable level before baseline observations are begin and has to maintain that level throughout the project
• ageeement has to be between 80-100%
• major factor of influence interconnects agreement is whether observers know that their agreement is being checked
• if they know their agreement is being checked, they may show higher agreement

Question 9

what are the general guidelines for assessments?

Answer 9

• check agreement at least a few times in each phase (ex. baseline, intervention)
• calculate agreement during the project rather than when the project is over to see if any changes need to be made
• ensure the observers work independently from each other
• avoid conveying expectations to the observers about what the data should show

Question 10

what are evaluating interventions?

Answer 10

• whatever measure is selected, it will need to be administered repeatedly for an ongoing basis overtime
• measure should be able to reflect change
• should reflect a continuous dimension, not binary
• measure should make sense for the assessment

Question 11

what is the use of multiple measures?

Answer 11

• problems or subjects often have multiple components that can be measured
• different measurements can yield different results

Question 12

what are the purposes of pre and post intervention assessments?

Answer 12

often easily obtained and can elaborate or expand on the operational definition of the observation measures.

Question 13

what is a probe assessment?

Answer 13

• measures that are administered occasionally (rather than continuously) in other settings or at other times than those in which the intervention is conducted
• probes test whether there are other changes than those that served as the intervention focus
• probes see if whether the behavior carries over to other settings, is maintained, or whether another behavior has changed

Question 14

how does one construct a recording sheet?

Answer 14

• look at sheets other have used
• arrange the sheets in the same way you scan or record the behavior
• use symbols or abbreviations when possible

Question 15

what is observer drift?

Answer 15

• observers gradually depart from the original definitions of behavior
• can happen if observed become more lenient, more stringent, or less consistent in their definitions of behavior

Question 16

what is interobserver agreement?

Answer 16

• the extent to which observers agree in their scoring of behavior
• two people independently score the same person for behaviors at the same time

Question 17

what is the purpose of independent observer/scorer agreement?

Answer 17

• allow us to evaluate the adequacy of definitions
• allows us to evaluate the recording sheets
• allows us to reduce risk of bias
• if not obtained, there could be a problem with the definition, task is too complex for the observer, observation period is too long, one of the observers may be biased

Question 18

what are the most common automated sensors?

Answer 18

• smart phones
• smart watches
• computers

Question 19

what is holistic recording?

Answer 19

• scorers should not know the purpose of the study
• scorers should score samples in a random order without knowing anything about the condition in effect
• scorers should be experts in the area being scored

Question 20

what is recording etiquette?

Answer 20

• have the supervisor introduce you with a statement about your interest in doing research
• entrance and departure should coincide with a natural break or change in activity
• when two or more observers are recording data, they should not talk with each other
• do not talk to the people you are observing
• disguise your interest by varying the apparent objects of your glances

Question 21

what are some concerns with assessments?

Answer 21

correlations.

Question 22

what is an odds ratio?

Answer 22

• examining a correlation between two variables by comparing how often the behavior of interest occurs in the presence and absence of a particular variable
• typically small odds ratios are not important
• large odds ratios, logical rational exists, etc. are important

Question 23

why is treatment integrity important?

Answer 23

• see need to see if the treatment works
• can depend on the motivation of those carrying out the procedures
• supports in place to assist them in learning consistently apply the procedures
• calculated the same way as the introobserbee agreement
• not needed for automated treatments