SCD IOA

Question 1

Interobserver Agreement (IOA)

Answer 1

The degree to which two or more independent observers report the same observed values after measuring the same events.

Question 2

Benefits and Uses of IOA

Answer 2

1. Determines the competence of new observers. 2. Detects observer drift. 3. Judge clarity of definitions and system. 4. Increase believability of data

Question 3

Requisites for obtaining valid IOA - Observers must:

Answer 3

• Use the same observation code and measurement system. - Observe and measure the same participants and events. - Observe and record independently of one another

Question 4

Total Count IOA

Answer 4

(Smaller Count/ Larger Count) x 100 = Total Count IOA %. Used in event recording. Not very stringent. Total count recorded by each observer per measurement period.

Question 5

Mean Count-per-Interval IOA

Answer 5

(Int 1 IOA + Int 2 IOA + Int n IOA)/n intervals x 100 = Mean Count per Interval IOA%.

*Used in event recording.

Question 6

Exact Count-Per-Interval IOA

Answer 6

(# of intervals of 100% IOA)/n intervals x 100 = Exact count- per-interval IOA%.

Used in event recording.
Most stringent of event recording
- percentage of total intervals in which two observers recorded the same count.

Question 7

Trial by Trial IOA

Answer 7

(# of Trials of Agreement)/Total # of Trials X 100 = Trial by Trial IOA%. Used for event recording. More conservative and meaningful index of IOA. - compares the observer’s counts on a trial-by-trial basis

Question 8

Total Duration IOA

Answer 8

(Shorter Duration/Longer Duration) x 100 = Total Duration IOA%. Used in data obtained by timing. Not very stringent. Appropriate when recorded as a dependent variable.

Question 9

Mean Duration-Per-occurrence IOA

Answer 9

(Dur IOA R1 + Dur IOA R2 + Dur IOA Rn)/ #responses wth Dur IOA x 100 = Mean Duration-per-interval IOA. Used in data obtained by timing. Also used to compute mean latency-per response IOA or mean IRT-per-response IOA.

Question 10

Interval by interval IOA

Answer 10

(# of intervals of agreement of occurrence or non- occurrence) / (# of intervals of agreement of occurrence or non-occurrence + # of intervals of disagreement of occurrence or non-occurrence) X 100 = Interval by interval IOA%. Used in interval recording/time sampling. Likely to overestimate the actual agreement between observers measuring behaviors that occur at very high or very low rates.

Question 11

Score Interval IOA

Answer 11

of intervals of agreement of occurrence/# of intervals in which either or both observers recorded occurrence x 100 = Scored interval IOA%. Used for interval recording/time sampling. *Used for Behavior that occurs at low rates.

Question 12

Unscored Interval IOA

Answer 12

of intervals of agreement of non-occurrence/# of intervals in which either or both observers recorded an occurrence x 100 = Unscored Interval IOA%. Used in interval recording/time sampling. *Used for Behavior that occurs at high rates.

Question 13

How often should IOA be obtained?

Answer 13

IOA should be obtained for a minimum of 20% of a study’s sessions, and preferably between 25% and 33% of sessions.

Question 14

Which method of calculating IOA should be used?

Answer 14

More stringent and conservative methods of calculating IOA should be used over methods that are likely to overestimate actual agreement.

Question 15

What are acceptable levels of IOA?

Answer 15

IOA should be 90% or greater for an established measure and at least 80% for a new variable. The smaller the change in behavior across conditions, the higher the criterion should be for an acceptable IOA percentage.

Question 16

Considerations in IOA

Answer 16

Obtain and report IOA at the same levels at which researchers will report and discuss in study results. - For each behavior. - For each participant. - In each phase of intervention or baseline

Question 17

Reporting IOA

Answer 17

• Narrative form. – Table. – Graphs. - In all formats, report how, when, and how often IOA was assessed.

Question 18

Assessing the Quality of Measurement

Answer 18

Indicators of the quality of data include: 1. IOA. 2. Accuracy. 3. Reliability.

Question 19

Interval Agreement IOA

Answer 19

Total # of agreements divided by total # of agreements plus disagreements times 100. - Commonly used

Question 20

Interval Agreement IOA Advantages:

Answer 20

more precise than total agreement, allows for estimation of how 2 observers score behavior, increases likelihood that you are documenting that observers recorded same event.

Question 21

Interval Agreement IOA Disadvantages:

Answer 21

may not adequately represent whether observers agreed on the occurrence or nonoccurrence of behavior.

Question 22

Occurrence/Nonoccurrence Agreement

Answer 22

Data recordings are compared on an interval-by-interval basis by scoring each interval as agreement or disagreement. Sum the agreements, sum disagreements, divide the number of agreements by disagreements, plus agreements and multiply by 100%. This is done separately for each. **Advantage: most rigorous, and most preferred approach