Chapter 5 Imporving and Assessing the Quality of Behavioral Measurement

Question 1

Data

Answer 1

primary material to guide and evaluate behavioral work

Question 2

3 indicators of a trustworthy measurement

Answer 2

Validity
Accuracy
Reliability

Question 3

What are the three elements for validity.

Answer 3

1. directly measuring socially significant target behavior.
2. Measuring dimension of the target behavior relevant to the question or concerning about the behavior.
3. Ensuring that the data are representative of the behavior’s occurrence under conditions during times that are most relevant to the concern about the behavior.

Question 4

If any of the 3 elements of validity are lacking even if the data seems valid and reliable, what happened to the data?

Answer 4

The validity of the resultant data are compromised, even meaningless.

Question 5

When does measurement have validity?

Answer 5

Measurement has VALIDITY when it yields data that are directly relevant to the phenomenon measured and to the reasons for measuring it.

Question 6

Validity if threaten when

Answer 6

a. indirect measurement
b. measuring the wrong dimension of the target behavior
c. measurement artifacts

Question 7

Indirect measurement

Answer 7

when what is actually being measured is in some way different from the target behavior of interest.
- secondhand or filtered info

Question 8

Measuring the wrong dimension of the target behavior

Answer 8

more threatening to validity than indirect measurement.

ex. sticking a ruler to measure water but it does not tell you about temperature.

Question 9

Measurement artifacts (AKA?)

Answer 9

AKA: artifacts

- Giving a misleading picture of the behavior because of the way measurement was conducted.

Question 10

Causes of Artifacts

Answer 10

1. Discontinuous Measurement: describes any form of measurement in which some instances of the response class of interest may no be detected.
   Ex. time sampling methods
   
   • poorly scheduled measurement periods
   • insensitive and/or limiting measurement scales: scales that imposes an artificial floor or ceiling on performance.

Question 11

Accuracy

Answer 11

The extent to which the observed value (i.e.,quantitative label) matches the true state or, TRUE VALUE of the event, as it exists in nature. For something to be considered a TRUE VALUE required special or extraordinary precautions that ensure that all possible sources of error have been removed.

Question 12

When is measurement accurate?

Answer 12

When observed values (numbers obtained by measuring an event) match the true values of the event.

Question 13

If measurement is not valid, accuracy is ____.

Answer 13

moot

Question 14

Accuracy trumps _______.

Answer 14

reliability

Question 15

Establishing True values

Answer 15

Compare observed values to true values.

Question 16

What happens when true values cannot be established?

Answer 16

Researchers must rely on reliability assessment and measure of inter observer agreement to evaluate the quality of their data.

Question 17

Reliability

Answer 17

The extent to which a measurement procedure yields the same value when brought into repeated contact with the same state of nature.

• consistent measurement
• the closer the values obtained by repeated measurement, the more reliable they are.

Question 18

What does poor reliability mean?

Answer 18

Means problem with accuracy.

Question 19

Threat to measurement accuracy and reliability?

Answer 19

Human error: biggest threat

Question 20

Factors that Contribute to Human Measurement Error?

Answer 20

1. Poorly designed measurement systems
2. Inadequate observer training
3. Expectation about what the data should look like

Question 21

Poorly Designed Measurement Systems

Answer 21

• difficult and cumbersome measurement systems
• # of participants to observe, number of behaviors recorded, duration of observation period, &/or duration of observation intervals

Question 22

Inadequate Observer Training

Answer 22

Observers need explicit and systematic training:

1. Select observers carefully.
2. Train observers to a standard of competency
3. provide ongoing training to minimize observer drift.

Question 23

Observer Drift

Answer 23

When observers unknowingly alter the way they apply a measurement system.

Question 24

When does observer drift occur?

Answer 24

When observers have a shift in how they interpret the definitions of the target behavior.