Chapter 5 Imporving and Assessing the Quality of Behavioral Measurement Flashcards
Data
primary material to guide and evaluate behavioral work
3 indicators of a trustworthy measurement
Validity
Accuracy
Reliability
What are the three elements for validity.
- directly measuring socially significant target behavior.
- Measuring dimension of the target behavior relevant to the question or concerning about the behavior.
- Ensuring that the data are representative of the behavior’s occurrence under conditions during times that are most relevant to the concern about the behavior.
If any of the 3 elements of validity are lacking even if the data seems valid and reliable, what happened to the data?
The validity of the resultant data are compromised, even meaningless.
When does measurement have validity?
Measurement has VALIDITY when it yields data that are directly relevant to the phenomenon measured and to the reasons for measuring it.
Validity if threaten when
a. indirect measurement
b. measuring the wrong dimension of the target behavior
c. measurement artifacts
Indirect measurement
when what is actually being measured is in some way different from the target behavior of interest.
- secondhand or filtered info
Measuring the wrong dimension of the target behavior
more threatening to validity than indirect measurement.
ex. sticking a ruler to measure water but it does not tell you about temperature.
Measurement artifacts (AKA?)
AKA: artifacts
- Giving a misleading picture of the behavior because of the way measurement was conducted.
Causes of Artifacts
- Discontinuous Measurement: describes any form of measurement in which some instances of the response class of interest may no be detected.
Ex. time sampling methods- poorly scheduled measurement periods
- insensitive and/or limiting measurement scales: scales that imposes an artificial floor or ceiling on performance.
Accuracy
The extent to which the observed value (i.e.,quantitative label) matches the true state or, TRUE VALUE of the event, as it exists in nature. For something to be considered a TRUE VALUE required special or extraordinary precautions that ensure that all possible sources of error have been removed.
When is measurement accurate?
When observed values (numbers obtained by measuring an event) match the true values of the event.
If measurement is not valid, accuracy is ____.
moot
Accuracy trumps _______.
reliability
Establishing True values
Compare observed values to true values.
What happens when true values cannot be established?
Researchers must rely on reliability assessment and measure of inter observer agreement to evaluate the quality of their data.
Reliability
The extent to which a measurement procedure yields the same value when brought into repeated contact with the same state of nature.
- consistent measurement
- the closer the values obtained by repeated measurement, the more reliable they are.
What does poor reliability mean?
Means problem with accuracy.
Threat to measurement accuracy and reliability?
Human error: biggest threat
Factors that Contribute to Human Measurement Error?
- Poorly designed measurement systems
- Inadequate observer training
- Expectation about what the data should look like
Poorly Designed Measurement Systems
- difficult and cumbersome measurement systems
- # of participants to observe, number of behaviors recorded, duration of observation period, &/or duration of observation intervals
Inadequate Observer Training
Observers need explicit and systematic training:
- Select observers carefully.
- Train observers to a standard of competency
- provide ongoing training to minimize observer drift.
Observer Drift
When observers unknowingly alter the way they apply a measurement system.
When does observer drift occur?
When observers have a shift in how they interpret the definitions of the target behavior.
