chapter 5 Flashcards
Reliability of Measures
- Reliability of Measures: refers to consistency or stability of measure (same or very similar results every time used)
- Measurement error: every time you measure something, you capture the true score and bits of error. The more reliable a measure is, the less measurement error present
- Crucial for any operational definition
Measurement error formula
X= T+E
X= The measurement we collect
T= The true measurement (idea situation with no error)
E= error (difference between measured score and true score)
- Unsystematic errors: motivation mood testing environment
- Measurement error: test issue
Reliability and Accuracy of Measures
- A measure can be highly reliable
but not accurate - Reliability indexes indicate amount
of error but not accuracy
Achieving Reliability
- Train observers well: ex. observational study on aggression in hockey - before sending them in, must clarify what aggression looks like
- Word questions well: make sure participants understand what is being asked of them
- Calibrate and place equipment well: Test equipment under different conditions so you can get consistent readings
- Observe construct multiple times: more items= more reliable
Reliability vs. Validity
Reliable: consistent in measurement
Validity: actually accurate asses variable of interest
Risk of attenuation (weakening results)
* Solutions:
* Use measures that have established reliability (predetermined measures)
* Use well-trained coders
* Do separate studies just on the reliability of the measure (study the studies)
What are the three ways we test reliability
- INTERNAL CONSISTENCY
- TEST-RETEST RELIABILITY
- INTERRATER RELIABILITY
- each item is a repeated attempt to test a concept
- Cronbach’s alpha
Consistency of the items on a multi-item measure that measure the same variable - Questionnaires and rating scales
- Split-half reliability (take first half of items on measure, then compare them to second half)
- Measured by a reliability coefficient: (compare how each item correlates on score with each of the other measures)
- Item-total correlations
Correlation Coefficients
- Asses stability of measure by using correlation coefficients
- look at how strong items of a mesure are
Test retest reliability
-Testing same people in same conditions at two points over time
- Measure is given at two time points and scores are compared to measure consistency
* High positive correlation coefficient shows two sets of scores very similar
* Measured by correlation coefficient (r)
* Alternate forms reliability
* May appear higher than it is if people remember previous answers
- You need at least 0.7 or 0.8
Issue? If you administer the same test twice, people can remember answers from the past.
Solution?Do half of the test at a time
Types of reliability: interrater reliability
- how much do the raters agree
- what is the average % of agreement
Book summary
What are the three types of validity
How well does the mesure mesure what its supposed to measure
- Focuses on measurement accuracy
*CONSTRUCT VALIDITY
*INTERNAL VALIDITY
*EXTERNAL VALIDITY
Construct validity
How well the measures represent the variable(s)
* A reflection of the quality of a researcher’s operational
definitions
* Good construct validity: hunger
* Not so good construct validity: How well they read menu
Different ways to look at construct validity
Face validity (based on theoretical): The extent to which the content of the measure appears to reflect the construct being measured (give it to an expert or non-expert, ask them to tell you what they think it’s for)
Content validity (based on theoretical): The extent to which the entire set of items represents all aspects of the
topic and nothing else; The breadth of the instrument. Does the survey capture everything, or are we missing something?
- Judged by an expert
Predictive validity (based on constructs): extent to which the measure predicts future behaviours or situations it would predict.
Established:
Collect data
* After time, measure & compare with
your measure;
* correlate or see how accurately first
measurement predicted criterion
Concurrent Validity: The extent to which the measure relates to a criterion behaviour that occurs at the same time as the measurement.
(behaviours related to construct of interest)
How established:
take two tests at same time… correlate them
* Collect data with your measure
* Compare to current criterion:
correlate or see how accurately
measure identifies criterion
Convergent Validity : See if measures relate to other measures of similar constructs
Met if they are related
Convergent delivery: show that measures that should be related are related
