Reliability

Question 1

What is reliability?

Answer 1

it is the “accuracy or precision of a measuring instrument”

Question 2

Theory of Reliability

Answer 2

“Any obtained score is made of two components, a “true” component and an error component (Xtotal = Xtrue + Xerror). Therefore total variance is also made up of true variance and error variance (Vtotal = Vtrue + Verror). This leads to an equation/definition,
where the reliability coefficient is equal to the total variance minus the error variance all over
the total variance (rttl = (Vtrue - Verror)/Vtrue)).

Question 3

How to know effect size is important?

Answer 3

Prentice, D.A. & Miller, D.T. (1992)
- Strategy 1: Look for when “the most minimal manipulations of the independent variable still
accounts for some variance in the dependent variable”
- Strategy 2: Find a dependent variable variable that is seriously difficult to influence, since it’ll
be novel and impressive if you’re able to influence it

Question 4

Process for creating a measurement

Answer 4

Hinton, T.R. 1998

1. Item generation: Need a “well-articulated theoretical foundation”; Things to consider: Deductive vs. Inductive scale development, number of items, item
   scaling etc.
2. Questionnaire administration: Present the scales to people - suggested to survey several independent samples and also administer more established scales alongside
3. Initial item reduction: Refine the new scale- this is based both on theory and the quantitative results
4. Confirmatory factor analysis: Helps to provide evidence of construct validity
5. Convergent Discriminant Validity: Does the scale correlate with other measures which assess similar constructs? Does the scale correlated with dissimilar measures?
6. Replication: With this iterative process, continue testing the scale with new and independent samples, and not just with people who are conveniently located, but from a group that can “provide evidence for construct validity”

Question 5

Alternatives to Cronbach’s Alpha

Answer 5

1) Omega coefficients - “There are multiple variations of omega including omega hierarchical, omega total, and what we will refer to as “Revelle’s omega total.” They can also each yield different results for variation.
2) Coefficient H - appropriate for scales that are optimally weighted (each item contributes unique (not the same) amount to the scale - then you should use a measure for maximal reliability (coeff H)
3) Greater Lower Bound - extends the classical test theory computation can be challenging, and it may overestimate reliability with smaller sample sizes.

Question 6

What is Cronbach’s alpha?

Answer 6

a measure of internal consistency, that is, how closely related a set of items are as a group

Question 7

5 methods of assessing reliability

Answer 7

1) Test & Re-Test – Look at either the correlation of each pair of values per subject, or compare the rank of subjects between test and retest.
2) Test A, Test B (equivalent or parallel forms) – Develop two different but statistically similar (same mean and std. dev.) versions of the test and have each subject take both. Compare the two score for each person.
3) Split Half Reliability – Splitting the test results in half and comparing the sums of the two halves, which should be equal.
4) KR-20 and KR- 21 – Based off of the split half concept, it applies a formula to assess the
correlation between subsamples of the data.
5) Coefficient α – An extension of KR20 & KR40, it is much more widely applicable.

Question 8

Which methods of assessing reliability are more conservative and which ones are less conservative?

Answer 8

1) Test & Re-Test – Not good if:
- Attrition is high
- Organisms go through dramatic developmental change between T1 and T2.

2) Test A, Test B – Advantage: minimizes attrition, no concern of subjects remembering responses. Disadvantages:
- Need to be measured for a longer period of time, causing fatigue and boredom which could lower the reliability coefficient.

3) Split Half Reliability: Longer test more reliable, and shorter less reliable
- Underestimate reliability since it’s correlation of two halves of the test

4) KR-20 and KR-21: Applicable to measuring instruments with a dichotomous or binary scoring
system. Most conservative estimate of reliability.

5) Coefficient α – Most generally used measurement. Allows for instruments using a Likert scale
to be assessed – most frequently used

Question 9

Ways to Improve Reliability

Answer 9

• MAX- MINCON (Kerlinger, 1986)- maximize variance of individual differences (btwn group variance) and minimize error variance (within-group variance)
• Make sure ppl understand items in scale
• Add more items, which balances random error
• Give clear and standard instructions