Reliability

Question 1

Who was Francis Galton?

Answer 1

childhood genius, Charles Darwin, human differences; first person to apply statistical methods in the differences between humans; found that the mean of OX weight guesses was the closest to the OX’s actual weight, even though the median would be best because it takes out extreme scores, which ended up accounting for each other as seen in the mean; plotted the scores and showed a bell curve. Came up with the idea that “error cancels out”

Question 2

What is Galton’s Ox?

Answer 2

guessing the weight of the OX; law of errors: the average of guesses is correct

Question 3

What is reliability?

Answer 3

reliability refers to the degree to which test scores are free from errors of measurement; the accuracy of a mean score from an unreliable test depends on how they are distributed, e.g., if they are unevenly distributed it’ll be harder to tell the true score

Question 4

What is the goal of psychological measurement?

Answer 4

Detect psychological differences. To be reliable.

test scores are used to indicate levels of psychological attributes; differences among people’s test scores are used to indicate true psychological differences among people; to what degree are differences in observed (test) scores consistent with differences in (true) levels of psychological attributes?

Question 5

What is Classical Test Theory?

Answer 5

x (observed) = T (true ability) + e (random error)

Question 6

What are the assumptions of classical test theory?

Answer 6

1. Observed scores on a psychological measure are determined by a respondent’s true score and by measurement error
2. Error is random (consequences: 1. Errors tends to cancel itself out across respondents 2. Errors scores are uncorrelated with true scores)

classical test theory says X=T+E; E(X)=T; if a person were to take a test repeatedly then the average would cancel out any error (testing is independent from one test to another AND error is random and independent) e.g., people guessing Ox’s weight didn’t powwow beforehand

Question 7

What is signal and noise referring to?

Answer 7

true score is “signal”; measurement error is “noise,” obscuring the signal; observed score is affected by signal and noise; reliability = signal / (signal + noise); if a person takes a test on two different occasions under the same conditions we would expect the same result: 1. That the conditions under which the test was taken were exactly the same 2. That the person’s underlying true score did not in fact change; it’s about how closely observed (X) approach true scores (T); X=T

Question 8

How do you estimate reliability?

Answer 8

Repeated Measures: logic: if the test was not reliable, then you would expect to see a difference in a person’s observed score, such that the score from Time 1 would be slightly different than the score from Time 2 score; if the test conditions were not the same then any unreliability might be due to the difference in conditions and not the unreliability of the test itself; it is possible that scores from both T1 and T2 could both be very poor estimates of the person’s true score for reasons other than the test; taking a mean of repeated measures is believed to be better estimate of the person’s true ability: sometimes the error would increase the score, sometimes decrease it, but after a while, the score would average out

Question 9

What is the relationship between error on different test scores?

Answer 9

Errors in the test score (E) will not be correlated with the test score (X)—errors should not be correlated on whether a person scored high or low on a test; correlation of E and X=0 (this can happen is error is random and independent); errors from two different tests are uncorrelated (parallel tests—errors from one test will be uncorrelated to true score on another test), conditions of test administration have to be exactly the same (very unlikely, however)

Question 10

What is the goal of reliability theory?

Answer 10

estimate errors in measurement to suggest ways of improving tests so that errors are minimized; errors are random across a large number of individuals; variance of obtained scores = Sx^2=ST^2+Se^2, V observed score=V true score+V error score

Question 11

What is variance?

Answer 11

reflects the extent to which individuals differ (compared to the test mean); it is determined by the degree to which the scores in the group actually differ

Question 12

How does random error impact distribution?

Answer 12

distributions with error have higher variance compared to scores with no random error, however mean stays the same as error cancels out

Question 13

What is the reliability coefficient?

Answer 13

the ratio of the true score variance to the total test of test scores; the proportion of the true variance to the total test of scores, that’s due to/ accounted for in all the variability in the test scores. Variance of true scores over variance of observed scores( true scores plus error).

Question 14

What are the two type of error?

Answer 14

1. Random error
2. Systematic error
   X = T + e (er random error and es systematic error)

Question 15

What is systematic error?

Answer 15

Any bias; the systematic error impacts the average, so we call it a bias

Question 16

What are some ways you can reduce measurement error?

Answer 16

pilot testing (making sure items are clear, etc), thorough training (for administrators), double-check the data, statistical correlation (any adjustment made to data due to knowing there’s a bias and accounting for it), multiple measures (give multiple measures of the same construct, e.g., create new measurements of depression, want to make sure it wasn’t biased, could give old/reliable tests on depression to make sure participant is depressed)

Question 17

Can we be sure about the reliability of a set of test scores?

Answer 17

NO!
reliability is defined by some terms we can’t truly know, we never truly know the reliability of a set of test scores; however, we might be able to estimate reliability, based on the information we do; various methods of estimating reliability; all involve administering two (or more) testings to a sample of respondents (two versions of the same test, one test administered at 2 times, dividing one test into two or more parts)

Question 18

What are four ways to estimate reliability?

Answer 18

1. Test-retest reliability
2. Parallel-forms reliability
3. Internal consistency reliability
4. Inter-rater reliability

Question 19

What is test-retest reliability?

Answer 19

(give the same test twice to one sample of respondents; compute the correlation between scores from the two testing occasions; correlation is an estimate of test’s reliability—can be influenced by reactivity [participants changing reactions to taking a test during taking it; history of taking the test, participants can be in different moods; test-retest is more of a stable estimate rather than reliability])

Question 20

What is parallel-forms reliability?

Answer 20

(create two versions of the test so don’t need to worry about history effects/learning; both forms measure construct equally well; same as alternate forms; internal consistency is difficult to achieve between tests as how do you choose which items belong in the test; administer both to a sample of respondents [must fit the parallel tests CTT model]; compute the correlation between scores from the two forms; correlation is an estimate of the test’s reliability; error should be uncorrelated while the true score should be the same; hard to get enough items and to randomly divide them so they are equivalent between tests as some questions tap into the constructs better than others)

Question 21

What is internal consistency reliability?

Answer 21

(many varieties of internal consistency estimates of reliability; most well-known is alpha [a, aka Chronbach’s alpha]; average inter-item correlation [how well a set of slightly different items measure the same underlying construct equally well], Chronbach’s alpha [takes every possible split-half and a correlation of it all, all the ways all of the items are related to each other], split-half reliability [how well one half of the test is correlated with other randomly chosen half])

Question 22

What is inter-rater reliability?

Answer 22

(degree of agreement among raters; how much homogeneity or consensus exists in the ratings given by various judges; Cohen’s kappa: developed to account for the possibility that raters actually guess on at least some variables due to uncertainty, range of -1 to +1)

Question 23

What is Cohen’s kappa?

Answer 23

developed to account for the possibility that raters actually guess on at least some variables due to uncertainty, range of -1 to +1; test of inter-rater reliability

Question 24

What impacts the accuracy of each reliability estimate method?

Answer 24

the accuracy of each method depends on the validity of key assumptions; each method depends on different sets of assumptions; sets of assumptions are referred to as measurement models

Question 25

What is Chronbach’s alpha? What are the steps for computing?

Answer 25

imagine a 6-item test; are the test’s parts consistent with each other? Do people with high scores on one item tend to have high scores on the other? Steps 1. Compute inter-item covariances and sum 2. Compute variance of total scores 3. Compute alpha

Question 26

What is the problem with alpha?

Answer 26

the accuracy of any estimation method depends on assumptions; alpha is appropriate only for tests meeting certain assumptions; concern is that alpha is too often used in data that do not meet the appropriate criteria, thus, many estimates based on alpha may be inaccurate

Question 27

How do you decide which reliability estimate to use? What is each one best for?

Answer 27

Interrater reliability is best for observation; parallel forms reliability is used when you intend to use the two forms to measure the same thing (constraint: must have multiple forms); test-retest reliability is best in most experimental and quasi-experimental designs that use a control group

Question 28

What is reliability of difference scores? Why would you calculate it?

Answer 28

pretest and posttest differences from an intervention; estimating reliability of difference scores (high correlation between the test scores leads to lower reliability of difference score; high reliability of each test, leads to higher reliability of difference score); the meaningfulness of the difference score depends on the degree to which the two tests have similar variabilities

Question 29

What are two ways to improve reliability?

Answer 29

1. Longer tests are better (diminishing returns)
2. Tests with stronger internal consistency are better (use clear, coherent items; identify and eliminate/replace bad items [items that aren’t consistent with the other items)

Question 30

Is it easier to make reliable inferences about variable or stable characteristics?

Answer 30

Stable

Question 31

What is a good alpha value (internal consistency)?

Answer 31

a>.9 Excellent, .9-.8 Good, .8-.7 Acceptable, .7-.6 Questionable, .6-.5 Poor, .5> Unacceptable

Question 32

When are high levels of test reliability necessary?

Answer 32

1. Tests are used to make final decisions about people

2. Individuals are sorted into many different categories based on relatively small individual differences

Question 33

When are low levels of reliability okay?

Answer 33

1. Tests are used for preliminary rather than final decisions
2. Tests are used to sort people into a small number of categories based on gross individual differences

Question 34

What is the relationship between reliability and validity?

Answer 34

Reliability places an upper bound limit on validity; tests are reliable are not necessarily valid; a test cannot be valid if it isn’t reliable