Reliability

Question 1

Difference between systematic error and random error?

Answer 1

• Systematic error - Where an error occurs consistently throughout the entire study and remains the same, can include things like measurement error
• Random errors - Occurs randomly due to chance. May be caused by changes in measuring instruments, or misreading the values represented.

Question 2

Can error correlate with anything?

Answer 2

• Error is assumed to be random, therefore it cannot correlate with anything

Question 3

What sources can we derive our values of X from?

Answer 3

• The true score and the error from the study
• Remember true score is theoretical

Question 4

What’s the major goal of determining reliability?

Answer 4

• Must determine the proportion of the observed-score variance that is composed of the true-score variance
• Called the proportion of reliability of measurement

Question 5

What’s the coefficient of determination?

Answer 5

• Explains the proportion of variation that is found in Y and can be explained by X

Question 6

How can a reliability of 0.7 be interpreted?

Answer 6

• Means that the true scores can explain 70% of of variation in the scores

Question 7

T/F: Reliability is a group statistic.

Answer 7

• TRUE
• Tells us how reliable a test is within a group of subjects
• Tells us little about the inconsistency or error to expect in an individual score

Question 8

What does a test with a very large variance indicate?

Answer 8

• Means the test is very reliable since we can encapsulate a higher number of scores an encapsulate more people within a population using one test.

Question 9

What does unrestricted variance usually indicate?

Answer 9

• Usually means the variance found in the general population

Question 10

T/F: Shorter tests are generally more reliable

Answer 10

• FALSE, longer tests are

Question 11

How can determine the change in reliability when we change the number of items found in a test?

Answer 11

• Use the Spearman-Brown formula
• Assume the items added are randomly parallel to the existing items

Question 12

What does it mean when the items in a test are randomly parallel to one another?

Answer 12

• All the items assess the same construct equally but with different prompts
• Also have the same degree of variance

Question 13

What does higher inter-item correlation linked to?

Answer 13

• Higher reliability
• Can use the average item inter-correlation (reliability of single items) to determine the overall reliability of the test (use the Spearman-Brown formula)

Question 14

What does attenuation mean in the context of reliability?

Answer 14

• The correlation between two variables may be attenuated if the reliability of measurement of the variables is not perfect.
• Ex. If both variables have reliabilities of 0.5, can’t expect them to correlate any higher than 0.5

Question 15

Is true score correlation always higher than observed score correlation?

Answer 15

• Yes because there’s less error

Question 16

What’s the test-retest method?

Answer 16

• Giving the same measure to the same respondents but at two different time points
• Then compute r between the two different time points

Question 17

Issues with the test-retest method?

Answer 17

• Effects of memory, may perform better/differently on second attempt due to practice effects
• Real changes over time

Question 18

What’s equivalent forms reliability?

Answer 18

• Have two different but completely parallel tests that are administered to the same respondents at the same time
• Compute r between form A and form B
• Must have same number and type of items, same level of difficulty, same averages and SD

Question 19

What’s a major issue with equivalent forms reliability?

Answer 19

• Can be extremely time-consuming and expensive to develop parallel forms
• Not commonly used because of this hurdle

Question 20

What’s internal consistency reliability?

Answer 20

• When only one form of the test is available
• Indicates the extent to which each item represents an observation of the same thing observed by other test items
• i.e., the degree of homogeneity of items in a scale

Question 21

What is internal consistency reliability a function of?

Answer 21

• The average inter-correlation among items in the scale
• The number of items in the scale

Question 22

WHat’s split-half reliability?

Answer 22

• A form of internal consistency reliability
• Compute correlations between two subset scores from the same test (ex. odd/even-numbered questions)

Question 23

What are the weaknesses of split-half reliability?

Answer 23

• Not good for timed tests (people may skip questions)
• There are a lot of possible split-halves which could produce slightly different reliabilities

Question 24

How can matrices be used to calculate Cronbach’s alpha?

Answer 24

• Unstandardized CA - add up all values in the var/cov matrix
• Standardized CA - add up all the values in the correlation matrix

Question 25

What does Cronbach’s alpha represent?

Answer 25

• Represents the mean of all the possible split-halves reliabilities