Reliability and coefficient alpha Flashcards
What is reliability?
Reliability is the desired consistency or reproducibility of test scores?
- A measure of the extent of error present in a test.
- The degree to which a test produces consistent results under similar conditions
Do we assume there is always some error in measurement?
Yes and that error is random.
What do we assume leads to the differences in a person’s score on a test?
Measurement error. It is unlikely that a person’s true score will change every time they take a test.
What do we expect the distribution of scores to be for a test?
Normal distribution.
What four assumptions underlie classical test theory?
- Each person has a true score we could obtain if there was no measurement error.
- There is measurement error, but this error is random.
- The true score of an individual doesn’t change with repeated applications of the same test even though their observed score does.
- The distribution of random errors (and thus observed test scores) will be the same for all ages.
What is the domain sampling model?
- It is another central concept of classical test theory.
- If we construct a test on something, we can’t ask all possible questions, so we only use a few test items (a sample)
- Using fewer test items can lead to the introduction of error. We need to determine whether the test items adequately sample the domain or construct.
What is the point of reliability analysis?
Reliability analysis is conducted to ascertain how much error we would make by using a score from a shorter test as an estimate of someone’s true ability.
What are three things to note regarding reliability analysis?
- Reliability= variance of observed score on short test/variance of true score.
- Observed test scores should be correlated with true score.
-As the sample gets larger, estimate is more accurate - It is easy to work out reliability if we have the true score.
What can affect reliability measurements?
- Different ways of measuring reliability are sensitive to measurement error.
- We consider various sources of measurement error.
What is “Standard Error of Measurement”?
- We can workout how much measurement error we have by working out how much, on average, an observed score on our test differs from the true score.
- We know that a person’s observed score differs from their true score, and that their true score is unknowable. But we can calculate the range in which a person’s true score should fall by calculating the Standard Error of Measurement.
What is the formula for standard error of measurement?
SEM= SD(sqrt)(1-r)
- SD of the scores
- r is the reliability of the test.
What do we do once we know the SEM?
- We can use it to create confidence intervals.
- The z-score for a 95% confidence interval= 1.96
- Lower bound= x-(1.96*SEM)
- Upper bound= x+(1.96*SEM)
where x is the person’s score on the test.
What are the different types of reliability?
- Test-retest reliability
- Parallel forms reliability
- Internal consistency (split-half reliability, Kuder-Richardson 20 reliability, coefficient/Cronbach’s alpha)
- Inter-rater reliability
What is test-retest reliability?
- The simplest way to establish reliability is to administer the test or
scale to a sample on two different occasions. If the scale is reliable,
the scores at the test and retest administration should be strongly
correlated. - The correlation between the 2 scores is also known as the coefficient of stability.
- The source of error measured is time sampling.
What are the issues with test-retest reliability?
- What is the optimal length of time that should elapse between the administrations? If it is too soon, the participants may recall their answers from the first administration. If left too long, extraneous
events may influence the scores on the scale. - There are issues around using it when measuring things that are more transient like mood.
- ## What if some event happens in between first and second administration?
What is parallel form reliability?
Alternate-forms reliability requires the construction of two equivalent versions of the same test, which have items that are closely matched. Then the two forms are administered to the same set of people either at different times or at the same time.
- The correlation between the two forms is known as the coefficient of equivalence
- The source of error measure is item sampling.
How do you change the form of a test for parallel forms reliability?
- Question response alternatives are reworded.
- Order is changed (this reduces practice effects)
- Can change the wording of the question.
What are the issues with parallel forms reliability?
- A problem with the alternative forms method is that it is both difficult and expensive to produce alternate forms that are sufficiently independent and similar.
- Difficult to generate a big enough item pool.
What is inter-rater reliability?
- Measures how consistently 2 or more rater/judges agree on rating something.
- Multiple raters for measurement can improve measurement reliability.
- Could do this by correlating raters’ scores.
- This method does not factor in the number of times raters are correct by chance.
What are the 2 different calculations used for inter-rater reliability?
- Cohen’s kappa is used when there are 2 raters or judges.
- Fleiss’ kappa is used when there are more than 2 raters or judges.
- Ranges from 1 (perfect agreement) to -1.
- >0.75 excellent agreement
- 0.50-0.75 satisfactory agreement
- 0.40 poor agreement
What is internal consistency reliability?
- It asks “do the different items within one test measure all the same thing to the same extent?”
- Are items within a single test highly correlated?
- The source of error measured is the internal consistency/reliability of one test administered on one occasion.
What are the three different ways of measuring internal consistency?
- Split-half reliability
- Coefficient alpha
- KR-20 which is a special case of co-efficient alpha for the dichotomous format.
What is split half-reliability?
- A test is split in half, each half is scored separately, total scores for each half are then correlated to determine whether they yield similar measures.
- A major advantage is that we only need one test.
- A major challenge is diving the test into equivalent halves.
- Calculated using Spearman-Brown.
What are two of the major issues with split-half reliability?
- The fewer items we have, the lower our reliability (reference to the domain sampling model)
- Therefore, each half of split test will have reduced reliability compared to the total test. - Dividing the test into equivalent halves: the correlation will change for each different split. Ideally the halves should be equivalent.
What is the solution to the first issue with split-half reliability?
- The Spearman-Brown formula.
- It is a more accurate reflection of internal consistency as it adjusts for the number of items.
What is Cronbach’s alpha?
- Cronbach’s alpha estimates the consistency of responses to different scale items.
- It measures the error associated with each test item as well as the error associated with how well the test items fit together (internal consistency)
How does the number of items in a test influence Cronbach’s alpha?
- There is a positive, non-linear correlation between the number of items and reliability.
- There is a rapid increase in internal consistency reliability from 2 to 10 items.
- Steady increase from 11 to 30.
- Tapers off after about 40 items.
What are the different levels at which we interpret Cronbach’s alpha?
- 0.00= no consistency in measurement
- 1.00= perfect consistency in measurement
- 0.70= this is the acceptable value for exploratory research.
- 0.80= this is the acceptable level for basic research.
-090= acceptable level for applied scenarios.
What factors affect cronbach’s alpha?
- Multidimensionality- Cronbach’s alpha is better suited to unidimensional data.
- Bad test items- the greater the number of bad items, the more negatively alpha is impacted.
- The number of test items- increases reliability to a certain point.
Recap of reliability
look at slides.
What is coefficient alpha commonly described as in the literature?
- The mean of all split-half reliabilities.
- The lower bound of the reliability of a test
- A measure of first-factor saturation
- Equal to reliability in conditions of essential tau-equivalence
- A more general version of the KR coefficient of equivalence.
- These are all wrong ? Well I know 1 and 3 are.
What do we know about covariance and split halves reliability?
- When we work out split-half reliability, we are taking into account the correlation between two groups of items.
- If we are splitting a test in every possible way, knowing all the covariances between items is useful. This will tell us how much an item in one group will correlate with an item in another group.
What is important to consider when splitting a test for split-halves reliability?
- We need to know how the items fit together.
What is first factor saturation?
- It is the extent to which one factor is present in a test. i.e the extent to which a test is made up of one factor (as opposed to 2 or 3)
- A factor is a dimension of a test.
What else do I need to know about factors?
- Factors are test dimensions, or aspects that are measured by a test.
- Some test measure one thing, e.g. BDI. Some measure more than one thing e.g. the big 5 personality test.
Why do people have this misconception about alpha?
- There is confusion between the terms internal consistency and homogeneity.
What is a misconception people have about the relationship between cronbach’s alpha and unidimensionality?
- There is a misconception that alpha measures the extent to which 1 factor is present in a test. i.e people think that the higher the level of alpha, the more likely it is that a test is made up of one factor.
What is the difference between internal consistency and homogeneity?
Internal consistency
- This is how inter-related items are
- The more the items are inter-related, the more internally consistent the test is.
Homogeneity
- Refers to unidimensionality (a test having only one factor)
- Something that is homogenous is made up of only one thing. E.g. if everyone in the class was female, the class would have homogeneity of gender
Why do people confuse internal consistency with homogeneity/unidimensionality?
- Alpha measures how well items fit together in terms of how much co-variance (shared variance) the items have)
- According to common sense, the more co-variance items have, the more they should fit together to make up one thing. i.e. they should measure one factor only. A high level of alpha should therefore mean that your test is measuring one factor.
What is the problem with the common sense view?
- Internal consistency and unidimensionality are not actually the same thing.
- Research shows that tests can have many different factors, but still have high levels of alpha. E.g. The WAIS-III comprises of two factors (VIQ and PIQ) but still has a high level of alpha. another example is the resiliency scale of Wagnield and Young (1993). It also has two factors (Self-motivation and optimism). The overall alpha for this is 0.90.
Why do these multidimensional tests have high levels of alpha?
- This is because they measure things that are related to each other- as well as to some overall thing.
- The WAIS measures VIQ and PIQ – both types of Intelligence, which the overall scale measures.
- The Resiliency Scale measures Self-motivation and Optimism – both related to each other, and both influence Resilience
- So they all have high levels of alpha because they are all largely internally consistent. Meaning that they measure things that are related to each other, so the items share a lot of covariance.
Summary of Alpha and ! factor confusion
- Alpha is sometimes said to be a measure of the extent to which a test measure 1 factor.
- This is because alpha is a measure of internal consistency, and the more internally consistent the test, the more covariance the items share
- The more covariance the items share, the more they are supposedly part of the same factor. But we know that this is not always the case.
- However, tests that measure only one factor often have high levels of alpha, as they are very internally consistent.
- If you are going to construct a test with more than one factor, you need to make sure that both factors are related to each other as well as to the overall thing you are trying to measure.
