Week 10 Reliability & Validity of Measures Flashcards

Still part of PCA & Factor Analysis but separated it for ease of learning

1
Q

What are the types of reliability?

A

*Test-retest reliability -
*Internal consistency reliability
- Split-half reliability
(use Coefficient alpha or Cronbach’s alpha)
- Inter-rater reliability (use Cohen’s Kappa)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

What types of validity are there?

A
Face Validity
Criterion Validity
Construct Validity
Content Validity
External versus Internal Validity
Convergent versus Discriminant Validity
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

How do we measure Test-Retest Reliability?

A

Test-retest measures temporal reliability.
*This can be obtained by checking the consistency of measurement on the value on at least 2 occasions of at least .70 correlations

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

Tell me more about Test-Retest Reliability

A
  • “Stability” is measured using Test-Retest Reliability. *Assessed by having an instrument (measure) completed by the same people during two different time periods
  • Strong positive correlations (r’s) between the repeated test times are sought.
  • Difficulties may arise because of familiarity with the test items when completed on subsequent occasions – best to do this at least two weeks apart. Then “alternate form reliability” may be used with two reliability of the two sets of scores
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

How do we measure Internal consistency reliability?

A

Internal consistency refers to the extent to which items in a measuring instrument or scale are all measuring the same thing.

  • It requires only one administration.
  • Internal consistency is usually reported with Cronbach’s Alpha.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

Tell me more about Internal Consistency Reliability

A
  • How well items “hang together” Also called reliability of components, it is the degree of relatedness of individual items on a test.
  • Split-half reliability – Spearman-Brown formula is frequently used as a measure of split-half reliability. Average inter-correlation of all items, the mean of item-to-item Pearson (r) correlation
  • Coefficient alpha or Cronbach’s alpha corresponds to the mean of all the split-half coefficients resulting from different splittings of a test
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

How do we test inter-rater reliability?

Probable exam Question

A
  • Cohen’s Kappa tests Inter-Rater Reliability
  • Undertaken using a cross-tabulations of scores where the value of Cohen’s Kappa is produced to evaluate 2 raters.
  • The output identifies where the two raters agree and gives the percentage of agreement on each scale rated.
  • Cohen’s Kappa is evaluated and “ideally .70 and above is required but in any clinical context it should be .8 or above” (Hills, 2008, p.288), as well as significance levels of whether differences emerge between the raters.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

What is the relationship between Reliability and External Validity?

A
  • External validity is the generalizability of causal relationships across people, settings, manipulations (not just time).
  • e.g. is learning and cognition for college students the same for the general population? or is a Biomedical experiment on male volunteers transferable to women or non-volunteers?
  • Standard treatment of phenomenon – is it the same for similar treatments?
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

What is Face Validity?

A

Face Validity asks if the test on the surface (face) measures what it purports to measure

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

What is Content Validity?

A

Content Validity evaluates whether there is adequate sampling of relevant material on the test that it purports to measure, that is does the test represent the kind of material it is measuring, for example, IQ.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

What is Criterion Validity?

A

Criterion Validity is based on the degree to which the test is correlated to the expected outcome criteria – statistical aspects of the test.
*Researchers choose the most sensitive and relevant present criterion (Concurrent Validity) or future (Predictive Validity) on which the test is based upon and correlate the performance of the test or questionnaire with that criterion

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

What is Construct Validity?

A

The degree to which the test assesses what purports to measure. This can be seen as either Convergent Validity or Discriminant Validity.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

What is Convergent Construct Validity?

A

Convergent Construct Validity

  • Convergence of related tests or behaviour
  • Scale on attitudes toward Presidents related to scales on attitudes toward other politicians, like judges and governors
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

What is Discriminant Construct Validity?

A

Discriminant Construct Validity

  • Distinctiveness of unrelated tests or behaviour
  • Scale on attitudes toward Presidents unrelated to scales on attitudes toward older men (even though all previous Presidents have been older men)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

What are the 4 types of validity in experimental design?

A
  • External Validity (“Generalizability”)
  • Construct Validity (Scale measures, what it says it measures)
  • Statistical-Conclusion Validity (Accuracy of drawing certain statistical conclusions)
  • Internal Validity
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

What does Validity of scale refer to?

Need to know differences for the exam

A
  • Validity of a scale refers to the degree to which it measures what it purports to measure. Can be either:
  • Content validity: Measuring adequacy of domain content.
  • Criterion validity: Measuring the relationship between the scale score and some measurable criterion.
  • Construct validity: Tests the nature of the underlying construct (theoretically) of the scale