W9 Reliability

Question 1

What is the definition of reliability?

Answer 1

• It is the consistency of measurements or the absence of measurement error (Atkinson & Nevill, 1998)

Question 2

What are the two types of Error Measurement?

Answer 2

1. Systematic Error
   - Consistent error that biases the true score
   - Won’t impact reliability
2. Random Error
   - Unpredictable error that biases the true score
   - Will impact reliability

Question 3

Can you give three sources of error?

Answer 3

1. Participant
2. Researcher
3. Instrument

Question 4

Can you give a couple of ways to reduce error?

Answer 4

• Test multiply times or repeat measures
• Compare research to more than two other researchers research
• Have a trained researcher to ensure correct use of instrument
• Choice of instrument
• Following protocols
• Having good protocols

Question 5

What are the two types of reliability and what do they mean?

Answer 5

1. Relative reliability
   - The degree to which data maintained its position within the data with repeated measures
2. Absolute reliability
   - The degree to which data varies over repeated measures

Question 6

• What is test retest reliability?

- What test could you use to test retest reliability?

Answer 6

• It is the reliability/stability across measurement occasions
• Correlate the scores obtained from a group of participants from two or more occasions
• The test you could use to compare the relationship of two or more scores is the Pearson’s Correlation Coefficient

Question 7

What is inter-rater reliability?

Answer 7

• Inter-rater reliability is the reliability/consistency between researchers
• Correlate the scores from the same group between different researchers

Question 8

• What is internal consistency?
• How do you measure internal consistency?
• What do the values range between on this measurement tool?

Answer 8

• It is the reliability across different areas of measurement instruments
• The way it is measured is using Cronbach’s alpha reliability coefficient
• The measures range between 0 & +1.0

Question 9

What are the four ways to measure absolute reliability?

Answer 9

1. Technical error of measurement
2. Standard error of measurement
3. Coefficient of variation
4. Limits of agreement

Question 10

What is the definition of validity?

Answer 10

It describes the degree to which a test or instrument measures what it is meant to measure

Question 11

What are the four types of measurement validity?

Answer 11

1. Face validity
2. Content validity
3. Construct validity (Convergent & Discriminant)
4. Criterion validity (Concurrent & Predicative)

Question 12

Validity can either be…

Answer 12

…internal or external.

Question 13

Can you describe what Face validity is?

Answer 13

• The method of measuring data obviously involves collecting the factor being measured.
  E.g. 100m sprint - time to complete measured by timing gates. So on the face of it timing gates are good measures of speed/validity

Question 14

Can you describe content validity?

Answer 14

• The instrument adequately covers the subject of measurement
• Instrument covers all aspects of relevance from population

Question 15

• Can you describe what construct validity is?

- What are the two ways of assessing constructive validity?

Answer 15

• Constructive validity assesses to what extent an instrument accurately measures the hypothetical constructs
  1. Convergent
• Scores on an instrument that are similar to those on another measure
  2. Discriminant
• Scores on an instrument are different from a measurement on a instrument that is measuring another score

Question 16

• Can you describe what citron validity is?
• What are the two types of citron validity?
• What does this tell us?

Answer 16

• Scores on a instrument are related to those on a previously validated measure
  1. Concurrent
• Scores collected at roughly the same time
  2. Predicative
• Citron instrument completed at a later date
• It tells us whether we can predict gold standard measures

Question 17

Relationship between reliability & validity:

- Can an instrument be reliable but not valid?

Answer 17

• Yes, it could be reliability (consistently) measuring the wrong thing

Question 18

Relationship between reliability & validity:

- Can an instrument be valid but not reliable?

Answer 18

• No, if it is measuring what it is supposed to measure it should consistently give the same score.
• So reliability is a necessary but not sufficient condition for validity.

Question 19

What does internal validity mean?

Answer 19

• Refers to the ability to attribute changes in the dependent variable to the manipulation of the independent variable

Question 20

What does external validity mean?

Answer 20

• Refers to the ability to generalise the results of a study to other settings and other individuals

Question 21

Can you name a few threats to internal validity and provide a solution?

Answer 21

• Maturation (Age/growth) = include control group
• Selection bias (participants allocated groups) = randomly allocated
• Expecting specific results = blinding/double blind
• Schedule of testing = counterbalance treatments or familiarisation
• Measurement and equipment = frequent calibration
• Mortality (drop out/with drawl) = account for this within sample size

Question 22

Can you name a few threats to external validity?

Answer 22

1. Reactive or interactive effects of testing
   - Pre-test makes a participant more aware or sensitive to the treatment
2. Interaction of selection bias and the treatment
   - treatment is effective only in the group selected
3. Reactive effects of experimental arrangements
4. Multiple-treatment inference