Validity, Errors, Reliability

Question 1

External Validity

Answer 1

• study design

- extent to which results from a study can be extended to a broader population

Question 2

Internal Validity

Answer 2

• adequacy of measurement

- how well does it measure what is is intended to measure

Question 3

Increasing External Validity

Answer 3

• subject selection ⇒ avoid confounding factors, random selection, recruitment methods
• subject number ⇒ sufficient statistical power to detect differences
• still need quality control measures and judgement about degree to which results can be extrapolated

Question 4

Increase Internal Validity

Answer 4

• study design

- data collection techniques ⇒ statistical analysis procedures

Question 5

Components of Internal Validity

Answer 5

• face validity
• content validity
• criterion validity
• reliability

Question 6

Face Validity

Answer 6

• judgement (often by experts) that the tool measures what it is supposed to, expert opinion, can be objective

Question 7

Content Validity

Answer 7

• are all components of the construct measured by the methods being used?

Question 8

Criterion Validity

Answer 8

• how well the tool performs compared with another tool such as a gold standard

Question 9

Reliability / Precision

Answer 9

• degree to which related measures give the same value

- almost impossible to achieve at the individual level

Question 10

Reliability is affect by….

Answer 10

• true variation in measurement
• random error / random variation
• systemic errors

Question 11

Safeguards to ensure reproducibility

Answer 11

• standardized protocols
• calibrate instruments
• suitable protocols for situation
• test-retest and take average

Question 12

Ensure Internal Validity, consider:

Answer 12

• reliability (standardized protocols, calibration, suitable protocols, test-retest)
• get expert opinion on method (face validity)
• conceptualize content of contract and choose methods accordingly (content validity) look at literature
• compare between and among methods (criterion)

Question 13

Measurement errors (2)

Answer 13

Random
Systematic (bias)

Question 14

Random Error

Answer 14

• unpredictable error
• affects reproducibility (precision) of method
• caused by true variation + measurement error
• reduces statistical power of studies
• variation but the average is about accurate (more error = weaker association)

Question 15

Reduce Random Error

Answer 15

• standard operating procedures
• train all examiners and maintain standardized measurement techniques
• select carefully and standardize all instruments
• refine and standardize questionnaires
• control confounding factors via same training, same procedures, same technique and instrument

Question 16

Systemic Error (bias)

Answer 16

• measurements consistently depart from the true value
• bias either negative or positive
• always under/overestimate due to technique or tool

ex - under/overestimating daily intake - average will not be true value
ex - if scale isn’t calibrated

Question 17

Types of Systemic Error (3)

Answer 17

⇒ person-specific bias individual level - someone is reporting their dietary intake or weight in a bias way
⇒ constant additive error - something that affects the entire population (tool)
⇒ intake-relative bias - differences when all underestimating but some more than others

Question 18

confounding factor

Answer 18

• special type of bias (influences validity)
• a characteristic or variable that is distributed differently in the study and control groups, and affects the outcome being assessed

Question 19

ways to control confounding factors

Answer 19

• randomization, stratification, recruitment, restriction, matching

Question 20

Measurement Error Sources

Answer 20

• respondent error ⇒ compliance, random or systematic
• interviewer bias ⇒ work harder on Monday, may not ‘click’ with some subjects, promote “healthy eating”
• respondent memory lapse ⇒ use memory aids, longer time period, minor, condiments etc not remembered
• incorrect estimation of portion size ⇒ asking someone average rather than estimating exact

Question 21

Additional sources of error

Answer 21

• account for supplement use?
• bioavailability of nutrients?
• processing losses/gains?
• foods not present in databases
• honest vs desirable responses
• food served vs food consumed

Question 22

Coding and Computer Errors

Answer 22

• poor description of foods
• inaccurate conversions from estimates
• may not have appropriate tables for food composition data
• systematic mistakes when entering data