Reliability and Validity

Question 1

Reliability

Answer 1

• refers to the replicable nature of research studies/tools

- high reliability does not guarantee scientific validity

Question 2

Testing reliability

Answer 2

• test-retest correlation
• instrument is twice administered to the same population
• 2-14 days in between

Question 3

Cronbach’s alpha

Answer 3

• measures internal consistency of a test by correlating each item with the total score and averaging the correlation coefficients
• it takes values between negative intinity and 1 as maximum but only positive values make sense
• arbitrary cut-off of 0.70 is used to call the evaluated test to be internally consisten

Question 4

Split-half reliability

Answer 4

-refers to splitting a scale into two parts and examining the correlation

Question 5

Interrater reliability

Answer 5

-measured using two or more raters rating the same population using the same scale

Question 6

Intraclass correlation coefficient

Answer 6

• used for continuous variables
• the proportion of total variance of the measurement that reflects true between subject variability
• ranges between 0 (unreliable) and 1 (perfect reliability)
• relative ICC is always higher than absolute ICC

Question 7

ANOVA intraclass coefficient

Answer 7

-used for quantitative data of more than 2 raters/groups

Question 8

Nominal data reliability

Answer 8

-if it has more than two categories then a kappa or weighted kappa can be used

Question 9

Validity of an instrument

Answer 9

-the extent to which an instrument measures what it proposes to measure

Question 10

Face validity

Answer 10

-refers to a subjective measure of deciding whether the test measures the construct of interest on its face value (what it was designed for)

Question 11

Construct validity

Answer 11

-measures whether a test really measures the theoretical construct of interest of something else

Question 12

Content validity

Answer 12

-refers to whether the contents ie each individual subscales, items or elements of the test are in line with the general objectives or specifications the test was originally designed to measure

Question 13

Criterion validity

Answer 13

-refers to performance against an external criterion such as another instrument (concurrent) or future diagnostic possibility (predictive)

Question 14

Concurrent validity

Answer 14

-refers to the ability of a test to distinguish between subjects who differ concurrently in other measures e.g those who score high on a scale of insomnia may score high on the scale of fatigue ratings

Question 15

Predictive validity

Answer 15

-the ability of a test to predict future group differences according to current group differences in score

Question 16

Incremental validity

Answer 16

-refers to the ability of a measure to predict or explain variance over and above other measures

Question 17

Convergent validity

Answer 17

• refers to the agreement between instruments that measure the same construct
• form of construct validity

Question 18

Discriminant validity

Answer 18

• refers to the degree of disagreement between two scales measuring different constructs
• form of construct validity

Question 19

Experimental validity

Answer 19

• refers to the sensitivity to change
• an instrument must show the difference in results when an intervention is carried out to modify the measured domain
• form of construct validity

Question 20

Factorial validity

Answer 20

-form of construct validity established via factor analysis of items in a scale

Question 21

Precision

Answer 21

• degree to which a calculated central value varies with repeated sampling
• the more narrow the variation the more precise the value is
• random error leads to imprecision

Question 22

Factors reducing precision

Answer 22

• wider limits of the interval

- expecting higher confidence interval

Question 23

Accuracy

Answer 23

-the correctness of the mean value

Question 24

Precision

Answer 24

-comparable to reliability while accuracy is comparable to validity

Question 25

Bias

Answer 25

-compromises validity/accuracy

Question 26

Face

Answer 26

-does the scale appear to be fit for the purpose of measuring the variable of interest?

Question 27

Content

Answer 27

-does the scale appear to include all the important domains of the measured attribute?

Question 28

Criterion

Answer 28

-is the scale consistent with what we already know (concurrent) and what we expect (predictive)?

Question 29

Convergent

Answer 29

-does this new scale associate with a different scale that measures a similar construct?

Question 30

Discriminant

Answer 30

-does the new scale disagree with scales that measure unrelated constructs?

Question 31

Kappa

Answer 31

observed agreement beyond chance/maximum agreement beyond chance OR (observed agreement-agreement expected by chance)/(100%-agreement expected by chance)

Question 32

Beyond chance agreement

Answer 32

-Kappa indicates the level of agreement that could be expected beyond chance

Question 33

When is Kappa calculated?

Answer 33

-only for agreement on categorical variables such as presence or absence of a diagnosis

Question 34

Weighted Kappa

Answer 34

-for ordinal variables

Question 35

Bland-Altman

Answer 35

-used for continuous variables where pairs of score differences are plotted against the mean

Question 36

Kappa values and degree of agreement

Answer 36

``` 0= no agreement 0-0.2=slight 0.2-0.4=fair 0.4-0.6= moderate 0.6-0.8=substantial 0.8-1.0= almost perfect ```

Question 37

What is kappa statistics?

Answer 37

- if two investigators independently assess the same group, there will be an extent to which their results 'agree' - simple percent agreement overestimates the degree of agreement and this is misleading. This is why kappa statistics are done - kappa indicates the level of agreement that could be expected beyond chance

Question 38

Interpreting kappa

Answer 38

- affected by the prevalence of the outcome studied -the higher the proportion of positive assessments the higher the kappa - statistical significance cannot be tested directly from kappa measurements but it guides the actual degree of agreement

Question 39

Observed agreement calculation

Answer 39

- draw two by two table - take the two values agreed and add them together - then divide by 100

Question 40

Kappa coefficient

Answer 40

- dependent on the prevalence of the measured condition - common disorders will have low kappa but rare disorders have low Kappa - one must look at the actual percentage afreement

Question 41

Phi

Answer 41

- similar to kappa - all cells are utilised and statistical significance is possible - small sample size can be used