Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

Research Stats > Week 6: Reliability, Validity, Epidemiologic Analysis and Dichotomizing Treatment Effect > Flashcards

Week 6: Reliability, Validity, Epidemiologic Analysis and Dichotomizing Treatment Effect Flashcards

1
Q

What is reliability?

A

Extent to which a measurement is consistent and free from error

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

All reliability scores have…

A

signal and noise

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

What is signal?

A

true score

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

What is noise?

A

error

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Reliability is the ratio of…

A

signal to noise

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

relative reliability

A

ratio of total variability of scores compared to individual variability within scores
unitless coefficient
ICC and kappa

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

absolute reliability

A

indicates how much of a measured value is likely due to error
expressed in the original unit
SEM is commonly used

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

Standard error of measurement (SEM) for relative measure of reliability

A

ICC (and kappa)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

Standard error of measurement (SEM) for absolute measure of reliability

A

SEM

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Most common types of reliability

A

test-retest, inter-rater, intra-rater, internal consistency

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

inter-rater

A

2+ or more raters who measure the same group of people

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

intra-rater

A

the degree that the examiner agrees with himself or herself
2+ measurements on the same subjects

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

in measurement validity, the test should…

A

discriminate, evaluate, and predict

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

reliability is a __________ for validity

A

prerequisite

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

content validity

A

establishes that the multiple items that make up a questionnaire, inventory, or scale adequately sample the universe of content that defines the construct being measured

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Criterion-related Validity

A

establishes the correspondence between a target test and a reference or ‘gold’ standard measure of the same construct

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

concurrent validity

A

the extent to which the target test correlates with a reference standard taken at relatively the same time

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

predictive validity

A

the extent to which the target test can predict a future reference standard

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

construct validity

A

establishes the ability of an instrument to measure the dimensions and theoretical foundation of an abstract construct

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

convergent validity

A

the extent to which a test correlates with other tests of closely related constructs

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

divergent validity

A

the extent to which a test is uncorrelated with tests of distinct or contrasting constructs

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

quantifying reliability: ‘old approach’

A

pearson’s r
assesses relationship
only 2 raters could be compared

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

Quantifying reliability: ‘modern’ approach

A

intraclass correlation coefficients (ICC)
cohen’s kappa coefficients
both ICCs and kappa give single indicators of reliability that capture strength of relationship plus agreement in a single value

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

ICC

A

values from 0 - 1.0
measures degree of relationship and agreement
can be used for > 2 raters
interval/ratio data

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
25
Q

ICC types

A

six types depending on purpose, design, and type of measurements

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
26
Q

ICC type is defined by

A

two numbers in parentheses
ex: ICC (2,1), ICC (model, form)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
27
Q

model 1

A

raters are chosen from a larger pop; some subjects are assessed by different raters (rarely used)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
28
Q

model 2

A

each subject assessed by the same set of raters

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
29
Q

when is model 2 used

A

for test-retest and inter-rater reliability

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
30
Q

model 3

A

each subject is assessed by the same set of raters, but the raters represent the only raters of interest

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
31
Q

when do you use model 3

A

used for intra-rater reliability or when you do not wish to generalize the scores to other raters

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
32
Q

ICC forms

A

second number in parentheses represents number of observations used to obtain reliability estimate

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
33
Q

form 1

A

scores represent a single measurement

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
34
Q

form k

A

scores based on mean of several (k) measurements

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
35
Q

ICC interpretation

A

no absolute standards

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
36
Q

ICC > 0.90

A

best for clinical measurements

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
37
Q

ICC > 0.75

A

good reliability

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
38
Q

ICC < 0.75

A

poor to moderate reliability

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
39
Q

cronbach’s alpha (a)

A

represents correlation among items and correlation of each individual item with the total score
between 0.70 to 0..90

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
40
Q

if cronbach’s alpha is too low it means

A

not measuring the same construct

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
41
Q

if cronbach’s alpha is too high it means

A

redundancy

42
Q

agreements of reliability for categorical scales are

A

diagonal

43
Q

disagreements of reliability for categorical scales are

A

all other parts of the table

44
Q

percent agreement

A

simply how often raters agree
range of 0% to 100%

45
Q

kappa coefficient

A

proportion of agreement between raters after chance agreement has been removed
can be used on both nominal and ordinal
can be interpreted like ICC

46
Q

weighted kappa

A

best for ordinal data
can choose to make ‘penalty’ worse for larger disagreements
weights can be arbitrary, symmetric or asymmetric

47
Q

kappa = <.4

A

poor to fair

48
Q

kappa = .4 - .6

A

moderate

49
Q

kappa = .6 - .8

A

substantial

50
Q

kappa = .8 - 1.0

A

excellent

51
Q

what does a diagnostic test do

A

focuses the examination
identify problems
assist in classification

52
Q

diagnostic test are all about

A

probabilities and limiting uncertainty

53
Q

pre-test probability

A

before any testing takes place

54
Q

post-test probability

A

outcome of the test

55
Q

clinical prediction rules (CPR)

A

combinations of clinical findings
predictions
quantifies the contributions of a set of variables to diagnosis, prognosis, and likely response to treatment

56
Q

concurrent validity is

A

sensitivity/specificity
correlation coefficients

57
Q

predictive validity is

A

correlation coefficients
regression

58
Q

what is sensitivity

A

proportion of people WITH the disease who have a positive test result
LEFT COLUMN, TOP BOX out of 100

59
Q

what is specificity

A

proportion of people WITHOUT the disease who have a negative test result
RIGHT COLUMN, BOTTOM BOX out of 100

60
Q

SpPin

A

test with HIGH specificity
Positive
helps rule a condition IN

61
Q

SnNout

A

test with HIGH sensitivity
Negative
helps rule a condition OUT

62
Q

PPV (positive predictive value)

A

patients with positive tests divided by all patients with positive test results

63
Q

NPV (negative predictive value)

A

patients with negative tests divided by all patients with negative test results

64
Q

Likelihood ratios

A

quantifies the test’s ability to provide persuasive information
NOT influenced by prevalence
ranges from 0 to infinity

65
Q

LR is 0 - 1

A

decreased probability of disease/condition of interest

66
Q

LR = 1

A

no diagnostic value; null value

67
Q

LR > 1

A

increased probability of disease/condition of interest
farther from 1 = more likely

68
Q

LR+ =

A

sensitivity/(1-specificity)

69
Q

LR- =

A

(1-sensitivity)/specificity

70
Q

diagnostic test is positive = what likelihood ratio

A

LR+

71
Q

diagnostic test is negative = what likelihood ratio

A

LR-

72
Q

LR+ : > 10

A

large and often conclusive shift

73
Q

LR+ : 5 - 10

A

moderate shift

74
Q

LR+ : 2 - 5

A

small: sometimes important

75
Q

LR+ : 1 - 2

A

small: rarely important

76
Q

LR- : < 0.1

A

large and often conclusive shift

77
Q

LR- : 0.1 - 0.2

A

moderate shift

78
Q

LR- : 0.5 - 0.2

A

small: sometimes important

79
Q

LR- : 0.5 - 1

A

small: rarely important

80
Q

case-control and cohort studies are…

A

intended to study risk factors
association between disease and exposure

81
Q

what is an example of a exposure?

A

cervical manipulation, smoking, running > 20 mi/wk

82
Q

what is an example of disease or outcome?

A

cancer, stroke, knee OA

83
Q

cohort studies subjects are selected based on

A

exposure

84
Q

cohort studies are usually

A

prospective but can be retrospective!

85
Q

case-control studies are selected based on

A

whether or not they have a disorder

86
Q

case-control studies are usually

A

retrospective

87
Q

Relative Risk is in

A

cohort studies (two ‘o’s’)

88
Q

Odds ratios are in

A

case-control studies (a and o, ‘at odds’)

89
Q

RR or OR = 1

A

null value
no association between an exposure and a disease
if 1 is in CI then it is not significant

90
Q

RR or OR > 1

A

positive association between an exposure and a disease
exposure is considered to be harmful

91
Q

RR or OR < 1

A

a negative association between an exposure and a disease
exposure is protective

92
Q

experimental event rate

A

% patients in experimental group with bad outcome

93
Q

control event rate

A

% patients in control group with bad outcome

94
Q

number needed to treat

A

how many patients you have to provide treatment to in order to prevent one bad outcome
- the closer to 1 the better

95
Q

Number Needed to Treat (NNT)

A

NNT = 1/ARR

96
Q

if NNT = 1.0 it means

A

need to treat 1 patient to avoid one adverse outcome

97
Q

if NNT = 10 it means

A

need to treat 10 patients to avoid one adverse outcome

98
Q

do we want a small or big NNT?

A

SMALL

99
Q

Number needed to harm (NNH)

A

NNH = 1/ARI
(ARI = EER - CER)

100
Q

if NNH is 1.0 it means

A

we need to treat 1 patient to cause an adverse outcome

101
Q

if NNH is 10 it means

A

we would need to treat 10 patients to cause an adverse outcome

102
Q

do we want a small or big NNH

A

BIG

Research Stats (4 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2025 Bold Learning Solutions. Terms and Conditions