EBP2 Correlation and Reliability Flashcards
What kind of data does correlation and regression deal with?
continuous
What is the question to ask during correlation?
strength of association
What is the question to ask during regression?
strength of prediction
Describe correlation study.
pairs of scores and whether they covary
how strong is their linear relationship? what is the nature of the relationship?
Correlations quantify strength of what type of relationship?
linear
What is used to quantify strength of correlation?
correlation coefficients
What is the range of correlation coefficients?
0 and +/- 1.00
closer to 1.00 = higher strength of relationship
sign indicates direction
tighter grouping means higher coefficient
An r of 0.00-0.25 generally signifies what kind of relationship?
little or no
An r of 0.26-0.50 generally signifies what kind of relationship?
fair
An r of 0.51-0.75 generally signifies what kind of relationship?
moderate to good
An r of 0.75-1.00 generally signifies what kind of relationship?
good to excellent
What is the coefficient of determination?
square of the correlation coefficient (r^2)
is the percent of variance in y that is explained (or accounted for) by x
(amount of overlap between two circles)
Describe the null hypothesis and significance of coefficient.
null hypothesis: correlation between variables is not significantly different from zero
significance means likely not 0 in population but not strong
very sensitive to sample size
What is the most common type of correlation coefficient and briefly describe it?
pearson product-moment correlation
both variables continuous (interval or ratio)
What is the non-parametric analog of Pearson r and briefly describe it?
Spearman rank (rho) (rs)
1 continuous, 1 ordinal or 2 ordinal variables
What type of correlation is used when one variable is dischotomous and the other is continuous (interval or ratio)?
point biserial correlation (rpb)
True/False: Point biserial correlation has the same results as t-test.
true
t test will only give statistical significance where rpb also gives strength or relationship
Describe Rank Biserial correlation (rrb).
one variable is dichotomous, other is ordinal
computationally about the same as Spearman Rank
Results same as Mann-Whitney U-Test
What correlation is used when both variables are both dichotomous?
phi coefficient
What correlation coefficient is used when both variables are continuous?
pearson r
What correlation coefficient is used when one variable is continuous and one is ordinal?
spearman rho
What correlation coefficient is used when both variables are ordinal?
spearman rho
What correlation coefficient is used when one variable it continuous and one it nominal?
point biserial
What correlation coefficient is used when one variable is ordinal and the other is nominal?
rank biserial
What correlation coefficient is used when both variables are nominal?
phi coefficient
True/False: Correlation does not assess differences or agreement.
true
ICCs do
True/False: Causation equals correlation,
False - both may be caused by another factor
causation statements come from controlled experiments (RCTs)
True/False: Can’t generalize beyond range of scores in sample.
true
What is reliability?
the extent to which a measurement is consistent and free from error
a reliable measure can be expected to repeat the same score on two different occasions provided that the characteristic of interest doesn’t change
What is used to quantify reliability?
reliaility coefficients
What are the reliability coefficients for continuous (interval/ratio) data?
Pearson correlation (r)
*Intraclass correlation coefficient (ICC)
What are the reliability coefficients for discrete (ordinal/nominal) data?
Percent agreement
*Kappa
What are the downfalls to pearson’s r reliability coefficient?
assesses relationship, not agreement
only two raters or occasions could be compared
so they can have perfect correlation but not agree
What is the formula for reliability coefficients?
rxx = true score variability / (true score variability + error variability)
Score _ = no reliability,
Score _ = perfect reliability
0, 1
The _ the true score variability reduces reliability.
less
The _ error variability, reduces reliability.
more
Which of the following is not true for ICC?
a. measures degree of relationship (association) and agreement
b. can only be used with one rater or ratings
c. designed for interval/ratio
d. Often reported in conjuction with SEM
b - can be used with >2 raters or ratings
ICC gives _ estimate of reliability.
SEM gives _ estimate of reliability.
standardized(no units), unstandardized (in units)
What are the factors that ICC type depends on?
purpose of study, design of study, types of measurements taken
ICC type is defined by two numbers in parentheses.
What position in parentheses describes the model? form?
first, second
What is a model 1 for ICC?
each subject measured by different set of raters randomly chosen
What is a model 2 for ICC?
each subject measured by same raters “randomly” chosen and representative of rater population
results are generalizable
most common for inter-rater reliability or test-retest reliability
What is a model 3 for ICC?
each subject measured by same rater(s) of interest so not generalizable
most common for intra-rater reliability
What does the form in the ICC type signify?
number of observations used to obtain reliability estimate
1 observation per subject rater = 1, 3=3, 4=4, etc
What does an ICC score of > 0.90 generally mean?
best for clinical measurements
What does an ICC score of >0.75 generally mean?
good reliability
What does an ICC score of < 0.75 generally mean?
poor- moderate reliability
ICC estimate us based on _ measures.
average
True/False: ICCs should always include a 95% CI.
true
In ICCs, P values tests whether point estimate is _.
statistically different from 0
Describe percent agreement
how often raters agree 0%-100%
divide # of agreements by total of all possible agreements
based on frequency table
agreements on diagonal, disagreements are all others
True/False: Percent agreement account for agreement due to chance.
false - it does not
it tends to overestimate reliability
Describe Kappa coefficient.
proportion of agreement between raters after chance agreement has been removed
used on nominal and ordinal data
Describe weighted Kappa.
can choose to make penalty worse for larger disagreements
weights can be arbitrary, and symmetric or asymmetric
best for ordinal data
Generally, a kappa value of <0.4 indicates what about agreement beyond chance?
poor to fair
Generally, a kappa value of 0.4-0.6 indicates what about agreement beyond chance?
moderate
Generally, a kappa value of 0.6-0.8 indicates what about agreement beyond chance?
substantial
Generally, a kappa value of 0.8-1.0 indicates what about agreement beyond chance?
excellent
What is the most commonly applied statistical index for internal consistency?
cronbach’s alpha
What is internal consistency?
reflection of the correlation among items and the correlation of each individual item with the total score
True/False: Cronbach’s alpha can be used for dichotomous or ordinal (multiple choice) data.
true
What is the recommended range for Cronbach’s alpha?
between 0.70 to 0.90
If too high, probably redundant. If too low, possibly measuring different traits
What is response stability?
stability of repeated measures over time
~test-rest reliability
What are the three commonly used statistical methods to express response stability?
Standard error of measurement, Minimal detectable difference/chance, coefficient of variation
What is the standard error of measurement?
measure of reliability using standard deviation of the measurement errors
SD x square root of 1-ICC
can be used to create a 95% CI around a measurement (95% CI = Score +- 1.96(SEM)
What is minimal detectable difference?
amount of change in a variable that must be achieved to reflect a true change/difference
mathematical multiple of SEM
MDC = mean +- (1.06)(SEM)(sqrt2)
What is coefficient of variation?
ratio of SD to mean, expressed as percentage
CV = (SD/mean) x 100
What are alternate forms?
comparing different methods of testing same phenomenon with different instruments
What is Limits of Agreement?
range that includes ~ 95% of differences
95%LOA = X +- 2s
Describe a Bland-Altman plot.
spread of scores around a 0 point to help decide if the observed error is acceptable if we substitute one measurement method for another
If all on one side of 0, consistently measuring more
tighter range = more agreement
What question does correlation seek to answer?
Is there a strength in the association between variables?
True/False: You are reading a study that has shown a strong correlation between hours studying (greater than 3 hours) for an EBP II exam and getting a 90% or above (r = .94). We can now infer that studying for 3 or more hours will cause one to get a 90% or above on the exam.
False - correlation does not equal causation
ICCs are primarily used with dichotomous or categorical data.
continuous data (ratio/interval)
You are wanting to use a reliability process that is the most conservative approach. What would be most appropriate if this were your goal?
model 1 ICC
You are reading a study that is reporting ICC values and note that the following was reported “ICC(2,1)”. What does this tell the reader?
Each subject measured by same raters using an inter-rater approach where only one observation per subject per rater was performed.
You are wanting to assess for the percentage of between raters who are performing the Lachman’s test on those suspected of an ACL tear. The raters are rating their findings as either positive or negative. What would most appropriately assist with this?
Kappa
You come across a Kappa value of .54. How would you interpret the magnitude of this value?
moderate agreement
True/False: SEM is a measurement that takes into account ICC.
true
