Week 6 & 7 Flashcards
When do you do a non-parametric test?
When the basic assumptions for a parametric test are not met
Non- parametric statistics are based on…?
- Comparisons of ranks of scores
* Comparisons of counts(yes/no) or “signs” of scores
Non- parametric statistics are ___ compared to parametric statistics
Non- parametric statistics are less powerful compared to parametric statistics
What kind of parametric test do you perform when you have 2 independent groups?
Unpaired t-test
What kind of parametric test do you perform when you have 2 related scores?
Paired t-test
What kind of parametric test do you perform when you have 3 or more independent groups?
One-way analysis of variance (ANOVA) (F)
What kind of parametric test do you perform when you have 3 or more related scores?
One-way repeated measures analysis of variance (MANOVA)
What kind of non-parametric test do you perform when you have 2 independent groups?
Mann-Whitney U test
What kind of non-parametric test do you perform when you have 2 related scores?
- Sign test
- Wilcoxon signed ranks test (T)
What kind of non-parametric test do you perform when you have 3 or more independent groups?
- Kruskal-Wallis analysis of variance by ranks (H or x^2)
What kind of non-parametric test do you perform when you have 3 or more related scores?
Friedman two way analysis of variance by ranks
True or False
You’re able to perform a non-parametric test on complex designs like a 2 x 3
FALSE
Unable to perform on more complex designs (e.g. 2x3)
What question is being asked in the comparison based on ranks in a non-parametric t-test?
Is the difference in ranks larger than would be expected by chance alone?
What question is being asked in the comparison based on signs in a non-parametric t-test?
Is the difference in sign frequencies larger than would be expected by chance alone?
What type of test do we use when the IV and DV are both on the nominal level?
Chi- Square
What are you looking at in a chi-square?
Are observed frequencies different than expected frequencies
What are the 2 types of chi square?
- Goodness of fit
* Tests of independence (association)
What do you do in the goodness of fit chi square test?
• Compare observed frequencies of 1 variable to uniform frequencies of another
What is an example of the goodness of fit chi square test?
• Eg: flip coin 50 times. Get 15 heads & 35 tails. Is this difference due to chance or a “real” bias?
____ chi square test is much more common?
Tests of independence (association)
What do you do in the tests of independence (association) chi square test?
Compare observed frequencies from 1 variable to observed frequencies of another variable
What is an example of the tests of independence (association) chi square test?
Eg: Is owning a mac laptop related to gender?
What is the McNemar test?
Requirement of chi-square is that variable levels must be independent (e.g. can’t be “healed” and “unhealed”)
___ is the form of a chi square test that is used for 2x2 with correlated sample
McNemar test is the form of a chi square test that is used for 2x2 with correlated sample
What is a phi coefficient?
A correlation coefficient for 2 nominal variables/ degrees of association for 2x2
The phi coefficient is based off the ___
The phi coefficient is based off the chi-square test
What is the IV level of measurement for a t- test?
Nominal
What is the IV level of measurement for an ANOVA?
Nominal
What is the IV level of measurement for a non parametric test?
Nominal
What is the DV level of measurement for a t- test?
Continuous
What is the DV level of measurement for an ANOVA?
Continuous
What is the DV level of measurement for a non parametric test?
Ordinal
What is the question asked with a t-test?
Difference between means?
What is the question asked with an ANOVA?
Difference between means?
What is the question asked with a non parametric test?
Ranks different?
What is the IV level of measurement for a correlation?
Continuous
What is the IV level of measurement for a regression?
Continuous
What is the DV level of measurement for a correlation?
Continuous
What is the DV level of measurement for a regression?
Continuous
What is the question asked with a correlation?
Strength of association?
What is the question asked with a regression?
Strength of prediction?
What does a correlation have to do with?
A pair of scores and how much they co-vary
What does it mean for something to co-vary?
Directly or inversely proportional. When one is high, so is the other and vice versa
What are the things that a correlation looks at?
- Do they vary together (covary)?
- How strong is their linear relationship?
- What is the nature of the relationship?
A correlation has to be ___
A correlation has to be linear
What is a correlation coefficient?
A number that quantifies the strength of a linear relationship that can range from -1 to 1
What does it mean when a correlation coefficient is closer to 1, whether positive or negative?
Closer to |1.00|, higher strength of relationship
What does the sign of the correlation coefficient indicate?
The direction
The tighter the grouping of the linear relationship, the ___ the correlation coefficient
The tighter the grouping of the linear relationship, the higher the correlation coefficient
What does a 0.00- 0.25 coefficient correlation mean?
Little or no relationship
What does a 0.26- 0.50 coefficient correlation mean?
Fair relationship
What does a 0.51- 0.75 coefficient correlation mean?
Moderate to good
What does a 0.75- 1.00 coefficient correlation mean?
Good to excellent
What is the coefficient of determination?
• The square of the correlation coefficient
What is the coefficient of determination equal to?
The percent of variance in one variable that is explained (or accounted for) by the other variable
What is the significance of the coefficient correlation?
To test the null hypothesis
What is the null hypothesis as it relates to the coefficient correlation?
The correlation between variable x and variable y is not significantly different from zero.
Coefficient correlation is very sensitive to ___
Coefficient correlation is very sensitive to * sample size*
What is the most common type of correlation coefficient?
Pearson Product-Moment Correlation Coefficient (r)
When is the Pearson Product-Moment Correlation Coefficient applicable?
When both variables continuous (Interval or Ratio scale)
What is the Spearman Rank (rho) Correlation Coefficient (rs)?
Non-parametric analog of Pearson r
When is the Spearman Rank (rho) Correlation Coefficient (rs) applicable?
When 1 continuous, 1 ordinal variable or 2 ordinal variables
When do you use a Point Biserial Correlation (rpb)?
When one variable is dichotomous, and the other variable continuous (interval or ratio)
When does a Point Biserial Correlation (rpb) not work?
Doesn’t work with non-dichotomous nominal (e.g Age & Race)
Computationally, a Point Biserial Correlation (rpb) is the same as a ___
Computationally, a Point Biserial Correlation (rpb) is the same as a Pearson’s r
The results of a Point Biserial Correlation (rpb) is the same as ___
The results of a Point Biserial Correlation (rpb) is the same as a t-test
When do you use a Rank Biserial Correlation (rrb)?
When one variable is dichotomous (nominal), and the other variable is ordinal
A Rank Biserial Correlation (rrb) is computationally about the same as ___
A Rank Biserial Correlation (rrb) is computationally about the same as Spearman Rank
When do you use a Phi coefficient (Φ)?
When both variables dichotomous
A Phi coefficient (Φ) is computationally same as ___ (special case)
A Phi coefficient (Φ) is computationally same as Pearson’s r (special case)
A scatterplot is ___ with a Phi coefficient (Φ)
A scatterplot is worthless with a Phi coefficient (Φ)
Can a Phi coefficient (Φ) work with a non- dichotomous nominal?
NO
A Phi coefficient (Φ) is similar to a ____, but unlike it, a Phi coefficient (Φ) gives gives strength of relationship, while the ___ only gives statistical significance
A Phi coefficient (Φ) is similar to a chi square test, but unlike it, a Phi coefficient (Φ) gives gives strength of relationship, while the chi-square test only gives statistical significance
A correlation does not tell you ___
Does NOT assess differences or agreement
How can an extreme outlier affect the interpretation of a correlation?
Can create inflated correlation with only a few extreme data points
Can a correlation data be generalized beyond the range of scores in the sample?
Can’t generalize beyond range of scores in sample
Low correlation may be due to ___ range
Low correlation may be due to limited range
What is reliability?
Extent to which a measurement is consistent and free from error
What can a reliable measurement be expected to do?
A reliable measure can be expected to repeat the same score on two different occasions provided that the characteristic of interest does not change
Reliability is closely tied to the concept of ___
Reliability is closely tied to the concept of measurement error
What are the continuous data reliability coefficients?
- Pearson correlation (r)
* Intraclass correlation coefficient (ICC) (best)
What are the discrete/ categorical data reliability coefficients?
- Percent agreement
* Kappa (best)
What are the problems with using a Pearson correlation (r) to quantify reliability?
- Assesses relationship, not agreement
2. Only two raters or occasions could be compared
Why do we prefer to use ICCs and Kappa for quantifying reliability?
Both ICCs and kappa give single indicators of reliability that capture strength of relationship plus agreement in a single value
____ is stated in terms of variance
Reliability coefficients is stated in terms of variance
What is the range of a reliability coefficient and what does it mean?
Range 0-1
0 = no reliability, 1 = perfect reliability
The more error variability you have, the ____ reliability coefficient will be
The more error variability you have, the lower your reliability coefficient will be
Reliability coefficient will be bigger, when ___ is larger
Reliability coefficient will be bigger, when true variance is larger
What is the equation for the reliability/ correlation coefficient?
True score variability divided by true score variability plus error variability
What does a high error variability do to correlation coefficient?
It will reduce it
What will not having enough true score variability do to correlation coefficient?
It will reduce it
What will happens to correlation coefficient with a large true variance?
It will be bigger
What are the things that an ICC measures?
Measures degree of relationship (association) and
agreement simultaneously
ICCs give ____ estimate of reliability (can compare different things)
ICCs give standardized estimate of reliability
ICC is often reported in conjunction with ____
ICC is often reported in conjunction with * Standard error of the measurement (SEM)*
ICC is designed for____ data but can be used with ___ data
ICC is designed for interval/ ratio data but can be used with ordinal data
When can can ICC be used with ordinal data?
If intervals “assumed” to be equivalent
SEM gives ____ estimate of reliability (i.e. in units
of measurement)
SEM gives “unstandardized” estimate of reliability (i.e. in units of measurement)
The 6 types of ICC dependent on ….?
- Purpose of study
- Design of study
- Type of measurements taken
ICC type defined by ___
ICC type defined by two numbers in parentheses
What does each number in the parenthesis of an ICC type mean?
The first number is the model and the second number is the form. (2, 6) 2 = model, 6 = form
How many models of ICC are there?
3
What is model 1 of an ICC?
- Each subject measured by a different set of raters; raters “randomly” chosen
- Rarely used in clinical research
What is model 2 of an ICC?
• Each subject measured by same raters; raters “randomly” chosen & representative of rater population; results generalizable
What is ICC model 2 commonly used for?
Most common for inter-rater reliability or test-retest reliability
What is model 3 of an ICC?
• Each subject measured by same rater(s); raters are only ones of interest; results not generalizable
What is ICC model 3 commonly used for?
Most common for intra-rater reliability
Rank the models of ICC in order from most conservative to least conservative
- Model 1 (most conservative, lowest number)
- Model 2 (neutral)
- Model 3 (least conservative, highest number)
When can a model ICC be used for inter rater reliability?
Can be for inter-rater reliability if study raters only ones of interest
What does the form/ 2nd number in parenthesis of an ICC represent?
Second number in parentheses represents number of observations used to obtain reliability estimate
When is form = 1?
If only one observation per subject per rater (or rating)
When is form a number more than 1?
If multiple observations averaged to get single number for analysis, form = number of observations averaged
What ICC is best for clinical measures?
• ICC > 0.90
What ICC has good reliability?
ICC > 0.75
What ICC has poor to moderate reliability?
ICC < 0.75
The interpretation of an ICC depends on ____
The interpretation of an ICC depends on intended use
ICC estimate based on ____ will always be substantially higher than estimate based on ____
ICC estimate based on average measures will always be substantially higher
than estimate based on single measure
What are the characteristics of reliability for categorical scales?
- Based on frequency table
- Agreements on on diagonal
- Disagreements are all others
What is percent agreement?
How often the raters agree
How do you calculate percent agreement?
Divide number of agreements by total of all possible agreements
What is the problem with a percent agreement?
- Does not account for agreement due to chance
* Tends to overestimate reliability
What is the kappa coefficient?
Proportion of agreement
between raters after chance agreement has been removed
On what kind of data is a kappa coefficient used?
Can be used on both nominal and ordinal data
What does a weighted kappa do?
Can choose to make “penalty” worse for larger disagreements
What can the weight of a weighted kappa be?
Weights can be arbitrary, and
symmetric or asymmetric
A weighted kappa is best for what kind of data?
Best for ordinal data
The kappa interpretation depends on ____
The kappa interpretation depends on the weights used
What does a kappa value of <0.4 mean?
Poor to Fair agreement beyond chance
What does a kappa value of 0.4–0.6 mean?
Moderate agreement beyond chance
What does a kappa value of 0.6–0.8 mean?
Substantial agreement beyond chance
What does a kappa value of 0.8–1.0 mean?
Excellent agreement beyond chance
Internal consistency is often used to do what?
Often used to construct and evaluate scale / questionnaires
What does internal consistency estimate?
Estimate how well the items that reflect the same construct yield similar results. So, do different questions measure same concept or indicator?
What does cronbach’s alpha (a) do?
Represents correlation among items and correlation of each individual item with the total score
What is recommended that cronbach’s alpha be between?
Recommended that cronbach’s alpha be between 0.70 to 0.90
Cronbach’s alpha can have ___ or ____ on test/questionnaire
Cronbach’s alpha can have dichotomous or multiple-choice responses on test/questionnaire
What can cronbach’s alpha (a) help eliminate?
Can help eliminate items from test/questionnaire that are not homogenous to the set or are not contributing unique information
What is response stability?
A way to quantify stability of repeated measures over time
Response stability is basically the same as ___
Response stability is basically the same as test-retest reliability
What are the different ways to test response stability?
- SEM: standard error of the measurement
- MDC: minimal detectable difference/change
- CV: coefficient of variation
Standard error of measurement is a ___ measure of reliability, while ICC and kappa is a ____ measure of reliability
Standard error of measurement is a absolute measure of reliability, while ICC and kappa is a relative measure of reliability
SEM is in units of _____
SEM is in units of measurement as variable
What is SEM theoretically?
Standard deviation of the distribution of theoretical multiple measurements
An SEM can be used to create a ____
An SEM can be used to create a 95% CI around a measurement
What is the MDC?
Amount of change in a variable that must be achieved to reflect a true change/difference
___ is a mathematical multiple of SEM
MDC is a mathematical multiple of SEM
What is the coefficient of variation (CV)?
A standardized way to measure variability. (SD divided by the mean times 100)
What is the coefficient of variation helpful in comparing and why?
Unit-less, so is helpful comparing variability between two distributions on different scales
What is an alternate form reliability?
Comparing different methods of testing same phenomenon with different instruments (goniometer vs inclinometer)
What analysis or agreement is seen with an alternate form reliability?
- Limit of agreement
- Bland- altman analysis
What is a bland- altman plot?
When you plot the mean of two measures on the x- axis and the difference between the 2 measures on the y- axis, and the center of the plots is a bias
What does a tighter range on the bland altman plot mean?
There is more agreement between the two measures
When is there no bias on a bland altman plot?
When the line of bias is at 0
When is there a consistent bias on a bland altman plot?
When the points on the plot are on one side of the bias line
When is there an asymmetrical bias on a bland altman plot?
When the points are split between the two sides of the bias line
