Categorical analysis 2

Question 1

What should you use if you are comparing two nominal variables?

Answer 1

Chi-square test of association or test of independence

tests if two nominal-scale variables are related to each other

Question 2

What is ‘effect size’?

Answer 2

The outcome of a hypothesis depends on the sample size (larger better)

Question 3

Why is it recommended that an independent measure of effect size be used when reporting a significant statistical effect?

Answer 3

Small treatment effect can be statistically significant if the sample is large enough

Question 4

What does the effect size estimate a metric provide information about?

Answer 4

The size of an effect that is not influenced by factors such as sample size

Measures how ‘big’ the difference between the data and the null hypothesis predictions actually were

Question 5

What does ‘cramer’s v’ do?

Answer 5

Measures effect size in categorical analysis (chi-square)

Question 6

What is the R command for ‘Cramers v’?

Answer 6

associationTest() prints it automatically but can also use cramersV() for it directly

Question 7

How should you roughly interpret Cramer’s V?

Answer 7

Question 8

Why are assumptions necessary in a test?

Answer 8

Necessary to allow inference

If assumptions are wrong though, you can make mistakes

Question 9

Is sampling distribution equal to ‘chi-square’ in chi-square tests?

Answer 9

No, only approximately

Question 10

What assumptions do both chi-square tests (‘goodness of fit’ and ‘association’) make?

Answer 10

‘Large’ expected frequencies

Independence of the data

Question 11

What are ‘large’ expected frequencies an assumption of chi-square tests?

Answer 11

Data only becomes chi-square if we can presume that there are enough observations for the underlying binominal distributions to be ‘normal’

Question 12

What test should you use for comparing nominal variables if frequencies are too small?

Answer 12

Fisher Exact Test

Question 13

What is the Fisher Exact Test?

Answer 13

An analogue of the chi-square test of association

However, it doesn’t require large expected frequencies (works best for small frequencies)

Question 14

What assumptions does the Fisher Exact Test make that the chi-square test of association doesn’t?

Answer 14

It assumes that row and column totals are fixed

(can’t be changed and are the same number)

Question 15

How does the Fisher Exact Test work?

Answer 15

By calculating the exact probability of obtaining a particular contingency table (i.e. cross-tabulation)

• The p-value is calculated by summing over all contingency tables that are “more extreme” than the observed one.*
• The definition of “more extreme” is tricky, but basically means “more uneven”*

Question 16

What is the main thing to note in the Fisher Exact Test when looking at the results?

Answer 16

p-value

Question 17

What does the second assumption of chi-square tests ‘independence of data’ mean?

Answer 17

Can’t have any ‘special relationship’ among some of your observations

(e.g. same people participating in two of the same experiments)

Question 18

What test should you use to analyse categorical data if the two sets of data are not independent of one another?

Answer 18

McNemar test

Question 19

What is McNemar’s ‘limited solution to a standard problem’?

Describe the problem and his solution.

Answer 19

What do you do when you have multiple observations from each person? (e.g. pre-test and post-test)

Can’t use chi-square because this violates the independence assumption

Solution:

You have a binary outcome measure (e.g. yes or no) and you measure it twice (e.g. pre- and post-)

Question 20

McNemar’s test is testing which two ‘cells’ in the cross-tabulation before answers (yes, no) and after answers (yes, no)?

Answer 20

Question 21

How do you do McNemar’s test in R?

Answer 21

allAds <- xtabs(~after+before, data=ads)

mcnemar.test(x=allAds)