Statistical inference

Question 1

Goal of statistics

Answer 1

To be able to make inferences about a population, but you can never truly know about an entire population

Instead, you must use a sample, and ensure that the sample and inferences are actually generalizable to the population

Question 2

Sampling variation

Answer 2

Refers to the inevitable differences between our population and our sample.

We want to minimize this as much as possible in order to have a more accurate sample

Have to think about getting best possible estimate in face of sampling variation

Question 3

Sampling bias

Answer 3

refers to systematic forces with regard to sampling
Ex. only using university students, or people who pick up the phone/agree to do a study over the phone

Question 4

Frequentist statistics

Answer 4

Frequentist statistics asks: how likely is it that this effect of the manipulation would have occurred if there was no effect in the actual population?
Ie. what is the chance this result was due to sample variation?

Is there another reason another reason this effect has arisen? Manipulation vs. random? What would happen if we did this study multiple times
Determining the answer to this question is the point of significance testing

Question 5

What is more common Frequentist vs Bayesian stats

Answer 5

Frequentist statistics is more common: t-tests, p-values, null hypothesis

Question 6

Sampling distribution

Answer 6

generally focuses on the shape of the charted data

Question 7

Central limit theorem

Answer 7

states that for a large enough sample size, the mean of the sample will be…

Normally distributed

Equal to the population mean

Have equal variance to the population variance (if divided by sq root of sample size)

This is true regardless of the distribution of the actual population

Question 8

Standard error

Answer 8

the standard deviation of the sampling error

Will get smaller as the sample size gets bigger, therefore can be used as a measure of sampling error variation/precision

Essentially, standard deviation is a measure of the spread of the data, standard error is a measure of the variation of the sampling error

We cannot actually know the true population distribution, but we can use CLT and standard error to make inferences

Question 9

Confidence intervals:

Answer 9

Using CLT and standard error to infer if sample mean is true to population mean

Through sub-sampling, we are able to make inferences called confidence intervals

After getting the mean of multiple subsamples, you can see what percent of the means fall within one SD on either side of the mean

The confidence error tells us that 95% of repeated measures will fall within the SE of the sample mean

95% CI allows you to estimate that 95% of estimates lie within the specified SD

Bigger sample size equals smaller confidence intervals, because bigger sample size equals smaller SD, therefore the mean is more precise

Question 10

Null hypothesis (h0)

Answer 10

assumes that results obtained are the result of random chance/sampling variation, and not an effect of the manipulation

Question 11

Alternative hypothesis (h1)

Answer 11

assumes the results obtained were due to an effect of the manipulation.

Question 12

p-value and null hypothesis

Answer 12

P-value refers to the probability that a test-statistic this extreme would be obtained given the null is true/by random chance (conditional probability)

If the p-value/the probability of receiving this result by chance is very small, you can reject null hypothesis and assume the result is the effect of the manipulation
I.e., if the p is very small, H0 is probably wrong, so we should reject it

P-value is industry standard/arbitrarily set at 0.05 (if smaller or equal too, usually reject null, if bigger, accept null

Question 13

How to get p-value

Answer 13

favored tool for hypothesis testing

Essentially creates a null population distribution: what would we expect to see if this study was repeated and there is no real effect?

To get p-value, take standard error and calculate how many SE away this is the sample’s mean from the pop mean

Do this with z-score
Z-score = (Samp-mean-pop mean)/SE

The probability of observing a sample with our mean randomly is our p-value.

If smaller than 0.05, this can be considered statistically significant.

Question 14

Type 1 error

Answer 14

False positive ie., reject the null despite there being no real effect

Alpha is the probability of a type 1 error

True positive is 1-beta
False positive is alpha

Question 15

Type 2 error

Answer 15

False negative ie., accept the null despite there being a real effect, can combat through increasing power

Beta is the probability of a type 2 error

True negative is 1-alpha
False negative is beta

Question 16

Alpha

Answer 16

Alpha is the probability of a type 1 error

Question 17

Beta

Answer 17

Beta is the probability of a type 2 error

Question 18

Definition of power

Answer 18

Your ability to find true positive results, also known as sensitivity.

Standard/arbitrarily set at 80%.

Property of a stat test, not a test itself

Low power is a waste of resource because high chance of false negatives, also have a higher chance of overinflated effect sizes (winners curse)

Question 19

Winners curse

Answer 19

small sample size have higher chance of random large effect sizes being flagged as significant)

Question 20

Power is dependent on…

Answer 20

sample size, stat test being utilized, sig threshold, and scale effect size you are looking for

While holding the other three factors constant…
One-sided tests have higher power
Larger samples have higher power
Larger effect sizes have higher power
Higher alpha values have higher power

Question 21

How to calculate sample size needed

Answer 21

We use power to calculate sample size needed, and can reduce alpha and beta with increased sample size

Question 22

One-sided test

Answer 22

when you do not care/are sure about the direction of the test. group mean > or < a given value

Question 23

Two sided test

Answer 23

when you are looking at which direction the effect goes in

When doing a two-sided test, the p-value under the curve dictates what is included in critical regions,

i.e., it dictates what values are above or below the significance.
Areas not included are rejection regions.

Question 24

T-test

Answer 24

Cannot do a z-test because we do not know true SD of population, must do t-test using the sample SD as an estimate

But this t-value doesn’t follow normal distribution, uses t-distribution which is slightly different instead

T-distribution changes depending on degrees of freedom, which is N-1
and accounts for the uncertainty that comes with a smaller sample size

The larger the sample size, the closer the t-distribution gets to normal distribution

Question 25

One sample t-test

Answer 25

compares the sample mean to the given mean value that is already known

Question 26

Two sample t-test or independent groups t-test

Answer 26

Compares means from two separate groups, between subjects and assumes…

Data normally distributed

Observations between and within groups are independent of one and other

Variance should be the same in both groups, but only in students’ t-tests, and we usually use the Welsch t-test, which does not assume this.

Question 27

Paired t-test

Answer 27

compares two means from the same sample, repeated measures/within subjects and assumes…

You are comparing between related groups or matched pairs/measuring the same individual twice

The difference scores should be normally distributed

T-test uses a difference score between variables

Question 28

Cohens d

Answer 28

measure of effect size between two means

d=0.2 is a small effect
d=0.5 is a medium effect
d=0.8 is a big effect

But these values are arbitrary and should not be interpreted rigidly

Interpret in the context of the study

Question 29

R: Power test

Answer 29

pwr.t.test(n= # of participants, d = cohens/effect size, sig.level = #, power = #, type = ‘one.sample’ or ‘two.sample’, alternative = ‘two.sided’)

Question 30

R: Effect size/cohens d

Answer 30

cohensD(group1, group2, method = ‘unequal’)

Question 31

R: QQ plot

Answer 31

qqnorm(dataset$column)

Question 32

R: Independent two-sided T.test

Answer 32

t.test(group1, group2, alternative = ‘two.sided’, var.equal = FALSE)

Question 33

R: Paired t-test

Answer 33

t.test(x= before, y = after, alternative = ‘greater’, paired = TRUE)

Question 34

R: get vector length

Answer 34

length(column)