T-distribution & independent t-test

Question 1

What do we use simulation studies for?

Answer 1

We can’t measure the whole population so we use a sample from that population and we can run a simulation to learn from the statistics of the sample

Question 2

How can we represent the difference between two means using a linear model?

This is just an interesting thing that you can calculate t and p-value with a linear regression model in JASP. But because we are focusing on the t-test, we don’t need to know how you get the linear equation. I included the steps but I don’t think we need to practice this since it gives the same info as a t-test

Answer 2

We include a two-category predictor and we can see, using JASP, the t-statistic and p-value which show us whether one category is significantly different from this other

How to do it in JASP?
1. After loading the dataset, on the top click on Regression > Classical > Linear regression
2. Drag the corresponding variable into dependent variable and the independent variable into factors
3. Watch the magic happen
4. Under the coefficients table you see the M1 (interectept) the t and the p values

Question 3

We can’t measure the whole population - how do we solve this issue?

Answer 3

We collect samples from the population to estimate the mean and the standard deviation of the population
However, each sample will have a different mean and SD as well - sampling variability - which is not the same as the population mean and SD
To fix this, we use a sampling distribution

Question 4

What is a sampling distribution?

Answer 4

• We repeatedly collect samples of the same size from the same population
• It represents how much variability there is from one sample to the next one
• It has its own mean (which is the closest estimate of the population mean) and its own SD which is called the standard error

Question 5

What does a large/small standard error mean?

Answer 5

SE is the standard deviation of the sampling distribution

• Large SE = on average, lot of flactuation between our observed means
• Small SE = on average, less flactuation; fairly close to the population mean
• SE is a good indicator of the size of the difference between sample means

Question 6

What influences SE?

Answer 6

1. Sample size - the bigger the sample, the less it will flactuate
2. The variability of the variable itself - the bigger our sigma, the higher our SE

Question 7

Overview

What are the steps of a t-test in general

Answer 7

1. State the hypotheses (H₀ and Ha)
2. Choose the type of the t-test
3. Set the significance level (α)
4. Calculate the test statistic (t-value) based on the type of the test chosen
5. Determine the degrees of freedom (df)
6. Find the critical value corresponding to our df and α in a database
7. Compare the t-statistic to the t-distribution to find the p-value
8. Make a decision
   If the p-value ≤ α: Reject the null hypothesis
   If the p-value > α: Fail to reject the null hypothesis
9. Calculate the effect size
10. Report the results

Question 8

What is the t-statistic?

Answer 8

It represents the deviation of the sample mean from the population mean μ, considering the sample size, expressed as the degrees of freedom df = n − 1
picture 1 for the formula

Question 9

What is the T-distribution and what do we use it for?

Answer 9

It is a continuous sampling distribution of the t-statistic (standardised scale) which tells us how likely a certain t-value is given the null hypothesis is true

• If the population is normaly distributed (assumption of normality) the t-distribution represents the deviation of sample means from the population mean, given a certain sample size
• It has uncertainty built into it (works fairly well when sample is small and/or population SD is unknown)
• picture 2 for formula (no need to remember)

Question 10

What role does the sample size play in a t-distribution?

Answer 10

• T-distribution is different for different sample sizes and converges to a standard normal distribution if sample size is large enough
  ↪ With infinite many df, the t-distribution would be identical to the population distribution
• The lower the sample size, the more if differs from pop. distribution - it will have chunkier tails which means it has more probability mass in its tail areas compared to larger samples (picture 3)

Question 11

What is the difference between one sided and two sided t-test?

Answer 11

We have to establish our alternative hypothesis before we do the analysis
Two sided:
H_A: x̄ ≠ μ

One sided (directional hypothesis):
H_A: x̄ < μ
H_A: x̄ > μ

Question 12

What does the p value represent in regards to the t statistic?

Answer 12

It represents the probability of observing a t-statistic as extreme as, or more extreme than, the value calculated from our sample data, assuming that the null hypothesis is true

• Compare it to the alpha (α) level (usually 0.05) and we can evaluate the significance of the results

Question 13

How does the numer of degrees of freedom affect the p-value?

Answer 13

P-value is higher if we have lower df than higher df so to reach significance we need larger t statistic to reject the null hypothesis if our sample is smaller
- because we’re estimating both the population mean and the standard deviation and for low sample sizes it’s hard to estimate population sd because it’s a noisy estimate
- T-distribution helps with this since it has uncertainty build into it so it can compensate for low sample sizes

Question 14

What does the α represent and where on the picture 4 can we see it?

Answer 14

The (red in the picture) area under the curve is equal to our α = the probability of rejecting the null hypothesis if it it true (Type I error)

• the smaller the α, the more evidence we need in order to reject the null hypothesis
• So we will make Type I error less frequently
• We want Type I error to be as small as possible

Question 15

What is the difference in the t-distribution for the two sided and one sided t test?

Answer 15

Picture 4
In two sided test, our α level is divided into the two tails, whereas in the one sided test, it is only in one tail

• We look at the tail/s to see whether they contain extreme values or higher than our set α (extreme enough to reject the null)
• The extreme value marks the threshold for when we will reach significance when our p-value will be lower than α
• With a one-sided test, we reach significance faster since we’re making a more specific statement and the p-value doesn’t get split between the two tails
• However if we make a mistake about the directionality - we’re more lost since we have to reject the null hypothesis

Question 16

What is the effect size?

Answer 16

The effect-size is the standardized difference between the sample mean and the expected population mean

• In the t-test, effect-size can be expressed as d (Cohen’s d)
• formula in picture 5 - we square n which eliminates any influence of the sample size
• Another option is to have the effect size expressed as (similar to correlation coefficient) - formula in picture 6

Question 17

What are two ways to reach significance faster?

Answer 17

1. The size of the effect
2. The larger the group difference the more power we have to distinguish between two groups

Question 18

How do we classify the effect sizes and what distribution do we use for that?

Answer 18

Picture 7
Effect-size distribution - Which values will be more and which will be less likely under my null hypothesis?
Cohen’s d sampling distribution under H0
small ≈ 0.3
Medium ≈ 0.5
Large ≈ 0.8

Question 19

What do we observe if we set the population difference to a non-zero value?

Answer 19

We explore the sampling distribution under the alternative hypothesis
- The further the true value (mean of population difference) deviates from 0, the larger our power is

Question 20

What is the power of a t-test?

Answer 20

The probability that the test will correctly reject the null hypothesis (H₀) when the alternative hypothesis (Ha) is true
- The effect size drives our power = the higher our effect size, the greater our power
↪ if the effect size is too small, we have to have a large sample size
- A power of 0.8 measn that there is an 80% chance of detecting an effect if it exists
- Power=1−β
- β = the probability of making a Type II error (not rejecting the null when it’s false)

Question 21

What are the different types of t-tests?

Answer 21

1. One-sample t-test
2. Independent t-test
3. Paired-samples t-test (dependent t-test)

Question 22

One-sample t-test

Answer 22

Compares the mean of a single sample to a known or hypothesized population mean

Question 23

What are the assumptions of one-sample t-test?

Answer 23

Normal samples distribution
Random samples
Measurement level
- Interval
- Ratio

Will return to those later

Question 24

What is a model and what does it contribute to?

Answer 24

Model says something about the average value in the population and we can fit such a model to our data
Outcome = model + error
The equation shows what our model predicts accurately and mistakingly and together these form our observed scores

• we want the model to be high and the error to be low

Question 25

Example of a model

Answer 25

Postulated average of IQ = 120 (null hypothesis) - our model
Collected data: n = 63, mean = 117, sd = 16
Is the difference statistically significant that we reject our model of 120? = We do inferential statistics since the data is just from a sample and we’re talking about a population
We determine our Ha and α level, check the assumptions (later), t -statistic, use the t distribution to quantify how extreme our t statistic is, locate our t statistic on the x axis (what is the probability of observing a t statistic of 1.2 or greater if the null hypothesis is true = p value), calculate p value based on the t-dist.

Question 26

Paired-samples t-test?

Answer 26

Compares means of the same entities tested in different conditions on a related observation (e.g., within person, before and after treatment)
- Test whether the average difference of my two groups is equal to the null hypothesis or not

Question 27

What do we calculate in a paired-samples t-test?

Answer 27

The difference (D) for each pair is calculated and the mean of these differences (D with a bar) is tested against the null hypothesis where μ=0
↪ compares the mean difference between our samples to the difference that we would expect to find between population means
Picture 8 - the formula

Question 28

Explain the formula of paired-sample t-test in terms of signal and noise

Answer 28

It’s signal-to-noise variation - if the experimental manipulation creates difference between conditions then we would expect the effect (signal) ti be greater than the unsystematic variation (noise) → t>1

Question 29

What hypothesis can we postulate in the paired-samples t-test?

Answer 29

Picture 9

Question 30

What does the structure of the data look like?

Answer 30

We have a participant number and then each participant has two rows (k1 and k2) which is the level of the categorical predictor variable and x is the value of the outcome/dependent variable
Picture 10

Question 31

Assumptions of paired-samples t-test

Answer 31

1. Normality (relates to the sampling distribution of the differences between scores, not the scores themselves)
2. Independence of pairs - each pair of observations should be independent of other pairs
3. Continous data - dependent variable measured at interval or ratio

Question 32

Independent t-test

Answer 32

Compare means of two independent groups of observations to see if they are significantly different
- we measure different subjects in different groups

Question 33

How do we calculate the independent-samples t-test?

Answer 33

The mean of both independent samples is calculated and the difference of these (X¯1−X¯2) means is tested against the null hypothesis where μ=0
↪ difference between sample means not between individual pairs of scores
- picture 11
- pooled standard error → combination of sd of group 1 and sd of group 2
↪ takes into account the sample size by weighting the variance of each sample

Question 34

What hypothesis can we postulate in an independent-samples t-test?

Answer 34

Picture 12
Null hypothesis - There is no difference between the two groups

Question 35

What does the data structure look like?

Answer 35

Picture 13
index column - number of participant
k column - the level fo the categorical predictor variable
x - the value of the outcome/dependent variable

Question 36

What are the assumptions of the independent-samples t-test?

Answer 36

1. Normality assumption
2. Equality of variance

Question 37

What is the equality of variance assumption of independent-samples t-test?

Answer 37

Whether we can assume that the sd of group 1 and sd of group 2 are the same
- use Levene’s test = hypothesis test that is significant when the assumption is violated and it’s not significant if the assumption is not violated
H0: variance = equal (p>0.05)
Ha: variance ≠ equal (p<0.05)

Question 38

What should we be careful with when interpreting Levene’s test?

Answer 38

• Levene’s test is heavily influenced by sample size so a significant test result doesn’t necessarily mean that you have a problem
• A more pragmatic rule of thumb is to look at the ratio of variances - a ratio greater than 2 is problematic
• Welch is a version of a t-test that handles unequal variances better
  ↪ the difference between welch test and normal t-test only matters when our group sizes differ from each other (not that important to know)

Question 39

How do we check for the normality assumption?

Answer 39

1. We can use Shapiro-Wilk hypothesis test
   ~If p < α then the assumption is violated
   ~Similar shortcomings to the Levene’s test
   ~Caution as with any p-value → black-and-white decision making (little bit violated vs a lot violated)
2. Assess using a plot (Q-Q plot)
   ~Points should be along the diagonal
   ~ Not exact - use eyeballing (subjective)

Q-Q plot not on the exam - hard to assess but for real world it’s useful

Question 40

So how do we handle assumption violations?

Answer 40

Assumption violations affect the shape of the sampling distribution and mess up the type 1/2 error rates

Unequal variances
- Welch t-test

Non-normality
- use nonparametric test (will come later)

Question 41

How ot report t-test results

This is an example, just know what you have to select if they ask

Answer 41

For the data based on independent samples we could report this:
On average, participants given a cloak of invisibility engaged in more acts of mischief (M = 5, SE = 0.48), than those not given a cloak (M = 3.75, SE = 0.55). This difference, -1.25, 95% CI [-2.77, 0.27,], was not significant, t(21.54) = −1.71, p = 0.101; however, it represented an effect of d = 0.70 [−1.52, 0.14]

For the data based on paired samples:
On average, participants engaged in more acts of mischief when having access to a cloak of invisibility (M = 5, SE = 0.48), than when lacking such access (M = 3.75, SE = 0.55). This difference, -1.25, 95% CI [−1.97, −0.527], was significant, t(11) = −3.80, p = 0.003, and represented an effect of d = 0.68 [−1.3, −0.04]