PSCH 443 - Final

Question 1

Basic Logic of NHST

Answer 1

1. Belongs to null hypothesis significance testing - evaluates the probability of observing the data under the assumption that the null hypothesis is true.
2. If we assume the null is true, we can generate a sampling distribution that characterizes the distribution of the sampling mean we expect to observe.
3. By looking at the mean and the expected total variance, we can guesstimate the difference b/w an observed sample mean and the population mean when a sample error is made.
4. We can use this difference to determine the likelihood of seeing an actual effect versus any difference being the result of sampling error alone.

Question 2

Grand Mean (GM)

Answer 2

The combined mean of all the different means for each group or condition used in the ANOVA.

• We can take the average squared deviation of all the means
• Estimates the population variance
• Estimates the expected distribution over an infinite number of samples

Question 3

Post Hoc Tests

Answer 3

• Used when we do not have a theoretical basis to expect any particular differences b/w groups or conditions
• More conservative measures of differences b/c they are not guided by theory
• Used when there is a significant omnibus F stat, but no specific differences b/w groups were originally predicted

Question 4

Two categories of Post Hoc Tests

Answer 4

Fall into 2 broad categories:

1. Adjusting type I error rate to accommodate multiple comparisons
2. Calculating new and more conservative test statistic

Question 5

When using Bonferroni correction:

Answer 5

1. Calculate a new alpha

2. Takes the desired level of family wise error for an experiment (i.e., 0.05) and divides it by the # of comparisons

Question 6

When using a Tukey HSD we:

Answer 6

1. Calculate a new test statistic that represents the mean difference that must be reached in a comparison to be statistically significant
2. Assumes we want to compare all means
3. Uses 0.05 as an arbitrary cut-off

Question 7

Planned Comparisons

Answer 7

• Planned b/c they should be guided by theory; # of planned comparisons is generally small relative to the # of conditions b/c this reduces family wise error by default
• Tests are only made b/w a few groups that have key differences as opposed to there being several tests across several conditions
• Uses the error term from the omnibus f test, or the Within Groups Mean Squares

Question 8

The two types of planned comparisons:

Answer 8

2 types:

Pairwise – analyze simple differences b/w 2 means
Complex – analyze the difference b/w sets of means

Question 9

What do we do in complex comparisons:

Answer 9

In complex comparisons, we need to come up w/ contrast weights. Contrast weights are sample means weighted by a coefficient

1. Choose sensible comparisons
2. Groups with positive weights will be compared to those with negative weights.
3. The sum of the weights should always be zero.
4. Groups not involved in a comparison always get a coefficient equal to zero

Question 10

Family-wise Type I Error

Answer 10

Inflated probability of making a type I error based on greater # of tests performed

1. Reflects that multiple tests are independent and have their own unique probability of committing a type I error (or of incorrectly rejecting the null)
2. Sums the total of all tests performed
3. Subtract them from 1 to standardize the probability of committing a type I error (or of incorrectly rejecting the null)

Question 11

ANOVA as Regression

Answer 11

Can be understood as:

Systematic Variation + Unsystematic Error / Unsystematic Error

• Both try to explain variability although model estimation is different
• Focuses on categorical variables
• If mean differences are larger than what we expect due to chance (error), the value of the F stat should increase
• The systematic variance that our model explains = the effect of our IV on our DV

Question 12

Partition of ANOVA

Answer 12

1. SSt represents the overall variability we are trying to explain
2. Partitioned into SSm (variance accounted for) and SSr (variance unaccounted for)
3. Unsystematic variance cannot be explained for in any meaningful way using ANOVA models

Question 13

Dummy Coding ANOVA for a regression analysis

Answer 13

1. We enter all dummy codes in one block
2. Comparison group is given a value of 0
3. Other groups are given a value of 1 in each row

Question 14

Orthogonal comparisons

Answer 14

• Info given by the comparisons is independent of other comparisons ran on the data
• Sum of weighted comparisons has to be equal to 0 to maintain independence
• Does not inflate familywise error b/c outcomes are treated independently so no test type I error probabilities are overlapping

Question 15

Assumptions of ANOVA

Answer 15

1. Normality
   - The distributions of the residuals are normal
2. Homogeneity of variance
   - Variances should be roughly equal across groups
3. Independence of observations
   - The error term is the same across all values of the independent variables
   - Spread is roughly the same across levels, so there is about equal random error

Question 16

Eta-squared

Answer 16

The most generally accepted measure of effect size/statistical power

• Will be = to R2 in one-way ANOVA
• tends to overestimate the effect size in the population
• the inverse of type II error, or the likelihood that we will detect a significant effect when none exists
• smaller range for effect size = better chance of detecting the effect

Question 17

Factorial ANOVA

Answer 17

Research designs that have more than one IV

Question 18

Main Effect

Answer 18

• The effect of one IV on the DV
• Breaks downs the sums of squares for the model into sums of squares for the main effect of variable A and sums of squares for the main effect of variable B

Question 19

Interaction

Answer 19

• The way that one IV affects the DV
• Depends on the level (condition) of the other IV and how the two measures are interrelated
• Need to be evaluated using comparisons in relation to both theory and what best matches interpretation

Question 20

Between Subjects Designs

Answer 20

Between Subjects design have the following features:

1. Typical experimental procedure
2. Each level of the independent variable is assigned to different groups of people
3. Comparisons are between different groups
4. Requires larger number of participants b/c statistical power is lower

Question 21

Within Subjects Design

Answer 21

Within Subjects designs have the following features:

1. The independent variable represents more than one assessment of the same group, under different conditions
2. Statistical power increased: requires fewer subjects.
3. Other IVs may be between subjects (Mixed Designs)

Question 22

Sums of Square Within Participant, SSW:

Answer 22

• Reflects both systematic variation due to treatment and unsystematic differences across individuals
• Calculate between condition sum of squares
• We subtract SSM from SSW to get our new error term SSR

Question 23

Sphericity

Answer 23

• Refers to the equality of variances of the differences between treatment levels
• The variability across people is relatively uniform across the levels of the IV
• If sphericity is violated, the omnibus error term may be too liberal for some comparisons and too conservative for others
• Huge issue for post hoc tests; Bonferroni is generally considered the safest

Question 24

Carry-over effects

Answer 24

• Tests subjects under all conditions, which can cause effects based on the order of the conditions
• Can prime subjects to respond in a specific way based on past experience w/ the testing conditions of the experiment
• Counterbalancing reduces the likelihood of carry-over effects b/c order is less of a problem when conditions are presented randomly

Question 25

Partial Eta Squared

Answer 25

• The ratio of variance accounted for by an effect and that effect plus its associated error variance
• Sums of squares of effect / sums of squares for effect + Sums of Squares error term
• Measures effect size w/ other variance of effects partitioned out
• Used when participants in levels are the same for different conditions or experimental tests
• Part of repeated measure designs

Question 26

Discuss the logic of the F-ratio. What is it about the structure of the F-ratio that enables us to evaluate our null hypothesis while also allowing for the possibility that differences in our condition are not due to chance?

Answer 26

F-ratio is the ratio of the population variance as estimated between groups vs. within groups. There is a certain amount of variance in our experience that will always be the result of sampling error, or variance between the means b/c baseline fluctuates per sample.

• A bigger sample size = more stable measure of group variance
• Larger amount of overall variance = more likely to observe a true effect
• If null is true, then both within and between groups estimate should equal roughly 1 b/c the means are the same

Question 27

What is the f-ratio?

Answer 27

When we take the means between groups and divide them by the within groups variance (or our total variance), we can find if our effect appears to exceed a 5% chance of type I error. If it does, we know that the effect is unlikely to be due to chance b/c it is unlikely such a larger amount of variance is not actually a product of true mean differences.

Question 28

Under what circumstances should we use post hoc tests? What are the advantages and disadvantages to post hoc tests?

Answer 28

1. Post Hoc - no theoretical basis to assume there is a difference b/w means. Not planned before the experiment; generally done to test variations in the result after-the-fact.

• Conservative
• Used when there is a significant f-ratio but no specific differences were predicted
• More difficult to reach significance

Question 29

Under what circumstances should we use planned comparisons? What are the advantages and disadvantages to planned comparisons?

Answer 29

1. Planned comparisons - based on theory and interpretation, and chosen before conducting an experimental procedure.

• generally a small # of tests to avoid inflating familywise error
• Uses several comparisons as controls and only examines some
• Can be pairwise or complex - analyzes the differences b/w pairs or sets of means
• If orthogonal, they can be very robust tests
• Can be ran even if omnibus F is not significant, which is a huge advantage
• Has to have own error term created in order to find if the comparison is too conservative or too liberal

Question 30

Why can’t we just do a bunch of t-tests to follow up a significant omnibus test?

Answer 30

We cannot run multiple t-tests b/c it would inflate the probability of committing a type I error. In other words, we would have too much family wise error and increase the risk of misidentifying an effect when there is not one.

Question 31

What is the relationship between the F-ratio and the t statistic?

Answer 31

The sampling distribution for a t-test was based on mean differences, and the t-distribution itself is based on mean differences relative to the sampling error. In ANOVA, the sampling distribution is based on a ratio called the F-ratio.

Question 32

What is interesting about the value for the f-ratio and the t-statistic?

Answer 32

The f-ratio is the value squared t statistic. By association, the t stat is the square root of the f-ratio. T-tests evaluate mean differences b/w groups, whereas f-ratio evaluates them relative to all the variance and is standardized.

Question 33

Why is it generally desirable to run orthogonal planned contrasts? When might it be permissible to run non-orthogonal comparisons?

Answer 33

It is good to run orthogonal planned comparisons b/c if the tests are independent of one another, the variance will not be inflated by having to assume the existence of multiple type 1 errors within our design. Each time we have to assess the probability of a type 1 error, we compound them based on any other tests.

• we can run non-orthogonal comparisons when running independent comparisons would not make theoretical sense, or we have no theoretical justification for them to be orthogonal

Question 34

Why might it be permissible to run a planned comparison even if your omnibus F is non-significant?

Answer 34

It can depend on how the different variables interact with each other. If an effect on the dependent variable is small, sometimes the omnibus f-test is not sensitive enough to detect it in comparison to the overall low change in variability our model accounts for.

• perhaps possible when testing a hypothesis that needs a comparison?
• when running few tests b/c difference is being washed out

Question 35

Jacob Cohen’s suggested guidelines for evaluating effect size

Answer 35

Effect size can be broken down into three general categories.

1. Small effect –η2 = .01
2. Medium effect –η2 = .06
3. Large effect –η2 = .15

Question 36

What is a shortcoming of Cohen’s guidelines for effect size?

Answer 36

A shortcoming of this approach is that these guidelines are arbitrary. We do not have a good set point for what constitutes a strong effect or otherwise. They only work in the absence of any other useful information.

Question 37

What factors should be considered when reviewing effect size?

Answer 37

When evaluating the practical significance of effects, we want to remember sample size. If we have an idea of how large an effect might be, we can determine the amount of subjects needed. In general, a larger sample size increases power and helps reduce the likelihood that we will fail to detect an effect if one does exist.

Question 38

When we conduct a factorial analysis of variance, we usually do not interpret the main effect and instead focus on the interaction. Why is it important that we focus on the variable insofar as it is involved in the interaction and not as a main effect?

Answer 38

Main effects only look for general mean differences, which just demonstrates that the dependent variable is related to the independent variable in some way. It does not outline where or in which ground/condition this difference originates from.

Interactions help illustrate how the variables effect is quantified based on some other second or third or nth variable; interactions imply the main effect is the result of something else, such as treatment or condition for example, and it has to be interpreted as such.

Question 39

When we have a factorial research design, we compute several estimates of between groups variance. Why?

Answer 39

To partition the amount of variance our model accounts for versus total variance. We then have to break it down further to see what portion of variance main and interaction effects constitute. We do this by using the variance to parse out unsystematic or random error to estimate each components effectiveness.

Question 40

When we have a within subjects factorial design, we compute several estimates of error variance. Why is this necessary?

Answer 40

To partition the amount of variance our model accounts for versus total variance for all participants. Basically the same as a typical factorial research design, but we get the mean differences for each participant as well.

Question 41

Why do within subjects factorial designs typically have far more statistical power than between subjects designs? Where in the statistical analysis is the advantage realized?

Answer 41

Within subjects designs have more statistical power b/c it observes changes within the same population of participants, so sampling variation stays the same.

The differences in the means reflect only systematic error if properly partitioned, and this advantage is realized when analyzing the sum of squares within participant for the independent variable.

Question 42

Be able to define a set of orthogonal comparisons for a set of condition means.

Answer 42

Choose sensible comparisons

Groups with positive weights will be compared to those with negative weights

The sum of the weights should always be zero

Groups not involved in a comparison always get a coefficient equal of zero

Question 43

Be able to articulate the logic of a within subjects ANOVA.

Answer 43

1. Eliminates major source of unsystematic error variance by using same population
2. Each participant is acting as his or her own experimental control across conditions
3. Able to partial out the unsystematic error from the systematic error in our analysis

• get variance for Sums of Square Within Participant, SSW
  SSW = the extent to which individuals tend to vary across conditions

4. Because the same people are compared across every condition SSM is now relatively free of unsystematic error
5. If we partial the sums of squares for our IV from the sum of squares within participants, we have a new measure of error variance
6. SSR = SSW - SSM