Week 4

Question 1

According to Simmons et al. (2011), what is the most costly error?

Answer 1

a false positive, the incorrect rejection of a null hypothesis

Question 2

What does it mean to set the significance value at p ≤ .05?

Answer 2

the maximum false-positive rate is 5%

Question 3

how do DoF make false positives more likely?

Answer 3

researchers don’t preregister designs, so they can explore various analytic alternatives to search for a combination that yields statistical significance and then report only what worked

Question 4

which are the 4 most common ways to choose your DoF?

Answer 4

1. choosing number of dependent variables
2. choosing sample size
3. using covariates
4. reporting subsets of experimental conditions

Question 5

Why is flexibility in sample size a problem?

Answer 5

because there is no linear relationship between sample size and likelihood of significant results

Question 6

What 6 requirements do Simmons et al. (2011) propose for authors?

Answer 6

1. decide rule for terminating data collection beforehand & report it
2. at least 20 observations per cell
3. list all measured variables
4. report all experimental conditions including failed manipulations
5. if observations are excluded, report results for when they are included also
6. If an analysis includes a covariate, also report analysis results without the covariate

Question 7

What are nonsolutions for the problem of researcher DoF?

Answer 7

1. correcting alpha levels
2. using bayesian statistics
3. conceptual replications
4. posting materials and data

Question 8

Why doesn’t correcting alpha levels help the DoF problem?

Answer 8

1. it is unclear which and how many DoF contribute to a given finding and what their effect is on the false-positive rate
2. there is no rule about how to adjust each DoF

Question 9

Why don’t Bayesian statistics solve the DoF problem?

Answer 9

it increases researcher DoF:
1. constitutes a new set of analyses that can be applied flexibly on data
2. require adjustments on a case-to-case basis

Question 10

What solution do Wagenmakers et al. (2012) propose for minimising false positives?

Answer 10

preregistration

Question 11

what is exploratory research?

Answer 11

hypotheses are formed after data analysis

Question 12

what is confirmatory research?

Answer 12

the entire analysis plan is made explicit before any analysis is done

Question 13

what is a methodologically sound way to combine exploratory and confirmatory research?

Answer 13

collect the data all at once and then split the data into exploratory and confirmatory subsets

Question 14

What are Latin squares?

Answer 14

partial counterbalancing method for condition order - each condition appears once/participant and once/position in the sequence of conditions

Question 15

What is a limitation of Latin squares?

Answer 15

possible sequential effects (condition B always follows condition A)

Question 16

What do balanced Latin squares do that Latin squares don’t?

Answer 16

each condition is preceded and followed once by each other condition

Question 17

What does Bradley’s method for counterbalancing control for?

Answer 17

immediate sequential effects (but not remote)

Question 18

Explain Bradley’s method for even numbers of conditions

Answer 18

1. first row = A, X, B, X-1…
2. next rows = next letter in alphabet (vertically)

Question 19

Explain Bradleys method for counterbalancing for odd numbers of conditions

Answer 19

1. make 1 balanced Latin square
2. mirror vertically to make 2nd one

Question 20

What is a benefit of Alimena’s counterbalancing method over Bradley’s?

Answer 20

it controls for immediate AND remote sequential effects

Question 21

explain Alimena’s method for counterbalancing

Answer 21

1. first column = letters in ascending order
2. last column = letters in descending order
3. 2 diagonals = the connecting letters
4. rest of letters filled alphabetically per column, leaving n+1 spaces between letters for each next column. When at top again, leave n-1 spaces

Question 22

how can you counterbalance stimulus assignment to conditions?

Answer 22

create one stimulus set for each level of your IV

Question 23

How can you counterbalance for condition order and stimulus assignment simultaneously?

Answer 23

Using Lewis’ method

Question 24

What are the benefits of Lewis’ method for counterbalancing?

Answer 24

it counterbalances condition order, stimulus set order, and assignment of stimulus set to conditions

Question 25

what is the disadvantage of Lewis’ counterbalancing method?

Answer 25

it doesn’t control for sequential effects

Question 26

explain how you can counterbalance for condition order, stimulus order, and control for sequential effects

Answer 26

1. create pair of Latin squares using Alimena’s method, the 2nd square mirroring the first (representing conditions)
2. create 2nd pair of squares representing stimulus sets
   - 3 is 1 with numbers & mirrored horizontally in centre
   - 4 is 1 but with numbers & adjacent rows are swapped except 1st and last
3. Pair first 2 with second 2 squares