18. Exploratory and Confirmatory Data Analysis

Question 1

When is/should an exploratory analysis be used?

Answer 1

Conducted when:
1. Have a hypothesis but no clear plan/analysis
2. Have lots of variables that may be associated with an outcome variable but not sure how (no clear prediction)

(can provide tools for hypothesis generation via visualisation)

Question 2

What are the issues with exploratory data analysis?

Answer 2

• No clear prediction about how variables are related
• Not obvious what predictors are important to question
• Hypothesis can be tested in multiple ways

Question 3

How can exploratory analysis be done wrong?

Answer 3

• Results are wonky as can’t find starting hypothesis results - post-hoc analysis is damaging
• P-value conditional on intended sample size (issue of overfitting and multiple comparisons)
• Done poorly = Only report on significant variables that are significant (leads to overfitting and papers only producing desired outcome

Question 4

What are the steps of exploratory data analysis?

Answer 4

1. Check coding of data and visualise variables of interest
2. Compare models of interest to find if your variables of interest are good predictors of your outcome variable
3. Compute the K-fold, cross-validation, MSE for each of your models
4. Identify the best fitting models

Question 5

What is overfitting?

Answer 5

Statistical model with fit sample specific noise as if it is signal

Overfitting refers to a model that models the training data too well. Overfitting happens when a model learns the detail and noise in the training data to the extent that it negatively impacts the performance of the model on new data

Can’t trust estimates based on a model that was trained on the same sample

Question 6

How does R2 change in overfitting?

Answer 6

Optimistic - Higher as saying it will account for DV more than it does (Replication crisis = Studies don’t find the same results)

Question 7

What is p-hacking?

Answer 7

Special case of overfitting
Procedural overfitting - Takes place in parallel to model estimation (During data cleaning etc.)

Exploitation of data analysis in order to discover patterns which would be presented as statistically significant, when in reality, there is no underlying effect.

Question 8

What is the best way to avoid overfitting when exploring data?

Answer 8

Bias-variance tradeoff

Question 9

What is bias?

Answer 9

Tendency for a model to consistently produce answers that are wrong in a particular direction

Question 10

What is variance?

Answer 10

Extent to which a model’s fitted parameters will tend to deviate from central tendency across different data sets

Question 11

What are the two different approaches in the bias-variance trade-off? Which one is preferred?

Answer 11

1. Liberal, flexible data-analysis (low bias but high variance) - Almost any pattern can be detected at the cost of high rate of spurious identifications) - (exploratory data analysis)
2. Approach that fixes high variance (only a limited range of patterns can be identified - risk of pattern hallucination is low (confirmatory data analysis)

use method one, otherwise just assessing noise

Question 12

How does what to do in exploratory research help tell you what to do in confirmatory research?

Answer 12

• Data sets are large enough to support training models
• Accurately prediction error to asses performance and improve model
• Exert control over bias-variance trade-off when appropriate

Question 13

What is cross-validation?

Answer 13

To assess models = Need to quantify out of sample prediction error
Cross-validation is a resampling method that uses different portions of the data to test and train a model on different iterations.

Question 14

What is canonical cross-validation?

Answer 14

Classic replication
Model tested on one data set is then tested on completely independent data set
More in experimental design than correlational

Question 15

What are some issues that can prevent canonical cross-validation from occurring?

Answer 15

Sometime can’t collect enough data as limited population, limited funds to collect more data, one giant study you want to study was only run once

Question 16

What solves the issues of cross-validation?

Answer 16

K-Folding (K = no. folds)

• Recycles the data set
• in one fold (subset of data), one half is used for training and the other for testing
• Second fold, datasets are reversed i.e. training and testing roles are reversed
• Typical number of folds is 10
• Minimises error predictions

Question 17

What is confirmatory research analysis?

Answer 17

Conducted when you have a specific research question to test (putting a hypothesis to trial)

Specify prior to data collection, the statistical analyses you intend to run and expectations of relationships

Question 18

What is mean squared error and what is it used for?

Answer 18

The average squared difference between the estimated values and the actual value

Used to assess model fit

Question 19

What would an increased MSE suggest about a model?

Answer 19

It is a worse model as there is a larger difference between estimated and actual values

e.g. MSE of 0 = “perfect accuracy”

Question 20

What is a disadvantage of using MSE?

Answer 20

Heavily weighing outliers (some use mean absolute error instead)

Question 21

What is a sensitivity analysis?

Answer 21

Sensitivity analysis determines how different values of an independent variable affect a particular dependent variable under a given set of assumptions. In other words, sensitivity analyses study how various sources of uncertainty in a mathematical model contribute to the model’s overall uncertainty.

Question 22

What is a limitation of using purely confirmatory data analysis?

Answer 22

Can only detect a limited range of patterns when it’s possible others exist in the data so may miss important results

Question 23

Is direct replication of a prior experiment a form of cross-validation?

Answer 23

Yes