Topic 5: Linear Regression

Question 1

Given bivariate data, what are the steps for a linear regression framework?

Answer 1

1) Produce scatter plot

2) Produce regression line

3) Calculate correlation coefficient

4) Produce residual plot

5) Check assumptions

6) Perform predictions

Question 2

What does checking assumptions involve? What happens if the assumptions are true?

Answer 2

CHecking scatterplot to look linear

Ensure residual plot looks random

If these assumptions are true, the linear model is appropriate for use

Question 3

What does a residual plot look at?

Answer 3

Looks at gaps between linear regression line and the different points

Question 4

WHat is a scatter plot?

Answer 4

It is the graphical summary of 2 variables on the same plane, resulting in a cloud of points

Question 5

What is linear association between 2 variables?

Answer 5

Describes how tightly the points cluster around a line

Question 6

What are strong and weak associations?

Answer 6

Strong: Cloud of points tightly clustered around a line

Weak: Points aren’t tightly clustered around the line

Question 7

What is a positive and negative association?

Answer 7

Positive association is when one variable increases, another increases as well

Negative association is when one variable increases, another decreases

Question 8

What are the 5 things that a scatter plot can be summarised by

Answer 8

mean of x
mean of y
sd of x
sd of y
correlation coefficient (r)

Question 9

What is the centre of the cloud represented by?

Answer 9

By the point of averages (mean of x, mean of y)

Question 10

What is the horizontal spread of cloud measured by?

Answer 10

sd of x

Question 11

What is the vertical spread of cloud measured by

Answer 11

sd of y

Question 12

What is the correlation coefficient?

Answer 12

A numerical summary which measures clustering around a line. Indicates sign and strength of linear association. It is between -1 and 1

Question 13

What is population correlation coefficient?

Answer 13

Mean of the product of variables in standard units

Question 14

How does r (correlation coefficient) measure association?

Answer 14

r divides scatter plots into 4 quadrants, at the point of averages (centre)

Majority of points in the upper right and lower left quadrants –> overall positive r

Majority of points in the upper left and lower right quadrants –> overall negative r

Question 15

What are some properties of correlation coefficient (r)

Answer 15

It is a pure number (no units)

lies between -1 and 1

r = 0 occurs when points dont fit around a line but could still happen in multiple ways just not linearly

Correlation coefficient isn’t affected by interchanging the variables (switching x and y axis)

Correlation coefficient is shift and scale invariant (doesn’t change with different shifts to the graph or different extent of scaling)

Question 16

What are the two options for a line which represents relationship between two variables?

Answer 16

SD line and Regression lineW

Question 17

What is the SD line and why isn’t it preferred

Answer 17

Connects points of averages (mean of x, mean of y) to (mean of x + sd of x, mean of y + sd of y) (for r > 0)

OR

(mean of x, mean of y) to (mean of x + sd of x, mean of y - sd of y) (for r < 0)

It isn’t preferred because it is insensitive to amount of clustering around the line and thus underestimates (LHS) and overestimates(RHS) at the extremes

Question 18

What is the regression line and why is it a better option

Answer 18

Connects the point of averages to (mean of x + SD of x, mean of y + r * SD of y)

Accounts for extremes and clustering through use of the correlation coefficient

Question 19

What is the point of averages

Answer 19

coordinate of (mean of x, mean of y)

Question 20

WHat is the graph of averages

Answer 20

Plots the average y for each x value

Regression line is a smoothed out version of a graph of averages

Question 21

What is a residual?

Answer 21

vertical distance or ‘gap’ of a point above or below the regression line

Represents error between actual value and prediction

Question 22

What do residual plots graph?

Answer 22

Graphs residuals vs x

Question 23

How do we know if linear fit is appropriate based on residual plots?

Answer 23

There shouldm’t be a pattern –> random, because it shows variance is constant, and if not residuals aren’t random and violates the assumptions

Question 24

What are 11 common mistakes of regression

Answer 24

1) r doesn’t mean percentile. I.e. r = 0.8 doesn’t mean 80% of points clustered around line

2) r = 0.8 doesn’t mean that points are twice as tightly clustered than r = 0.4

3) Outliers can overtly influence correlation coefficient

4) Nonlinear associations can’t be detected by correlation coefficients

5) The same correlation coefficient can arise from very different data –> still need to be careful

6) Rates of averages tends to inflate correlation coefficient. I.e. a line between the two variables which are group means tends to overestimate strength of association between the two variables

7) Association doesn’t mean causation

8) Small SDs can make correlation look bigger

9) Beware of extrapolating beyond the range of the regression line

10) A high correlation coefficient that fits regression line might not even have data which is linear

11) Beware of refitting – even though correlation coeff might be same if x and y are switched, we need to refit the model depending on what fits the context

Question 25

What is the prediction error?

Answer 25

Difference between the line of regression (predicted value) and a certain point (actual value)

Question 26

What can be used to measure prediction error?

Answer 26

RMS error. It represents the average gap between the points and the regression line. I.e. ‘standard deviation for the line’

Question 27

What is the formula for RMS error (pop)

Answer 27

root of (mean of (gaps) ^2)

= root (1 - r^2 x SD of y)

Question 28

What are some important notes of RMS error?

Answer 28

Perfect correlation ( r = -1, 1) –> RMS error = 0

r = 0 –> RMS error = SD of y

If we use mean of y for any x (baseline prediction) –> RMS error = SD of y

To calculate RMS errors for sample vs population, only difference is multiplication of popsd for population compared to sd for sample

Question 29

What are 4 methods of making predictions

Answer 29

Baseline prediction

Prediction in strip

Regression line

Predicting percentile ranks

Question 30

What does baseline prediction involve?

Answer 30

Given a certain value of x, basic prediction of y would be the avg of y over all x values in the data

Question 31

What does prediction in strip involve?

Answer 31

Average of all y values in data corresponding to that x value

Question 32

What does prediction through regression line involve?

Answer 32

Use the given equation for regression line to predict y

Question 33

What does prediction through percentile ranks invovle?

Answer 33

If x is at a certain percentile of all x’s, find the percentile which we coiuld predict y to be in

Steps:
Find z score in x direction (Zx)

FInd predicted z score in y direction ( = r x Zx)

Translate z score in y direction back to percentile in y direction

Question 34

What does homoscedastic mean?

Answer 34

Variance of residual or error term is constant

Question 35

What does heteroscedastic mean

Answer 35

Unequal variance of residual

Question 36

How do we know if something is homoscedastic

Answer 36

You can tell if a regression is homoskedastic by looking at the ratio between the largest variance and the smallest variance. If the ratio is 1.5 or smaller, then the regression is homoskedastic.

If vertical strips on a scatter plot show equal spread in the y direction it is homoscedastic

RMS error can be used as a measure of spread for individual strips

Question 37

How do we know if something is heteroscedastic

Answer 37

If vertical strips dont show equal spread on a y direction in a regression line (or the residual plot)

Question 38

What are the implications of homoscedasticity?

Answer 38

Normal approximation can be used within the vertical strips

Question 39

How do we get the normal distribution of the strip?

Answer 39

New mean of y in new strip = mean of y + r x (Z score of x) x (SD of y)

New SD of y = RMS error (assume population)