Linear Regression

Question 1

Regression
- what is it used for?

Answer 1

a mathematical tool to:
1. analyse direction and strength of the relationship between the variable of interest (target) and other known variables

2. predict the value of the unknown variable (target) based on its past and other variables

Question 2

Linear Regression

Answer 2

linear approach for modelling the relationship between a scalar dependent variable y and on or more explanatory/independent variables X

• simple: 1 explanatory value
• multiple: >1 explanatory variable

Question 3

Supervised Learning

Answer 3

training set contains actual outcome (target)
- analyses training data and produced an inferred function, used to map

Question 4

Variable type

Answer 4

independent
- explanatory
- control
- input
- predictor

dependent
- response
- outcome
- target

Question 5

Pearson Correlation

Answer 5

quantifies dependence between 2 variables
- r (coefficient of correlation) : extent of interdependence
- value indicates strength
- sign indicates direction
- between -1 (negative total correlation) to +1 (positive total correlation); 0 = no linear correlation

Question 6

Types of relationships between 2 variables

Answer 6

• strong positive linear relationship
• positive linear relationship
• perfect negative linear relationship
• perfect parabolic relationship
• negative curvilinear relationship
• no relationship

Question 7

Regression vs correlation

Answer 7

regression allows use of MULTIPLE independent variables while correlation is only between 2 variables

Question 8

Statistical significance

Answer 8

• regression: evidence based analysis
• provides a mathematical way to find out the simultaneous impact of multiple independent variables on the dependent variable

Question 9

Application of Regression

Answer 9

Analytical (explanatory models)
- analyse if increase in police force has impact on crime rae
Predictive
- given persons lifestyle, predict % body fat

Question 10

Types of data used in regression

Answer 10

Cross sectional data : collected at one point in time
Time series: collected over a period of time
Pooled: combination of both cross-sectional and time series data ( eg population survey over 5 years)

Question 11

Simple linear regression analysis model

Answer 11

𝑦 = 𝛽0 +𝛽1𝑥

y = predicted/dependent variable
x = predictor/independent variable
𝛽0 = y-intercept
𝛽1 = slope / beta coefficient of x

For predictions:
𝑦_𝑖=𝑦̂_𝑖+𝜀_𝑖
=𝛽_0+𝛽_1 𝑥_𝑖+𝜀_𝑖
Error term 𝜀𝑖= is the difference between actual value 𝑦_𝑖 and the predicted value 𝑦̂𝑖

Question 12

Determine best fit line

Answer 12

sum of squared distances
- penalises or magnifies large errors
- cancels effect of sign

minimise sum of squared errors
- method of least squares : minimises the sum of squares of the vertical distance between data and fitted line

Question 13

Assessing goodness of fitted line

Answer 13

coefficient of determination (R square)
- amount of variation in Y that is explained by the regression line
- higher value, better regression line
= SSR/SST

• takes values [0,1]

Question 14

what is multiple linear regression

Answer 14

• increase accuracy: use more than 1 independent variable to estimate the dependent variable
• allows us to use more of the information available to estimate the dependent variable
• the best MLR accounts for the largest proportion of the variation in the dependent variable with the fewest number of independent variables

Question 15

parsimonious model

Answer 15

accomplishes a desired level of explanation or prediction with as few predictor variables as possible

Question 16

multiple regression equation

Answer 16

𝒀=𝛽_0+𝛽_1 𝑥_1+𝛽_2 𝑥_2+…+𝛽_𝑘 𝑥_𝑘

model significance using F-test and coefficient (variable) significance using T-test

Question 17

adjusted R squared

Answer 17

considers the number of variables being included in the model and penalises for overfitting
- decrease: added variables that are not significant or have multicollinearity with outher predictors

𝑅̅^2 = 1 − (1 −𝑅^2) (𝑛−1)/(𝑛−𝑝−1)

Question 18

MLR model significance

Answer 18

f-test: statistical tests for checking model significance in regression

• H0 : model with no independent variables (intercept-only model) fits the data as well as your model
• H1: model fits data better than the intercept model

Prob(F-statistic) = p-value
- if < significant level: reject null hypothesis

Question 19

MLR coefficient significance

Answer 19

t-Test: statistical tests for checking coefficient (variable) significance

• H0: coefficient 0, no r/s
• H1: coefficient not 0, variable can explain the model, regression model fits better

P>|t| = p-value
- if <significant level: reject null hypothesis

Question 20

estimate accuracy of prediction model

Answer 20

Mean Squared Error (MSE) (Variance of Error) = 1/𝑛 ∑( y_i − 𝑦_𝑖̂)^2

Root Mean Squared Error (RMSE) (Std Dev. Of Error) = √𝑀𝑆𝐸 = √1/𝑛 ∑( y_i − 𝑦_𝑖 ̂ )^2

Question 21

Assumptions for Linear Regression

Answer 21

LINE

Linearity relationship
Independence of errors
Normality of error distribution
Equal Variance of errors

Question 22

Residual Plots

Answer 22

𝑦=𝛽_0+𝛽_1 𝑥1 + … + 𝛽_𝑗 𝑥𝑗+ 𝜀_𝑖

Residual (a.k.a. errors 𝜀_𝑖)
= Observed Value of Y – Predicted Value of Y
= 𝑦_𝑖−(𝑦_𝑖 )̂

Residual plot: predicted values (𝑦_𝑖 )̂ (x-axis) vs residual values 𝜀_𝑖 (y-axis)

Question 23

if points are along the prediction line, how do residual plots look like?

Answer 23

• symmetrically distributed, tending to cluster towards the middle of the residual plot
• no clear patterns

Question 24

Multi-collinearity

Answer 24

• high level of correlation between independent variables.
• regression coefficients often become less reliable (or unstable) as the degree of correlation between the independent variable increases

Question 25

Multicollinearity problems

Answer 25

• hide significant variables
• causes parameter estimates to become unstable by increasing their variance

Question 26

Checking for multicollinearity

Answer 26

• pairwise correlation among predictors
• compute Variance Inflation Factor for each variable used in the regression model (remove predictors with VIF values> 5 or 10; OR use domain knowledge to decide which predictors have a dependence on another)