Regression

Question 1

What questions can regression answer?

Answer 1

How do systems work?
ex: how many runs the avg homerun is worth
-effects of economic factors on pres. election

Make Predictions about what will happen in the future?
-height in the future
-price of oil in the future
-housing demand in next 6 months

Question 2

Simple Linear Regresssion

Answer 2

-one predictor
-y = response
x = predictor

Equation

y = a0 + a1x1

Question 3

general linear regression equation

Answer 3

with m predictors
y = response
x = predictor
y = a0 + sum from j =1 to m ajxj

Question 4

How do you measure the quality of a regression line’s fit?

Answer 4

the sum of squared errors

-distance between true response and our estimate

Question 5

simple linear regression prediction error

Answer 5

Yi - actual
yhati - prediction

Yi - Yhati or yi - (a0+a1xi1))

Question 6

Sum of squared errors equation

Answer 6

sum from i = 1 to n (yi-yhati)^2
or
sum from i = 1 to n (yi-(a0+a1xi1))^2

Question 7

What is the best fit regression SLR line?

Answer 7

minimizes sum of squared errors
-defined by a0 and a1

Question 8

How do we measure the quality of a models fit?

Answer 8

likelihood

Question 9

What is likelihood? What is maximum likelihood?

Answer 9

-measure the probability (density) for any parameter set; we assume the observed data is the correct value and we have information about the variance

-parameters that give the highest probability

Question 10

What Maximum Likelihood Estimation (MLE). What are you minimizing to calculate this?

Answer 10

the set of parameters that minimizes the sum of squared errors

zi = observations
yi = model estimates
minimize sum from i = 1 to n (zi-yi)^2

Question 11

Maximum likelihood in the context of linear regression

Answer 11

LR - y = a0 + sum from j =1 to m ajxj
sum square errors = sum from i = 1 to n (zi-yi)^2

substitute regression equation for yi in sum of squared errors

minimize sum from i = 1 to n (zi-(a0 + sum from j =1 to m ajxj))^2

Question 12

How can you use likelihood to compare two different models?

Answer 12

the likelihood ratio

Question 13

Akaike Information Criterion equation. What is the penalty terma nd what does it do?

Answer 13

L*: maximum likelihood value
K: # of parameters we’re investigating

AIC = 2k -2ln(L*)

Penalty term - (2k) balances likelihood with simplicity
-helps avoid overfitting

Question 14

AIC with regression? Do you want AIC to be smaller or higher?

Answer 14

substitute maximum likelihood reg. equasion and the # of parameters is m+1

-we prefer models with smaller aic, aic smaller encourages fewer parameters and higher likelihood

Question 15

corrected AIC

Answer 15

-works well if we have infitiely many data points
-this never happens

-add a corrections term

AICc = AIC 2k(k+1)/ n-k-1

Question 16

Comparing models with AIC

Answer 16

relative likelihood that lower AIC model is better =
e^((AIC1-AIC2)/2)

Question 17

Bayesian Information Criterion (BIC)

Answer 17

L*: maximum likelihood value
K: # of parameters we’re investigating
n: number of data points

BIC = kln(n) - 2ln(L*)

Question 18

AIC VS BIC

Answer 18

BICs penalty term >AICs penalty term
-BIC encourages models with fewer parameters than AIC does

-only use bic when there are more data points than parameters

Question 19

BIC comparison between 2 modesl on the same dataset…

Answer 19

is abs(BIC1-BIC2) >10 the smaller bic model is very likely to be better

if between 6 and 10 smaller bic models is likely better

between 2 and 6 somewhat likely better

between 0 and 2 is slightly likely to be better

Question 20

Is there a hard an fast rule for choosing betweeen AIC, BIC, or maximum likelihood?

Answer 20

No, all 3 can give valuable information. Looking at all 3 can help you decide which is best

Question 21

Regression coefficients for predictions and forecasting

Answer 21

the response increases by the coeeficient * the variable

in other words if the variable= 1 , that increases the response by the coefficient amount (descriptive)

if we are forecasting
-same thing but the coefficient is increase the response by its amount when the variable =1 (predictive)

Question 22

Which of the components of analytics can regression be used for?

Answer 22

Descriptive and predictive analytics
not prescriptive

Question 23

Causation

Answer 23

one thing causes another thing

Question 24

correlation

Answer 24

two things tend to happen together or not together
- they don’t nescessarily cause each other

Question 25

When is there causation?

Answer 25

-cause is before effect
-idea of causation makes sense
-no outside factors that could cause the relationship
-be careful before claiming causation

Question 26

Transforming data

Answer 26

-adjust the data so the fit is linear
-quadratic regression
-response transform
-box-cox transformation

Question 27

variable interaction

Answer 27

ex- 2 yr olds height @ adulthood. if both parents are tall maybe the kid will be even taller ie their heights interact

-y = a0 + a1x1 + a2x2+a3(x1x2)
-the interaction term is a new column of data that we can use as a new input x3

Question 28

p-value of coefficient

Answer 28

estimate the probability that the coefficient is really 0
-form of hypothesis testing

if p value > 0.05
- can remove from model

-other thresholds can be used
-higher thresholds - more factors can be included
-possibilitie of including irrelevant factor

-lower thresholds - less factors can be included - possibility of leaving out relevant factor

Question 29

p-value warnings

Answer 29

with large amounts of data p values get small even when attributes are not at all related to the response

p values are only probabilities even when meaningful
-100 attributes p values of .02 each, 2% chance of not being significant
-expect 2 that are not really relevant

Question 30

confidence interval

Answer 30

where the coefficient probably lies and how close it is to 0

Question 31

T-statistic

Answer 31

the coefficient divided by it’s standard error
-related to p value

Question 32

interpreting coefficient

Answer 32

-sometimes you discover the coefficient when multiplied by attribute still doesn’t make much of a difference even if the pvalue is very low

ex: estimate household income with age as one of the attributes
-if the coefficient is 1
even with low p value the attribute really isn’t very important. its unlikely to mkae even a $100 difference

Question 33

R squared value (coefficient of determination)

Answer 33

-estimate of how much variability your model accounts for
-ex rsquared = 59%
-accounts for about 59% of the variability in the data
-the remaining 41% is either randomness or other factors

Question 34

adjusted r dquared

Answer 34

rsquared adjusted for # of attributes used

Question 35

interpreting r squared, what is a good value?

Answer 35

-some things aren’t easily modeled
-things can affect real life systems especially when humans are involved
-r-squared values of .4 or .3 are quite good

Question 36

what is the null hypothesis?

Answer 36

the hypothesis that there is no significant difference between specified populations, any observed difference being due to sampling or experimental error.

Question 37

r squared formual

Answer 37

1-SSEresiduals/SSEtota