Ch 7 Notes

Question 1

7.1 Linear Regression Model

Captures relationship between two or more variables

Predicts an outcome of a target variable based on several input variables

Make assessments and robust predictions by determining which of the relationships matter the most and which we can ignore

Answer 1

Regression Analysis

Question 2

7.1 Linear Regression Model

Formulate a mathematical model that relates the outcome of a target variable, called the response variable, to one or more other input values called predictor values

Use information on the predictor variables to describe and/ or predict changes in the response variable

Answer 2

Predictor Variables

Question 3

7.1 Linear Regression Model

Known to perform well for making predictions

Fail to establish a cause-and-effect relationship between the variables because of the nonexperimental nature of most applications

May appear to search for casuality when it basically detects correlation

Casuality: can only be established through randomized experiments and/ or advanced statistical models

Answer 3

Regression Model

Question 4

7.1 Linear Regression Model

Cannot expect to predict its exact (unique) value

If value is uniquely determined by the values of the predictor variables, the relationship between the variables is deterministic

Relationship between this variable and predictor variable is stochastic, due to the omission of relevant factors that influence this variable

Answer 4

Response Variable

Question 5

7.1 Linear Regression Model

How to develop a mathematical model that captures the relationship between the response variable 𝑦 and the 𝑘 predictor variables 𝑥_1,𝑥_2,⋯,𝑥_𝑘.

*Must account for randomness that is a part of real life

Answer 5

1. Start with a deterministic component that approximates the relationship
2. Then add a random error term to it, making the relationship stochastic
3. Use economic theory, intuition, and statistical measures to determine which predictor variables might best explain the response variable

Question 6

7.1 Linear Regression Model

Assumes meaningful numeric values

Answer 6

Numerical Value

Question 7

7.1 Linear Regression Model

Reflect categories

Answer 7

Categorical Variable

Question 8

7.1 Linear Regression Model

Used to describe two categories of a categorical variable, denoted d

Indicator or binary variable
𝑑=1 for one of the categories
𝑑=0 for the other(s)
The category with 𝑑=0 is called the reference or benchmark category
All comparisons are made relative to this category

Answer 8

Dummy Variable

Question 9

7.1 Linear Regression Model

**Uses only 1 predictor value

𝑦=𝛽_0+𝛽_1 𝑥_1+𝜀

𝛽_0 is the unknown intercept

𝛽_1 is the unknown slope

𝛽_0+𝛽_1 𝑥_1 is the deterministic component

𝜀 is the stochastic component or random error term

**The expected value of 𝑦 for a given value of 𝑥_1 lies on a straight line: 𝐸(𝑦)=𝛽_0+𝛽_1 𝑥_1.

Answer 9

Simple Linear Regression Model

Question 10

7.1 Linear Regression Model

𝛽_1 > 0

Answer 10

Positive Linear Relationship

Question 11

7.1 Linear Regression Model

𝛽_1< 0

Answer 11

Negative Linear Relationship

Question 12

7.1 Linear Regression Model

𝛽_1 = 0

Answer 12

No linear Relatioship

Question 13

7.1 Linear Regression Model

Model with more than one predictor

𝑦=𝛽_0+𝛽_1 𝑥_1+𝛽_2 𝑥_2+…+𝛽_𝑘 𝑥_𝑘+𝜀

Replace 𝑥 with 𝑑 for a dummy variable.

The population parameters 𝛽_0,𝛽_1,𝛽_2,…,𝛽_𝑘 are unknown and must be estimated using sample data.

Sample data: 𝑛 observations of 𝑦,𝑥_1,𝑥_2,…,𝑥_𝑘

Use the sample data to obtain 𝑏_0,𝑏_1,𝑏_2,…𝑏_𝑘 which are estimates of 𝛽_0,𝛽_1,𝛽_2,…,𝛽_𝑘.

Answer 13

Multiple Linear Regression Model

Question 14

7.1 Linear Regression Model

Predicted value of the response variable given specified values of predictor varibles

Answer 14

𝑦̂ (read as y-hat)

Question 15

7.1 Linear Regression Model

𝑒=𝑦−𝑦̂

Difference between observed and predicted values

Answer 15

Residual

Question 16

7.1 Linear Regression Model

𝑆𝑆𝐸=∑(𝑦−𝑦̂ )^2 = ∑𝑒^2

Chooses the sample regression equation by minimizing the error sum of squares

Desirable properties is certain assumptions hold

Gives an equation “closest” to the data

Answer 16

Ordinary Least Squares (OLS)

Question 17

7.1 Linear Regression Model

𝑏_0 is the estimate of 𝛽_0

𝑏_𝑗 is the estimate of 𝛽_𝑗

Answer 17

Estimated regression coefficients

Question 18

7.1 Linear Regression Model

Predicted value of 𝑦̂ when each predictor variable assumes a value of 0

Not always meaningful

Answer 18

𝑏_0 is the estimate of 𝛽_0

Question 19

7.1 Linear Regression Model

Change in the predicted value of the response given a unit increase in 𝑥_𝑗, holding all other predictor variables constant

Partial influence of 𝑥_𝑗 on 𝑦̂

Answer 19

𝑏_𝑗 is the estimate of 𝛽_𝑗

Question 20

7.1 Linear Regression Model

Subject to sampling variability

Will change if we use a different sample to estimate the regression model

Always wider than the corrisponding confidence interval

Answer 20

Predictions

Question 21

7.1 Linear Regression Model

point estimate ± margin of error

Use a confidence interval as the interval estimate for the mean (expected value) of y

Use a prediction interval as the interval estimate for the individual values of y

We use the same point estimate for constructing both

Answer 21

Interval Estimate

Question 22

7.1 Linear Regression Model

𝑦̂^0±𝑡_(𝛼∕2,𝑑𝑓) 𝑠𝑒(𝑦̂^0)

𝑦̂^0=𝑏_0+𝑏_1 𝑥_1^0+ 𝑏_2 𝑥_2^0+ ⋯ + 𝑏_𝑘 𝑥_𝑘^0

𝑑𝑓=𝑛−𝑘−1

One way to obtain this model is to estimate a modified regression model

y is the response variable

Explanatory variables defined as 𝑥_𝑗^∗=𝑥_𝑗−𝑥_𝑗^0

The resulting estimate of the intercept and its standard error are 𝑦̂^0 and 𝑠𝑒(𝑦̂^0)

Answer 22

For specific values 𝑥_1^0,𝑥_2^0,⋯,𝑥_𝑘^0, the 100(1−𝛼)% confidence interval for the Expected value of y

Question 23

7.1 Linear Regression Model

𝑦̂^0±𝑡_(𝛼∕2,𝑑𝑓) √(〖(𝑠𝑒(𝑦̂^0))〗^2+𝑠_𝑒^2 )
𝑦̂^0=𝑏_0+𝑏_1 𝑥_1^0+ 𝑏_2 𝑥_2^0+ ⋯ + 𝑏_𝑘 𝑥_𝑘^0

𝑑𝑓=𝑛−𝑘−1

Prediction Interval is wider than the confidence interval

Prediction Interval Incorporates the variability of the random error term

Higher variability makes it more difficult to predict accurrately, necessitating a wider interval

Answer 23

For specific values 𝑥_1^0,𝑥_2^0,⋯,𝑥_𝑘^0, the 100(1−𝛼)% prediction interval for the Individual Value of y

Question 24

7.1 Linear Regression Model

*A categorical variable may be defined by more than two categories

*Use multiple dummy variables to capture all categories, one for each category

*Given the intercept term, we exclude one of the dummy variables from the regression

-Excluded variable represents the reference category

-Including all dummy variables creates perfect multicollinearity

Question 25

7.2 Model Selection

Summarize how well the sample regression equation fits the data

Standard error of the estimate: 𝑠_𝑒

The coefficient of determination, 𝑅^2

The adjusted coefficient of determination, adjusted 𝑅^2

Answer 25

Goodness of Fit Measures

Question 26

7.2 Model Selection

Difference between the observed and predicted value of the response, 𝑒_𝑖=𝑦_𝑖−𝑦̂_𝑖

Sample regression equation provides a good fit when the dispersion of residuals is relatively small

Sample variance: 𝑠_𝑒^2 is the average squared deviation between the observed and predicted values

Answer 26

Residuals

Question 27

7.2 Model Selection

Has the same units of measurement as the response

𝑠_𝑒=√(𝑆𝑆𝐸/(𝑛−𝑘−1))

𝑆𝑆𝐸 is the error sum of squares, 𝑘 denotes the number of predictors, 𝑛 is the sample size.

Answer 27

Standard Deviation of Residuals (standard error of the estimate)

Question 28

7.2 Model Selection

For a given sample size, increasing the number of predictors reduces the numerator and denominator

The net effect allows us to determine if the added predictor variables improve the fit

When comparing models with the same response, the model with the smaller 𝒔_𝒆 is preferred

Question 29

7.2 Model Selection

Quantifies the sample variation in the response that is explained by the sample regression equation

Ratio of the explained variation of the response variable to its total variation

Answer 29

Coefficiant of Determination 𝑅^2

Question 30

7.2 Model Selection

Use in the context of regression

The total variation in 𝑦 is the total sum of squares, 𝑆𝑆𝑇=∑(𝑦_𝑖−𝑦̅ )^2

Break into two parts: explained variation and unexplained variation

𝑆𝑆𝑇=𝑆𝑆𝑅+𝑆𝑆𝐸

The variation in 𝑦 explained by the sample regression equation is the regression sum of squares,
𝑆𝑆𝑅=∑(𝑦̂_𝑖−𝑦̅ )^2

Answer 30

Analysis of Variance (ANOVA)

Question 31

7.2 Model Selection

The proportion of the sample variation in the response explained by the sample regression equation

Falls between 0 and 1

The closer to 1, the better the fit

Answer 31

𝑅^2=𝑆𝑆𝑅/𝑆𝑆𝑇=1−𝑆𝑆𝐸/𝑆𝑆𝑇

Question 32

7.2 Model Selection

Can’t use 𝑅^2 when the competing models do not include the same number of predictor variables (but have the same response)

𝑅^2 never decreases as we add more variables

May include variables with no economic or intuitive foundation

Answer 32

Comparing Models

Question 33

7.2 Model Selection

Explicity accounts for the sample size n and the number of predictor variables k

1−(1−𝑅^2 )((𝑛−1)/(𝑛−𝑘−1))

Imposes a penalty for any additonal predictors

The higher the adjusted 𝑅^2, the better the model

When comparing models with the same response, the model with the higher is preferred

Answer 33

Adjusted 𝑅^2

Question 34

7.2 Model Selection

𝑦=𝛽_0+𝛽_1 𝑥_1+𝛽_2 𝑥_2+…+𝛽_𝑘 𝑥_𝑘+𝜀

*We can conduct hypothesis tests about the unknown parameters 𝛽_0,𝛽_1,𝛽_2,…,𝛽_𝑘.

-Joint test about all of the parameters
-Individual tests about a single parameter

*For the tests to be valid, 𝑏_0,𝑏_1,𝑏_2,…𝑏_𝑘 must be normally distributed.

-Satisfied if the random error term 𝜀 is normally distributed

-If 𝜀 is not normally distributed, the tests are valid only for large sample sizes.

Answer 34

Linear Regression Model with k predictor variables

Question 35

7.2 Model Selection

Regarded as a test of the overall usefulness of a regression

Determines whether the predictor variables have a joint statistical influence on the responds

𝑦=𝛽_0+𝛽_1 𝑥_1+𝛽_2 𝑥_2+…+𝛽_𝑘 𝑥_𝑘+𝜀

-All of the slope coefficients equal zero; all of the predictors drop out; none of the predictors have a linear relationship with the response

-At least one of the slope coefficients does not equal zero; at least one predictor influences the response

𝐻_0:𝛽_1=𝛽_2=…=𝛽_𝑘=0
𝐻_𝐴:𝐴𝑡 𝑙𝑒𝑎𝑠𝑡 𝑜𝑛𝑒 𝛽_𝑗≠0

Test statistic measures how well the sample regression equation explains the variability in the response

Answer 35

Test of Joint Significance

Question 36

7.2 Model Selection

𝐹_(𝑑𝑓_1, 𝑑𝑓_2 ) =(𝑆𝑆𝑅⁄𝑘)/(𝑆𝑆𝐸⁄(𝑛−𝑘−1)=𝑀𝑆𝑅/𝑀𝑆𝐸

𝑑𝑓_1=𝑘
𝑑𝑓_2=𝑛−𝑘−1
𝑆𝑆𝑅 is the regression sum of squares.
𝑆𝑆𝐸 is the error sum of squares.
𝑀𝑆𝑅 is the mean square regression.
𝑀𝑆𝐸 is the mean square error.

Answer 36

Test Statistic

Question 37

7.2 Model Selection

p-value: 𝑃(𝐹_(𝑑𝑓_1, 𝑑𝑓_2 ) ≥𝑀𝑆𝑅/𝑀𝑆𝐸)

-A larger test statistic provides us with more evidence to reject the null hypothesis

-A larger test statistic indicates that a large portion of the sample variation in the response is explained by the regression model, thus the model is useful

Answer 37

Right Tailed F Test

Question 38

7.2 Model Selection

*If 𝛽_1=0, then 𝑥_1 basically drops out of the equation.

There is no linear relationship between 𝑥_1 and 𝑦.

𝑥_1 does not influence 𝑦.

Answer 38

Linear regression model with 𝑘 predictor variables

𝑦=𝛽_0+𝛽_1 𝑥_1+𝛽_2 𝑥_2+…+𝛽_𝑘 𝑥_𝑘+𝜀

Question 39

7.2 Model Selection

-Two-tailed
-Right-tailed
-Left-tailed

Typically 𝛽_𝑗0=0, but it could be a nonzero value.

Answer 39

Let 𝛽_𝑗0 be a hypothesized value of 𝛽_𝑗

Question 40

7.2 Model Selection

H_0: B_j = B_j0
H_A: B_j does not equal B_j0

Answer 40

Two Tailed Test

Question 41

7.2 Model Selection

H_0: B_j less than or equal to B_j0

H_A: B_j greater than B_j0

Answer 41

Right Tailed Test

Question 42

7.2 Model Selection

H_0: B_j greater than or equal to B_j0

H_A: B_j less than B_j0

Answer 42

Left Tail Test

Question 43

7.2 Model Selection

𝑡_𝑑𝑓=𝑏_𝑗−𝛽_𝑗0/𝑠𝑒(𝑏_𝑗)

𝑑𝑓=𝑛−𝑘−1

𝑏_𝑗 is the estimate for 𝛽_𝑗

𝑠𝑒(𝑏_𝑗) is the standard error of the estimator 𝑏_𝑗

𝛽_𝑗0 is the hypothesized value of 𝛽_𝑗

If the hypothesized value is zero,
𝑡_𝑑𝑓=𝑏_𝑗/𝑠𝑒(𝑏_𝑗)

Note that the testing framework for the intercept 𝛽_0 is similar.

Statistical packages report test statistics and p-values for a two-tailed test that assesses whether the coefficient differs from zero.

Answer 43

Test statistic for a test of individual significance

Question 44

7.2 Model Selection

Expresses the risk-adjusted return of an asset as a function of the risk-adjusted market return.

𝑅−𝑅_𝑓=𝛼+𝛽(𝑅_𝑀−𝑅_𝑓 )+𝜀

𝑅 is the rate of return on a stock or portfolio.

𝑅_𝑀 is the market return (typically the S&P 500).

𝑅_𝑓 is the risk-free interest rate (typically a Treasury bill).

𝛼 and 𝛽 are used in place of 𝛽_0 and 𝛽_1

Answer 44

Capital Asset Pricing Model (CAPM)

Question 45

7.2 Model Selection

Called the stock’s beta

𝛽=1: any change in the market leads to an identical change in the stock

𝛽>1: stock is more aggressive or riskier than the market

𝛽<1: stock is conservative or less risky than the market

Answer 45

𝛽: measures how sensitive the stock’s return is to changes in the market

Question 46

7.2 Model Selection

Called the stock’s alpha

𝛼>0: positive abnormal returns

𝛼<0: negative abnormal returns

Answer 46

𝛼: predicted to be zero, thus nonzero values indicate abnormal returns

Question 47

7.2 Model Selection

*Regression results are often reported in a table

Question 48

7.3 Model Assumptions and Common Violations

Statistical properties and validity of testing procedures depend on assumptions.

Under the assumptions, have desirable properties.

They are unbiased, 𝐸(𝑏_𝑗 )=𝛽_𝑗.

They have minimum variations between samples.

These are compromised when one or more of the model assumptions are violated.

*The validity of the significance test is also impacted

-Estimated standard errors are inappropriate
-Not possible to make meaningful inferences from the t and F tests

Answer 48

OLS Estimators

Question 49

7.3 Model Assumptions and Common Violations

Based on the error term 𝜀

The residuals, or the observed error term, 𝑒=𝑦−𝑦̂, contain useful information about 𝜀

Use the residuals to investigate the through residual plots.

Residuals vs. predictor variable: nonlinear patterns

Residuals vs. predicted values 𝑦̂: changing variability

Residuals sequentially: correlated observations

Residuals can also be used to detect outliers.

Answer 49

Assumptions

Question 50

7.3 Model Assumptions and Common Violations

Observations that stand out from the rest of the data

Observation stands out in the residual plot

Can greatly impact the estimates

It is not always clear what to do with them

May indicate bad data due to incorrectly recorded observations

-These observations should be corrected or deleted

May just be due to random variations in which case the relevant observations should remain

Answer 50

Outliers

Question 51

7.3 Model Assumptions and Common Violations

*Plot the residuals on the vertical axis and the predictor variable or predicted values on the horizontal line

Question 52

7.3 Model Assumptions and Common Violations

Linearity is in the parameters not the in the variables

Make economic and intuitive sense

Include all relevant predictor variables

Incorporate nonlinearalities

Detection:
-Use residual plots to identify nonlinear patterns
-Linearity is justified if the residuals are randomly dispersed
-A discernible trend in the residuals indicates a nonlinear pattern

Remedy:
-Employ nonlinear regression
-Transform the response and predictor variables

Answer 52

Assumption 1: Regression Model is linear in the parameters and is correctly specified

Question 53

7.3 Model Assumptions and Common Violations

Exists when two or more predictor variables have an exact linear relationship

Difficult to disentangle the seperate influences of the predictor variables

Results in imprecise estimates of slope coefficients

Detection:
-Model can’t be estimated
-High 𝑅^2 and significant F statistic coupled with individually insignificant predictor variables
-High correlations between predictor variables
-Wrong signs of the estimated coefficients
-Insignificant important predictor variables

Remedy
-Drop one of the collinear variables
-Obtain more data
-Express variables differently
-Do nothing

Answer 53

Assumption 2: There is no exact linear relationship among the predictor variables, there is no multicollinearity

Question 54

7.3 Model Assumptions and Common Violations

-Constant variability

-No heteroskedasticity

-Often breaks down cross sectional data

-Exists when the variability in the response changes with at least one predictor value

-OLS estimators are still unbiased
-Standard errors are not appropriate
-Can’t use t and f tests

Detection
-Plot the residuals against each predictor variable or predicted values
-Residuals should be randomly dispersed
-Violated when the variability of residuals increases or decreases

Remedy
-Correct the standard errors
-Called White’s robust standard errors
-Not available in Excel
-R has packages

Answer 54

Assumption 3: Conditional on the values of the predictors, the variance of the error term is the same for all observations

Question 55

7.3 Model Assumptions and Common Violations

-No serial correlation

-Often breaks down with time series data

-OLS estimates are still unbiased
-Standard errors are innapropriate, distorted downward making the model look better than it really is

-The t and F tests may suggest the predictor variables are significant when they are not

Detection:
-Plot the residuals sequentially over time
-Should show no pattern
-Postive (or negative) residuals followed by postive (or negative) residuals consecutively
-Switching between positive and negative residuals

Remedy:
-Use Newey West robust standard errors in R

Answer 55

Assumption 4: Conditonal on the predictor variables, the error term is uncorrelated across observations

Question 56

7.3 Model Assumptions and Common Violations

Allows construction of interval estimates and hypothesis tests

If the error term is not normally distributed, then the interval estimates and hypothesis tests are valid only for large samples

Answer 56

Assumption 6: The error term is normally distributed

Question 57

7.3 Model Assumptions and Common Violations

Use to obtain a residual plot

Open the stores data file

Data > Data Analysis > Regression

For Input Y Range, select the sales observations and for Input X Range, select the Sqft observations. Select residual plots

Obtain a residual plot sequentially over time

Open Sushi data file

Data > Data Analysis > Regression

Given the regression output, select the residuals and choose Insert > Scatter

Answer 57

Exel

Question 58

7.3 Model Assumptions and Common Violations

Use to obtain White’s robust standard errors

Import the stores data file in a data frame (table) and label it myData

Use to obtain Newey West Robust Standard Errors

Import Sushi Data File

Answer 58

R