Lesson 4.2: Regression of 2 Variables

Question 1

Solution for Regression Line

Answer 1

Approach #1
- Closed Form Solution
- Compute Gradient
– Vector of Partial Derivatives Vector of Partial Derivatives of RSS (Residual Sum of Squares) w.r.t. predictor variables
– Set gradient to zero
– Compute slope and intercept
- Matrix Approach

Approach #2
- Gradient Descent Algorithm
- Gradually change slope and intercept till we reach at the optimum solution
- Can be computed by Excel Solver

Question 2

Closed Form Solution
(Calculus)

Answer 2

• a mathematical expression that can be evaluated in a finite number of operations (eg. a formula)
• It may contain constants, variables, certain “well known”
  operations (e.g., + − × ÷ ), and functions (e.g., nth root,
  exponent, logarithm, trigonometric functions, and inverse
  hyperbolic functions), but usually no limit.
• The set of operations and functions admitted may vary with author and context.

Question 3

Iterative approach / solution

Answer 3

• no formula
• trial and error
• keep adjusting x value until you find solution (y value)

Question 4

Gradient

Answer 4

• a vector of partial derivatives

Question 5

Computing Intercept and Slope

Answer 5

Residual = Observed value - Computed Value

Suppose regression equation is
𝑦=𝑚𝑥+𝑏
𝑦= 𝑒𝑥𝑝𝑙𝑎𝑛𝑎𝑡𝑜𝑟𝑦 𝑣𝑎𝑟𝑖𝑎𝑏𝑙𝑒
𝑥= 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑜𝑟 𝑣𝑎𝑟𝑖𝑎𝑏𝑙𝑒
𝑚= 𝑠𝑙𝑜𝑝𝑒 𝑜𝑓 𝑡ℎ𝑒 𝑙𝑖𝑛𝑒
𝑏= 𝑖𝑛𝑡𝑒𝑟𝑐𝑒𝑝𝑡
𝑅𝑒𝑠𝑖𝑑𝑢𝑎𝑙=𝑦𝑖−𝑚𝑥𝑖+𝑏
𝑅𝑒𝑠𝑖𝑑𝑢𝑎𝑙²=(𝑦𝑖−𝑚𝑥𝑖+𝑏)²
𝑅𝑒𝑠𝑖𝑑𝑢𝑎𝑙𝑠 𝑆𝑢𝑚 𝑜𝑓 𝑆𝑞𝑢𝑎𝑟𝑒𝑠=(𝑅𝑆𝑆)=σ 𝑖=1𝑁(𝑦𝑖−𝑚𝑥𝑖+𝑏)2

Question 6

Partial Derivatives of the RSS
w.r.t. Intercept and Slope

Answer 6

• 𝑅𝑒𝑠𝑖𝑑𝑢𝑎𝑙𝑠 𝑆𝑢𝑚 𝑜𝑓 𝑆𝑞𝑢𝑎𝑟𝑒𝑠=(𝑅𝑆𝑆)=σ 𝑖=1𝑁(𝑦𝑖−𝑚𝑥𝑖+𝑏)2
• To find minimum point of this function, we will take the partial derivative of RSS with respect to ‘m’ and ‘b’ and set that to zero.
• lowest RSS point = partial derivative of 0