Lecture 3

Question 1

Regression

Answer 1

Using supervised learning to predict continuous outputs

Question 2

Cost Function

Answer 2

Tells us how well our model approximates the training examples

Question 3

A Model

Answer 3

A function that represents the relationship between x and y

Question 4

Optimization

Answer 4

A way of finding parameters for the model while minimizing the loss function

Question 5

Linear Regression

Answer 5

aka ordinary least squares;

Given input feature x we predict output y;

Follows basic y = mx + e equation with m as the parameter and e the measurement or other noise;

supervised learning

Question 6

Why use Least Squares Linear Regression?

Answer 6

It minimizes the squared distance between measurements and regression line and is easy to compute (even by hand)

Question 7

What do you do when your least squares minimization line doesn’t pass through the origin?

Answer 7

Introduce a bias term (intercept labeled as ‘b’).

This gets you y = mx+b+e

Question 8

Curve Fitting

Answer 8

Finding a mathematical function (constructing a curve) that has the best fit on a series of data points

Question 9

Smoothing

Answer 9

When you don’t look for an exact fit but for a curve that fits the data approximately

Question 10

Perfect Fit

Answer 10

Fits the data precisely; goes through all data points

Question 11

Best Fit

Answer 11

May not be the perfect fit; should give the best predictive value

Question 12

Fit

Answer 12

The accuracy of a predictive model;

the extent to which predicted values of a target variable are close to the observed values of that variable

Question 13

R^2

Answer 13

How the fit is expressed for regression models;

The percentage of variance explained by the model

Question 14

Underfitting Performance

Answer 14

Performs badly on both training and validation sets

Question 15

Overfitting Performance

Answer 15

Performs better on the training set than on the validation set

Question 16

How do you tune hyperparameters?

Answer 16

1. Divide training examples into training and validation sets.
2. Use the training set to estimate the coefficients m for different types of hyperparameters (degree of the polynomial)
3. The validation set estimates the best degree of the polynomial by evaluating how well the classifier does on this validation set
4. Test how well it generalizes on unseen data

Question 17

Regularization

Answer 17

The reduction of the magnitude of the coefficients;

Helps to prevent overfitting by limiting the variation of the parameters which prevents extreme fits to the training data

Question 18

Ridge Regression

Answer 18

• Reduces model complexity by coefficient shrinkage.
• The penalty term is controlled by alpha
• The higher the value of alpha, the bigger the penalty. Therefore, the magnitude of coefficients are reduced

Question 19

Least Absolute Shrinkage Selector Operator (Lasso) Regression

Answer 19

The magnitude of the coefficients reduce at even small values of alpha;

Lasso reduces some of the coefficients to zero (known as feature selection) which is absent in ridge regression

Question 20

Feature Selection

Answer 20

Reducing some of the coefficients to zero.

Present in Lasso but not Ridge regression

Question 21

Gradient Descent

Answer 21

An iterative optimization algorithm for finding the local minimum of a function. To find the local minimum of a function using gradient descent, we must take steps proportional to the negative of the gradient (move away from the gradient) of the function at the current point.

Question 22

The coefficients of the least squares regression line are determined by minimizing the sum of the squares of ____?

Answer 22

residuals (y-coordinates - slope*x-coordinates)

Question 23

What do the outputs of a linear regression look like if we have 1, 2, or more than two inputs?

Answer 23

The output of a one-dimensional input is a line (y = w1x1+ b).
The output of a two-dimensional input is a plane (y = w1x1 + w2x2. + b).
The output of multi-dimensional inputs (more than two) is a hyperplane

Question 24

How is the residual variance computed?

Answer 24

The residual variance is computed as the sum of the squares of the y‐coordinates from the data minus the y‐coordinates predicted by linear regression.

See slide 15 of the lecture notes

Question 25

Is linear regression sensitive to outliers?

Answer 25

Yes, the slope of the regression line will change due to outliers in most of the cases. Therefore, linear regression is sensitive to outliers.