empirical risk

Question 1

simple linear regression model

Answer 1

H(x) = w0 + w1x

Question 2

slope in simple linear regression model

Answer 2

w1

Question 3

intercept in simple linear regression model

Answer 3

w0

Question 4

loss function

Answer 4

quantifies how bad a prediction is for a single data point

Question 5

if our prediction is close to the actual value

Answer 5

we should have low loss

Question 6

if our prediction is far from the actual value

Answer 6

we should have high loss

Question 7

error

Answer 7

difference between actual and predicted values (yi - H(xi)

Question 8

squared loss function

Answer 8

computes (actual - predicted)^2

Question 9

constant model

Answer 9

Lsq(yi, h) = (yi - h)^2

Question 10

another term for average squared loss

Answer 10

mean squared error

Question 11

best prediction, h*

Answer 11

Rsq(h) = 1/n(Summation of i = 1, n) (yi - h)^2

Question 12

constant model

Answer 12

H(x) = h

Question 13

simple linear regression

Answer 13

H(x) = w0 + w1x

Question 14

how do we find h* that minimizes Rsq(h)

Answer 14

using calculus

Question 15

minimize Rsq(h)

Answer 15

1. take its derivative with respect to h
2. set it equal to 0
3. solve for the resulting h*
4. perform a second derivative test to ensure we found a minimum

Question 16

derivative of Rsq(h)

Answer 16

-2/n(SUMMATION of n starting w/ i = 1)(yi - h)

Question 17

Mean minimizes…

Answer 17

mean squared error

Question 18

absolute loss

Answer 18

Labs(yi, H(xi)) = |yi - H(xi)|

Question 19

average absolute loss

Answer 19

Rabs(h) = 1/n summation of n from i = 1 |yi - h|

Question 20

to minimize mean absolute error

Answer 20

1. take its derivative with respect to h
2. set it equal to 0
3. solve for the resulting h*
4. perform a second derivative test to ensure we found a minimum

Question 21

derivative of |yi - h|

Answer 21

it is a piece-wise function, so will be undefined

Question 22

derivative of Rabs(h)

Answer 22

d/dh(1/n SUMMATION of n from i = 1, |yi - h|) = 1/n[#(h > yi) - #(h < yi)]

Question 23

median minimizes

Answer 23

mean absolute error

Question 24

best constant prediction in terms of mean absolute error

Answer 24

median
1. when n is odd, answer is unique
2. when n is even, any number between the middle two data points also minimizes mean absolute error
3. when n is even, define the median to be the mean of the middle two data points

Question 25

process for minimizing average loss

Answer 25

empirical risk minimization

Question 26

another name for “average loss”

Answer 26

empirical risk

Question 27

corresponding empirical risk when using the squared loss function

Answer 27

Rsq(h) = 1/n SUMMATION of n from i = 1 (yi - h)^2

Question 28

if L(yi, h) is any loss function the corresponding empirical risk is

Answer 28

R(h) = 1/n(SUMMATION Of n from i = 1, L(yi, h)

Question 29

Modeling recipe

Answer 29

1. choose a model
2. choose a loss function
3. minimize average loss to find optimal model parameters

Question 30

empirical risk minimization

Answer 30

formal name for the process of minimizing average loss

Question 31

corresponding squared loss function to Lsq(yi, h) = (yi - h)^2

Answer 31

Rsq(h) = 1/n Summation of n from i = 1 (yi - h)^2

Question 32

For the mean

Answer 32

sum of distances below = sum of distances above

Question 33

Mean is the point where

Answer 33

Summation of n from i = 1 (yi - h) = 0

Question 34

Median is the point where

Answer 34

(yi < h) = #(yi > h)

Question 35

Lp loss

Answer 35

Lp(yi, h) = |yi - h|^p

Question 36

Corresponding empirical risk to Lp

Answer 36

Rp(h) = 1/n summation of n from i = 1|yi - h|^p

Question 37

midrange minimizes

Answer 37

L(infinity loss)

Question 38

As p –> infinity,

Answer 38

the minimizer of mean Lp loss approached the midpoint of the minimum and maximum values in the dataset or midrange

Question 39

The general form of empirical risk for any loss function

Answer 39

R(h) = 1/n Summation of n from i = 1 (L(yi , h))

Question 40

input h* that minimizes R(h) is…

Answer 40

some measure of the center of the dataset

Question 41

minimum output R(h*) represents

Answer 41

some measure of the spread or variation in the dataset

Question 42

Minimum value of Rsq(h)

Answer 42

Rsq(h*) = Rsq(Mean(y1, y2…yn))
= 1/n SUMMATION of n starting from i = 1(yi - Mean(y1, y2,… yn))^2

Question 43

Variance

Answer 43

minimum value of Rsq(h) is the mean squared deviation from the mean, measures the squared distance of each data point from the mean, on average

Question 44

standard deviation

Answer 44

square root of variance

Question 45

empirical risk for absolute loss

Answer 45

Rabs(h) = 1/n summation of n starting from i = 1|yi - h|

Question 46

Rabs(h) is minimized when

Answer 46

h* = Median(y1, y2,… yn)

Question 47

Minimum value of Rabs(h) is…

Answer 47

mean absolute deviation from the median
(1/n)SUMMATION of n from i = 1|yi - Median(y1, y2,…yn)|

Question 48

empirical risk for 0-1 Loss

Answer 48

R0,1(h) = 1/n Summation of n starting from i = 1 {0 - yi = h, 1 yi doesn’t equal h

proportion (between 0 and 1) of data points not equal to h

Question 49

R0,1(h) is minimized when

Answer 49

h* = Mode(y1,y2…yn)

Question 50

the minimum value of R0,1(h)

Answer 50

proportion of data points not equal to mode

Question 51

simple linear regression model

Answer 51

H(x) = w0 + w1x

Question 52

when using squared loss

Answer 52

h* = Mean(y1, y2… yn)
Rsq(h*) = Variance(y1, y2, … yn)

Question 53

When using absolute loss

Answer 53

h* = Median(y1, y2… yn)
Rabs(h*) = MAD from median

Question 54

R0,1(h) is minimized when

Answer 54

h* = Mode(y1,y2,… yn)
so therefore R0,1(h*) is the proportion of data points not equal to the mode

Question 55

minimum value of R0,1(h) is the

Answer 55

proportion of data points not equal to mode

Question 56

higher value means

Answer 56

less of the data is clustered at the mode

Question 57

hypothesis function

Answer 57

H, takes in an x as input and returns a predicted y

Question 58

parameters define

Answer 58

the relationship between the input and output of a hypothesis function

Question 59

Since linear hypothesis functions are of the form H(x) = w0 + w1x, we can re-write Rsq

Answer 59

Rsq(w0, w1) = 1/n Summation n from i = 1 (yi - (w0 + w1xi))^2

Question 60

Minimize mean squared error

Answer 60

Take partial derivatives with respect to each variable
set all partial derivatives to 0
solve the resulting system of equations
ensure that you’ve found a minimum, rather than a maximum or saddle point

Question 61

We have a system of two equations and two unknowns (w0 and w1)
-2/n Summation of n from i = 1 ( yi - (w0 + w1xi)) = 0

-2/n Summation of n from i = 1 (yi - (w0 + w1xi))xi = 0

Answer 61

solve for w0 in first equation, result becomes best intercept
plut w0* into second equation and solve for w1

Question 62

correlation

Answer 62

linear association, pattern that looks like a line

Question 63

association

Answer 63

any pattern

Question 64

correlation coefficient ,r

Answer 64

measure of strength of linear assocaition of two variables, x and y, measures how tightly clustered a scatter plot is around a straight line, between -1 and 1

Question 65

correlation coefficient, r is defined

Answer 65

average of the product of x and y when both are in standard units

Question 66

slope : w1* = r(sigma y / sigma x)

Answer 66

units of y per units of x