HW 3 solutioun

Question 1

linear or non linear: perceptron

Answer 1

linear

Question 2

linear or non linear: multiplayer perceptron

Answer 2

nonlinear

Question 3

linear or non linear: decision tree

Answer 3

nonlinear

Question 4

linear or non linear: knn

Answer 4

nonlinear

Question 5

linear or non linear: logistic regression

Answer 5

linear

Question 6

Decision trees can perform _ classification

Answer 6

multiclass

Question 7

Decision trees can reduce overfitting by having a _ max depth

Answer 7

smaller

Question 8

An ensemble of decision trees is called a

Answer 8

forest

Question 9

The prediction is given by the _ node in a decision tree

Answer 9

leaf

Question 10

Feature values are encoded in the _ of decision tree

Answer 10

edges

Question 11

If there are N features, a decision tree can perfectly encode all permutations of
features with a depth of _

Answer 11

N

Question 12

If the decision boundary is linear, do we expect LDA or QDA to perform better on the
training set? On the test set?

Answer 12

We would expect QDA to perform better on the training set because its
increased flexibility will result in a closer fit. If the Bayes decision boundary is linear, we
expect LDA to perform better than QDA on the test set, as QDA could be subject to
overfitting.

Question 13

If the decision boundary is non-linear, do we expect LDA or QDA to perform better
on the training set? On the test set?

Answer 13

If the Bayes decision boundary is non-linear, we expect QDA to perform better
on both the training and test sets.

Question 14

In general, as the sample size n increases, do we expect the test prediction
accuracy of QDA relative to LDA to improve, decline, or be unchanged? Why?

Answer 14

We expect the test prediction accuracy of QDA relative to LDA to improve as n
gets bigger. In general, as the the sample size increases, a more flexible method will
yield a better fit as the variance is offset by the larger sample size.

Question 15

True or False: Even if the Bayes decision boundary for a given problem is linear, we
will probably achieve a superior test error rate using QDA rather than LDA because
QDA is flexible enough to model a linear decision boundary. Justify your answer.

Answer 15

Solution: False. With fewer sample points, the variance from using a more flexible
method, such as QDA, would likely result in overfitting, yielding a higher test error rate
than LDA.

Question 16

Q5 For parts each of the following, indicate whether each method is more or less
flexible than least squares (plain linear regression). Describe how each method’s trade-
off between bias and variance impacts its prediction accuracy. Justify your answers.

Answer 16

Solution: Puts a budget constraint on least squares. It is therefore less flexible. The
lasso will have improved prediction accuracy when its increase in bias is less than its
decrease in variance.