Week 6

Question 1

Overfitting

Answer 1

When fitting the observed facts, the data seen so far, well, does no longer indicate a small out-of-sample error.

Question 2

Deterministic noise

Answer 2

The part of the target function that is outside of the best approximation to the target function.

Question 3

Stochastic noise

Answer 3

Random noise that cannot be modeled.

Question 4

State two differences between deterministic and stochastic noise

Answer 4

1) If we generate the same data again, the deterministic noise would be the same but the stochastic noise would be different.
2) Different models capture different parts of the target function -> deterministic noise depends on the learning model you use.

Question 5

The variance of the stochastic noise is captured by the variable…

Answer 5

sigma_squared

Question 6

What is the cause of overfitting?

Answer 6

Noise

Question 7

Name two cures for overfitting:

Answer 7

1) Regularization
2) Validation

Question 8

Regularization

Answer 8

Attempts to minimize Eout by working through the equation
Eout(h) = Ein(h) + overfit penalty

Question 9

Validation

Answer 9

Estimates the out-of-sample error directly

Question 10

validation set

Answer 10

A subset from the data that is not used in training.

Question 11

When is a set no longer a test set?

Answer 11

When it affects the learning process in any way.

Question 12

How is the validation set created?

Answer 12

The data set D is divided in a training set of size (N-K) and a validation set of size K. A final hypothesis is learned by the algorithm using the training set. Then the validation error is calculated with the validation set.

Question 13

What is the rule of thumb for determining K in validation?

Answer 13

K = N/5
Use 80% for training and 20% for validation.

Question 14

Cross validation estimate

Answer 14

The average value of the error made by gn on its validation set.

Question 15

Wat denoteert H.theta?

Answer 15

De polynomen van graad d (~erboven)

Question 16

Wat is theta(x)? (z)

Answer 16

de x-vector (bijv (1,x).T)p met daarbij nog x^2 … x^d(~erboven)

Question 17

Wanneer zijn we aan het overfitten?

Answer 17

Als het algorithme probeert te leren van de ruis in plaats van van het patroon.

Question 18

Wat is de oorzaak van overfitting in een voorbeeld zonder ruis?

Answer 18

Deterministische ruis: we kunnen met de beste hypothesefunctie niet perfect de target functie benaderen

Question 19

Hoe kun je E.out(g.D) uitdrukken in 3 dingen?

Answer 19

var+bias+sigma_squared

Question 20

Wat is de var bij het berekenen van E.out(g.D) in 3 delen?

Answer 20

E.D,x [ ( g.D(x) - gemiddelde g(x) ) ^2 ]

Question 21

Wat is de bias bij het berekenen van E.out(g.D) in 3 delen?

Answer 21

bias = E.x [ (gemiddelde g(x) - f(x) ) ^2 ]

Question 22

Wat is de sigma_squared bij het berekenen van E.out(g.D) in 3 delen?

Answer 22

sigma_squared = E.epsilon,x [ (epsilon(x))^2]

Question 23

Hoe bereken je in lineare regressie met ruis de verwachte in-sample error?

Answer 23

sigma_squared * (1 - (d+1)/N )

Question 24

Hoe bereken je de verwachte out-of-sample error in lineare regressi met ruis?

Answer 24

sigma_squared * (1 + (d+1)/N)

Question 25

Hoe bereken je de verwachte generalization error in lineare regressie met ruis?

Answer 25

2*sigma_squared * ((d+1)/N)

Question 26

Wat is H0 in principe?

Answer 26

De verzameling van alle hypotheses van de vorm f(x) = b

Question 27

Wat is H1 in principe?

Answer 27

De verzameling van alle hypotheses van de vorm f(x) = ax + b

Question 28

Diff. linear classification and linear regression:

Answer 28

Classification = binary (or trinary…)
Regression = real numbers

Question 29

Diff. Logistic regression and linear regression:

Answer 29

Logistic regression = real between 0 and 1
Linear regression = just real.

Question 30

What does linear regression use to measure the distance between h(x) and f(x)

Answer 30

Mean Square Error (MSE)

Question 31

What is the formula for the mean squared error?

Answer 31

E.in = 1/N * (h(x.i) - y.i) ^2
for all i in N.

Question 32

What is h(x) in 1) linear regression, 2) perceptron and 3)logistic regression?

Answer 32

1) h(x) = s
2) h(x) = sign(s)
3) h(x) = theta(s)
for s = w.T * x

Question 33

Give the logistic regression algorithm (2 steps):

Answer 33

For every time step, do
1) compute the gradient
2) Update the weights with fixed learning rate eta:
w(t+1) = w(t) - eta * E.in gradient

Question 34

Stochastic gradient descent

Answer 34

Does not use all examples for E.in, it uses a single data sample error or several.

Question 35

How do represent the XOR function in Ands and Ors?

Answer 35

f(X) = (not h1 AND h2) OR (h1 AND not h2)

+1 if exactly one of h1, h2 equals +1

Question 36

What does more nodes per hidden layer do with the approximation and generalization of the MLP?

Answer 36

approximation goes up,
generalization goes down.