Terminology Week 2

Question 1

What are hyperparameters?

Answer 1

Hyperparameters are the configuration settings used to tune how the model is trained.

Question 2

Describe gradient descent.

Answer 2

It is a strategy to minimize loss gradually.

We repeatedly take small steps in the direction that minimizes loss. We call these Gradient Steps (But they’re really negative Gradient Steps)

Question 3

What is a gradient?

Answer 3

Is the slope of a given point of a curve. I can be positive or negative. Or zero (which is at the bottom of a convex curve)

Also called the derivative of the loss function.

The gradient of is a two-dimensional vector that tells you in which direction to move for the maximum increase in height. Thus, the negative of the gradient moves you in the direction of maximum decrease in height. In other words, the negative of the gradient vector points into the valley.
In machine learning, gradients are used in gradient descent. We often have a loss function of many variables that we are trying to minimize, and we try to do this by following the negative of the gradient of the function.

Question 4

When does weight initialisation matter (in gradient descent)?

Answer 4

For Non-convex problems it matters!
think of an egg crate - more than one minimum;
Strong dependency on initial values

For convex problems, weights can start anywhere (say, all 0s) (convex: think of a bowl shape) = Just one minimum

Question 5

Describe stochastic gradient descent.

Answer 5

Stochastic Gradient Descent: calculate the loss with one example at a time. The example is chosen randomly.

Question 6

Describe Mini-Batch Gradient Descent.

Answer 6

Mini-Batch Gradient Descent: batches of 10-1000

Loss gradients are averaged over the batch

Question 7

Describe gradient descent step by step with your words.

Answer 7

I choose a set of parameters. (w and b for linear regression). This is means, we have a new model.
Calculate y (label) for with the given parameters for the  example feature (x). (we apply the model)
Calculate the loss (e.g. with square loss function) - i.e. how far away is my predication y' from the real value (example y).
Do this for all examples and calculate the mean square loss.
Now calculate the gradient of the loss function.  Which gives a direction in which to change the parameters to minimize the loss (it is 0 when you found a minimum loss.)

Question 8

What can you do to find out the direction in which the hyperparameter should be changed?

Answer 8

Calculate the (negative) gradient of the loss function.

Question 9

How is a machine learning model trained?

Answer 9

A Machine Learning model is trained by starting with an initial guess for the weights and bias and iteratively adjusting those guesses until learning the weights and bias with the lowest possible loss.

Question 10

What does it mean when a model is convergent?

Answer 10

The changes of parameters/model result only in very small changes in loss function.

Usually, you iterate until overall loss stops changing or at least changes extremely slowly. When that happens, we say that the model has converged.

Question 11

What is a batch (in gradient descent)?

Answer 11

In gradient descent, a batch is the total number of examples you use to calculate the gradient in a single iteration.

Question 12

When is redundancy in example data useful?

Answer 12

Some redundancy can be useful to smooth out noisy gradients