Lecture 4

Question 1

Parameters

Answer 1

Variables learnt (found) during training e.g. weights

Question 2

Hyperparamters

Answer 2

Variables whose value is set before the training process begins (alpha and k (from k-NN))

Question 3

Loss Function (or error)

Answer 3

Should be minimized for a single training example to achieve the objective; Part of a cost function

Question 4

Cost Function

Answer 4

average of your loss functions over the entire training set (mean square error); A type of objective function

Question 5

Objective Function

Answer 5

Any function that you optimize during training (maximum likelihood, divergence between classes)

Question 6

Gradient Descent

Answer 6

Finds optimal values for w;

Is an iterative optimization algorithm that operates over a loss landscape (cost function);

Follows the slope of the gradient W to reach the minimum cost

Question 7

Gradient of Cost Function

Answer 7

The direction of the steps to achieve the goal

Question 8

Learning Rate

Answer 8

The size of steps took in any direction

Question 9

Sigmoid Function (Logistic Function)

Answer 9

Assumes a particular functional form (a sigmoid) is applied to the linear function of the data; the output is a smooth and differentiable function of the inputs and the weights

Question 10

Cross Entropy (Logarithmic Loss)

Answer 10

Predicts class probability compared to actual class for an output of 0 or 1;
The score calculated penalizes probability based on how far it is from actual value;
Penalty is logarithmic in nature

Question 11

Regularization

Answer 11

Any modification to a learning algorithm that is intended to reduce its generalization error but not its training error;
Solves overfitting;
Used when we don’t have enough samples to create a good logistic regression classification model.

Question 12

Activation Functions

Answer 12

Applied on the hidden units;
achieve nonlinearity;
popular activation function

Question 13

Feedforward

Answer 13

Calculates the predicted output (y); inference

Question 14

Backpropogation

Answer 14

Updating the weights and biases; learning

Question 15

Artificial Neural Networks

Answer 15

• Highly expressive non-linear functions
• Highly parallel network of logistic function units
• minimizes sum of square training errors plus weight squared (regularization)
• Uses gradient descent as training procedure

Question 16

Is logistic regression used for classification or regression?

Answer 16

Logistic Regression is used for classification only.

Question 17

Does regularization work better with a small set of features (low dimensions) or a large set of features (high dimensions)?

Answer 17

Regularization works well with a large number of features.

Question 18

Is logistic regression used for binary or multiclass classification?

Answer 18

It can be used for both

Question 19

What are the steps for a gradient decent algorithm?

Answer 19

1. Initialize random weights and bias
2. Pass an input through the network and compute predicted values from output layer
3. Calculate error between the actual value and the predicted value
4. Go to each weights which contributes to the error and change its respective values to reduce the error
5. Reiterate until you find the best weights of network