Foundations

Question 1

Even with Xavier and Kaming initialization, it can occur by chance that the weights of a neural network are initialized in such a way that the network is unable to learn anything useful.

Answer 1

True

Question 2

If a pre-trained model is used and no new weights are added, we do not need Xavier and Kaming initialization at all.

Answer 2

True

Question 3

It is satisfactory to have the mean and variance of the distribution of output values average out to zero and one, respectively, across multiple initializations. In individual cases, these values may deviate.

Answer 3

True

Question 4

Which tensors can be added to each other?

Answer 4

Same shape, not caring about commas or spaces

Question 5

All standard weight operations can be expressed as matrix multiplications. This makes neural network operations so efficient when executed on GPUs.

Answer 5

True

Question 6

Gradient of bias

Answer 6

2⋅(out−target)⋅1/n

​

Question 7

Gradient of weight

Answer 7

Gradient of bias x input

Question 8

Gradient of input

Answer 8

Gradient of bias x weight

Question 9

Elementwise arithmetics for tensors

Answer 9

• with tensors all basic operators (+, -, *, /, >, <, ==) are applied elementwise
• both tensors involved in the calculation need to have the same shape
• operation is executed for every pair of elements at the same positions in the two tensors and thus, the result is a tensor with same shape as input tensors

Question 10

Kaming Initialization

Answer 10

When using a ReLU activation the scaling factor of √(2/n_input) preserves the standard deviation

Question 11

Xavier Initialization

Answer 11

• suitable scaling factor (1/√n_input)
• Passing the values through the activation function may alter the mean and standard deviation again causing the values to vanish or overflow

Question 12

What is calculated during backpropagation?

Answer 12

Gradients

Question 13

What gradients show?

Answer 13

Gradients indicate how the network should adjust its parameters to minimize the loss, not directly the quality of the network.

Question 14

What is forward pass?

Answer 14

process of passing input data through the layers of a neural network to produce an output (e.g., predictions or logits).

Question 15

What is backpropagation?

Answer 15

process of computing gradients of the loss function with respect to the network’s parameters (weights and biases) using the chain rule of calculus