Foundations

Question 1

What’s broadcasting?

Answer 1

It enables element-wise operations on arrays or tensors of different shapes efficiently and intuitively. It plays a critical role in simplifying the implementation of mathematical operations in high-dimensional data processing and deep learning frameworks.

Question 2

What’s the mathematical formula of the first part of a neural network?

Answer 2

out = xi· wi + b

x = input
w = weight
b = bias

Question 3

What´s a scalar?

Answer 3

During broadcasting with a scalar the scalar can be thought of as being expanded to a tensor of the
same shape as the input tensor

Question 4

What are the two criteria that tells us if a tensor is broadcastable or not?

Answer 4

• Two tensors of higher dimensions are broadcastable if the length of the axes of the lower ranked tensor
  match the length of the trailing axes in the higher ranked tensor
  – Two lengths of axes match in the sense of broadcasting if either one of them is 1 or they are equal

trailing axes: trailing axes = n last dimensions, depends on the size of the tensor you’re broadcasting with

Question 5

What does the “expand_as” method do?

Answer 5

it virtually expands the tensor c to have the same shape as m

Question 6

What does the method “unsqueeze()” do?

Answer 6

it adds an axes of length one at the defined position

Question 7

Why is it important to initialize the weights?

Answer 7

to prevent that In the case of an increasing standard deviation the values may overflow and in the case of too small weights the values may vanish eventually

Question 8

What are the different types of initialization techniques?

Answer 8

• Xavier: scaling factor (1/√n), n = # of inputs
• Kaming: scaling factor (2/√n)

Question 9

What disadvantages does the Xavier initialization has over the Kaming?

Answer 9

Xavier method doesn’t preserve the std dev

Question 10

Which of the following statements is true about weight initialization? (Multiple Choice)
1. The ReLU activation function preserves the distribution of values by leaving the majority of them unchanged (the linear part) and mapping values below zero to zero, which is acceptable given the desired mean of zero.
2. It is satisfactory to have the mean and variance of the distribution of output values average out to zero and one, respectively, across multiple initializations. In individual cases, these values may deviate.
3. If a pre-trained model is used and no new weights are added, we do not need Xavier and Kaming initialization at all.
4. In larger networks, the initialization process is relatively less critical due to the involvement of numerous random numbers. As a result, the likelihood of individual numbers impacting the overall outcome is mitigated.
5. Even with Xavier and Kaming initialization, it can occur by chance that the weights of a neural network are initialized in such a way that the network is unable to learn anything useful.

Answer 10

2,3,5

Question 11

Which of the following statements is true about ANNs? (Multiple Choice)
1. All standard weight operations can be expressed as matrix multiplications. This makes neural network operations so efficient when executed on GPUs.
2. A single neuron cannot be implemented in plain Python, PyTorch or a similar deep learning library is required.
3. It is not possible to express the weights of a layer in a single matrix because the biases have to be separated from the input weights.
4. If one could obtain a fast enough GPU, while using only plain Python code, one could beat PyTorch’s CPU execution time for matrix multiplication.

Answer 11

1

Question 12

How do we know if two tensors are broadcastable?

Answer 12

All their dimensions (in the same positions) are compatible. If one of the two is not equal, but one of them is equal to 1, it can be broadcasted

Question 13

Why is PyTorch generally faster than plain Python for deep learning tasks?

A) It has a more intuitive API
B) It uses functions implemented in C/C++
C) It has better visualization tools
D) It requires less memory

Answer 13

Answer: B) It uses functions implemented in C/C++

Question 14

Which of the following is a common issue that can occur with improper weight initialization in a neural network?

A) Faster convergence
B) Overfitting
C) Vanishing or exploding gradients
D) Reduced model complexity

Answer 14

Answer: C) Vanishing or exploding gradients

Question 15

If a neural network has 3 layers with weights initialized using a normal distribution with a mean of 0 and a variance of 1, what is the expected variance of the output for each layer?

Answer 15

Answer:

The variance of the output remains 1 if the weights are initialized properly considering the input dimensions, assuming no activation or normalization layers that affect the output variance.

Question 16

Explain the concept of broadcasting in the context of neural network operations. Provide an example of how it can be used in matrix operations.

Answer 16

Answer:
Broadcasting is a technique that allows numpy or PyTorch to perform element-wise operations on arrays of different shapes by automatically expanding the smaller array to match the shape of the larger one. This is useful for efficient computation without the need for manual array resizing.

Example:
If A=np.array([[1,2],[3,4]]) and
B=np.array([1,2]), broadcasting allows you to perform C=A+B, resulting in C=np.array([[2,4],[4,6]]).

Question 17

Discuss the importance of proper weight initialization in deep learning models. What issues might arise from poor initialization?

Answer 17

Answer:
Proper weight initialization is crucial because it affects the convergence speed and stability of a deep learning model during training. Poor initialization can lead to problems such as vanishing or exploding gradients, where gradients either become too small to propagate back effectively or grow too large, causing numerical instability. Good initialization techniques, like Xavier or Kaming initialization, aim to maintain a stable variance of outputs and gradients throughout the network.

Question 18

You implemented a neural network but noticed that the gradients of the weights are either too large or too small during training. What could be the possible reasons, and how would you address this issue?

Answer 18

Answer:
The issue could be due to improper weight initialization, leading to vanishing or exploding gradients. To address this, one could use initialization methods like Xavier (Glorot) or Kaming initialization, which take into account the number of input and output units to maintain gradient flow. Additionally, gradient clipping can be used to prevent gradients from becoming too large.

Question 19

Explain the difference between the forward pass and the backward pass in a neural network. Why is the backward pass critical for model training?

Answer 19

Answer:
The forward pass in a neural network involves computing the output predictions from the input data by passing it through the network layers sequentially. The backward pass, also known as backpropagation, involves computing the gradients of the loss function with respect to the model parameters, enabling the model to learn by updating its weights. The backward pass is critical for training because it allows the model to minimize the loss function by adjusting the weights in the direction that reduces the error.