Quiz #3

Question 1

What three partial derivatives must we calculate for backpropagation in a convolutional layer?

Answer 1

1. dL/dh_in = dL/dHout * dHout/dHin (i.e. the partial derivative of the loss w.r.t. to the input from the previous layer. This is what gets passed back to the previous layer.)
2. dL/dk = dL/dHout * dHout/dK (i.e. the partial derivative of the loss w.r.t. the kernel values)
3. dL/dh_out (i.e. the partial derivative of the loss w.r.t. the output from the current layer. Remember that this is given because it is the “upstream gradient”)

Question 2

When calculating dL/dK, a kernel pixel does not affect all the values in the output? (True/False)

Answer 2

False, it does impact all the values of the output map. This is because we stride the kernel across the image, and we share weights in output maps.

Question 3

In a convolutional layer, when calculating the partial derivative of the loss w.r.t. the output of the layer (dL/dHout), we must incorporate ALL the upstream gradients and apply the chain rule over all the output pixels? (True/False)

Answer 3

True. This is because a single kernel pixel impacts the entire output since the kernel is strided across the image and weights are shared.

Question 4

If a node in a computation graph impacts multiple values in the output, what operation must be applied in the backward pass to ensure that information from each of those individual connections is incorporated in the backprop update?

Answer 4

We SUM the gradients from each of the upstream connections.

Question 5

If we take the partial derivative of the output pixel located at (r, c) w.r.t the kernel pixel located at (a’, b’), what expression represents the value of dY(r,c)/dK(a’,b’) if a’=b’=0?

Answer 5

dY(r,c)/dK(a’,b’) = x(r + a’, c + b’), so if a=b=0 then the derivative for this location is simply x(r, c)

Question 6

When calculating the partial derivatives for backpropagation in a convolutional layer, it is unnecessary to calculate the partial derivative of the loss L with respect to the input x (i.e. dL/dx) because that derivative does not impact the kernel weight value updates? (True/False).

Answer 6

False. While it’s true that dL/dx isn’t needed for updating the kernel values, this derivative is important because it is the gradient that gets passed back to the previous layer.

Question 7

What gradient needs to be calculated in order to pass back to the previous layer?

Answer 7

dL/dx, i.e. the partial derivative of the loss w.r.t the input of the current layer.

Question 8

For input pixel x(r’, c’), what impact does this pixel have on the output when calculating the gradient dL/dx?

Answer 8

It impacts the neighborhood around it (where part of the kernel touches it).

Question 9

When calculating the loss w.r.t. the input x (dL/dx), each pixel in the output is impacted by the input pixel? (True/False)

Answer 9

False. Since we’re striding the kernel across the input x, only the region where the kernel touches that input pixel are impacted. In those regions, we have to sum the gradients, hence the impact of all the pixels in this neighboring region.

Question 10

When calculating the gradient for a max pooling layer, every input pixel into the max pool layer impacts the gradient? (True/False)

Answer 10

False. The entire point of the max pooling operation is to perform dimensionality reduction by zeroing out all non-max pixels within the kernel region. Since only one pixel in the region will have a non-zero value, the gradients with respect to any other pixel in the region are also zero.

Question 11

A single pixel deep in a multi-layered CNN is only sensitive to the receptive field from the n -1 layer? (True/False)

Answer 11

False. A single pixel in the deeper layers is impacted a larger receptive field from the previous layer, which in turn is influenced by a larger receptive field from the previous layer, and so on. This is what gives CNN their representational power.

Question 12

What was the first major 21st century CNN architecture and when was it introduced?

Answer 12

AlexNet in 2012

Question 13

We tend to use fewer convolutional kernels (i.e. feature maps) as we go deeper into the network? (True/False)

Answer 13

False, generally speaking.

Question 14

What was the first modern CNN architecture to use ReLU instead of sigmoid or tanh?

Answer 14

AlexNet

Question 15

What activation function is used in AlexNet?

Answer 15

ReLU (it was the first to do this)

Question 16

What are the 5 key aspects of the AlexNet architecture (per the lectures)?

Answer 16

1. ReLU instead of sigmoid or tanh
2. Specialized normalization layers
3. PCA-based data augmentation
4. Dropout
5. Ensembling (7 models were trained together)

Question 17

As we go deeper into a CNN, the receptive field increases?

Answer 17

True

Question 18

What layers uses the most memory and why?

Answer 18

Convolutional Layers. Because we have to store the activations we obtained from the forward pass because the gradient calculation requires them for the backward pass. Since the output from the forward pass is so large (we’re striding across the entire image, remember) this leads to a large memory footprint.

Question 19

Convolutional layers tend to have more parameters than FC layers? (True/False)

Answer 19

False. Convolutional layers have a higher memory footprint, but FC layers have many more parameters since every weight is connected.

Question 20

What layers tend to have the most parameters and why?

Answer 20

Fully connected layers. This is because every weight is (as implied by its name) connected.

Question 21

For a fully connected layer with 12 input neurons, 10 output neurons and 3 channels, how many parameters are there (excluding bias terms)?

Answer 21

12103 = 360

Question 22

What are the two key aspects of the VGG architecture?

Answer 22

1. Repeated application of blocks:
   
   • 3x3 conv (stride=1, padding=1)
   • 2x2 max pool (stride=2)
2. Very large number of parameters (mostly from big FC layers)

Question 23

What are some of the main architectural differences between VGG and Alexnet?

Answer 23

1. Alexnet used a large stride, but this loses information. VGG uses a much smaller stride (1 for conv layers, 2 for maxpool) to preserve information.

Question 24

Roughly how many trainable parameters are required for VGG architectures versus Alexnet?

Answer 24

Hundreds of millions for VGG compared to 60-70M for AlexNet

Question 25

What are some of the key ideas used in the Inception architecure?

Answer 25

1. Repeated blocks
2. Multiscale features (i.e. concatenating convolutional features created using different kernel sizes and using the concatenated stack as the final output map).

Question 26

What is one of the downsides of the Inception architecture?

Answer 26

The use of multiscale features means that if each layer uses N multiscale convolutional features, we’ll have to perform N number of convolutions, instead of just a single one as in a normal architecture.

Question 27

What is the key idea of Residual Blocks?

Answer 27

1. Help prevent issues with vanishing gradients
2. Allow information from a layer to propagate to any future layer (forwards or backwards!)

They are useful because they improve gradient flow.

Question 28

What is optimization error?

Answer 28

It is the idea that even if your NN can theoretically perfectly model the world, there’s no guarantee that your optimization algorithm can find an optimal set of weights that will achieve that level of performance.

Question 29

What are the three types of error that pose a challenge to generalization?

Answer 29

1. Optimization error
2. Estimation error
3. Modeling error

Question 30

What is estimation error?

Answer 30

It is the idea that even if we find a set of weights that works well on the training set, there isn’t a guarantee that it will generalize to the test data. This could be because of overfitting, learning features that are good for the training set but don’t generalize to the test set, etc.

Question 31

What is modeling error?

Answer 31

It is the idea that their may be a disconnect between how the world actually works (reality) versus what the model is actually capable of representing. This could be because of insufficient capacity of the model, or using a model that isn’t suited to the task (for example trying to simple multi-class logistic regression for semantic segmentation - there’s no set of weights that could reasonable manage that complexity for such a simple model)

Question 32

In the context of transfer learning, when performing fine tuning we only update the parameters in the last layer? (True/False)

Answer 32

False. When fine tuning all the parameters are updated by training the pre-trained model on our smaller, domain specific dataset.

Question 33

In the context of transfer learning, when freezing the feature layer only the weights in that final layer are updated during training? (True/False)

Answer 33

True (this is often done when there isn’t enough data to train from scratch).

Question 34

What are two reasons you might want to reconsider using transfer learning for some specific problem?

Answer 34

1. If the source dataset you train on is very different from the target dataset
2. If you have enough data for the target domain (if so, then probably the only benefit of using transfer learning will be faster convergence)

Question 35

What are four visualization methods we can use to try to understand what a trained NN has learned?

Answer 35

1. Weights
2. Activations (output maps)
   3 Gradients
3. Robustness to perturbation

Question 36

Using dimensionality reduction, we can plot the activations of any layer (conv, linear, etc.) in 2D to try to understand the output space visually? (True/False)

Answer 36

True. PCA and t-SNE (most common) are frequently used to do this.

Question 37

What is a Saliency Map and what is it useful for?

Answer 37

The idea behind a saliency map is that we can backprop through a network all the way back to the image (or any arbitrary point in the computation graph) and look at the sensitivity of the loss to individual pixel changes. Large sensitivity implies important pixels.

Question 38

When visualizing gradients of loss w.r.t. an input image, why do we use the gradient of the classifier scores BEFORE the softmax layer?

Answer 38

Because the softmax layers can also improve the loss by “pushing down” the scores of the non-predicted classes to try to improve separability.

Question 39

What is guided backprop used for?

Answer 39

Many areas of an input image might actually DECREASE the feature activations. This can make trying to visualize gradients difficult. Guided backprop zeros out the negative gradients so that we only see the POSITIVE contributions to the activation.

Question 40

Why is optimizing the input image to GENERATE examples to increase class scores or activations useful, and how do we do this in practice?

Answer 40

It can be used to aid interpretability. Specifically, it can visually show us a great deal about what examples (not in the training set) are able to activate the network. We can do this by performing gradient ascent instead of descent.

Question 41

What is a “Gram Matrix”?

Answer 41

It represents feature correlations, and can be used when performing style transfer to represent a content/texture.

Question 42

You have an input volume of 32×32×3. What are the dimensions of the resulting volume after convolving a 5×5 kernel with zero padding, stride of 1, and 2 filters?

Answer 42

28 x 28 x 2

Output Size: [(H - K_h + 2*P) / S] + 1
H: Input height
K_h: Kernel height
P: Padding
S: Stride

[(32 - 5 + 2*0) / 1] + 1 = 28

Since we have square kernel and two feature maps (F=2) final output is 28 x 28 x 2

Question 43

You have an input volume of 32×32×3 and convolve it with a 5×5 kernel with zero padding, stride of 1, and 2 filters. How many total weights and biases will be in this layer?

Answer 43

F(K_h * K_w * C + 1) = 2 * (55*3 + 1) = 152

F: Number of feature maps
K_h: Kernel height
K_w: Kernel width
C: Number of input channels

Question 44

Suppose you have an input volume of dimension 64x64x16. How many parameters would a single 1x1 convolutional filter have, including the bias?

Answer 44

F(K_h * K_w * C + 1) = 1 * (11*16 + 1) = 17

F: Number of feature maps
K_h: Kernel height
K_w: Kernel width
C: Number of input channels

Question 45

Suppose your input is a 300 by 300 color (RGB) image, and you use a convolutional layer with 100 filters that are each 5x5. How many parameters does this layer have including the bias parameters?

Answer 45

F(K_h * K_w * C + 1) = 100 * (55*3 + 1) = 7600

F: Number of feature maps
K_h: Kernel height
K_w: Kernel width
C: Number of input channels

Question 46

You have an input volume that is 63x63x16 and convolve it with 32 filters that are each 7x7, and stride of 1. You want to use a same convolution. What is the padding?

Answer 46

Answer: 3

Use the formula [(H - K_h + 2*P) / S] + 1 and solve for P:
P = (S(X_h - 1) - H + K_h) / 2

P = (1*(63 - 1) - 63 + 7) / 2 = 3

X_h: Output height
H: Input height
K_h: Kernel height
P: Padding
S: Stride

Question 47

What is the resulting volume of padding a 15x15x8 input volume using pad=2?

Answer 47

19x19x8

Question 48

What is the output volume of a 32x32x16 input data after applying max pooling with a filter of size 2 and stride = 2? (Assume no padding)

Answer 48

[(H - K_h) / 2] + 1 = [(32 - 2) / 2] + 1 = 16 x 16 x 16

Question 49

If a feature (such as a bird’s beak) were translated a little bit, the location of the output values from convolutional layer would remain unchanged? (True/False)

Answer 49

False. Convolution has the property of ‘Equivariance’. A translation of the feature results in the output being shifted by the same amount.

Question 50

What are two important properties of convolution?

Answer 50

1. Invariance (features with small transformations/deformations should still activate the output)
2. Equivariance (no matter where the feature occurs in the image, the feature map will be activated, with the output values moving by the same translation)

Question 51

Explain the four different types of computer vision tasks?

Answer 51

1. Classification: Class distribution per image
2. Semantic Segmentation: Class distribution per pixel
3. Object Detection: List of bounding boxes with class distribution per box
4. Instance Segmentation: Class distribution per pixel with unique ID

Question 52

What is the output shape of a network used for semantic segmentation?

Answer 52

H x W x number of classes

Question 53

Fully connected layers are good for retaining spatial information? (True/False)

Answer 53

False. FC layers are simply vectors, so they don’t explicitly retain spatial information.

Question 54

In an encoder/decoder type network, how do we perform the de-convolution required in the decoder side of the network?

Answer 54

Transpose convolution, which is essentially just the inverse of the usual convolution operation.

Question 55

How does a max unpooling layer work in an encoder/decoder type network?

Answer 55

We simply cache the location of the max elements in the encoder, then on the decoder side, we set the same location to the max value, and zero the rest of the elements in that patch. It effectively upsamples the input.

Question 56

In an encoder/decoder type network, in a max unpooling layer, contributions from multiple windows must be multiplied together? (True/False)

Answer 56

False. They should be summed, not multiplied.

Question 57

When is an encoder/decoder type network considered to be symmetric?

Answer 57

This occurs when the decoder side of the network is an exact 1-to-1 inverse of the encoder network. So if the encoder network was [conv2d, maxpool, conv2d, maxpool], the symmetric decoder would be [deconv2d, maxunpool, deconv2d, maxunpool], and each of the conv/deconv/pool/unpool layers would have the exact same kernel sizes, strides, etc.

Question 58

There are two learnable parameters in a max unpooling layer? (True/False)

Answer 58

False. There are no learnable parameters. This is actually one of the challenges of unpooling layers: we’re not actually learning how to upsample, we’re simply just using the indices of the max in the encoder stage.

Question 59

It is not possible to learn a kernel to upsample an image with a convolution type layer? (True/False)

Answer 59

False. This is where we use Transposed Convolution (a.k.a fractionally strided convolution).

Question 60

What is Transposed Convolution used for, and how do we perform it in practice?

Answer 60

It’s used to create a learnable kernel for upsampling an image (useful for encoder/decoder style networks). It works by taking each input pixel, multiplying it by a learnable kernel, and “stamping” it on the output.

Question 61

When performing Transpose Convolution, contributions from multiple windows are summed together? (True/False)

Answer 61

True.

Question 62

In an encoder/decoder style network, if we don’t want to use learnable deconvolutional kernels in the decoder side of the network, what variant on this approach can we take to accommodate that desire?

Answer 62

We can take the corresponding encoder kernel and rotate it 180 degrees. (This is less common to do than the learnable kernel in practice, but could be useful in situations where we want to reduce the number of parameters are network has to learn).

Question 63

It is not possible to use a pre-trained backbone network trained on a basic classification task (i.e. transfer learning) for use in a semantic segmentation task? (True/False)

Answer 63

False. We would simply have to chop off the FC layers on the pre-trained network, then add in the corresponding symmetric decoder network.

Question 64

If we use a pre-trained network for an instance segmentation task, how many losses will there be in the network when training on our segmentation targets?

Answer 64

One per pixel. This is because for segmentation we get a probability distribution over all the classes for each pixel, so we simply use Cross Entropy Loss on a per pixel basis and calculate gradients w.r.t the loss for each pixel.

Question 65

What type of network architecture has been quite successful for image segmentation tasks, and how does it work?

Answer 65

U-Net. It works by using a symmetric encoder/decoder network, but it instead of just decoding from the bottleneck that sits in between the encoder/decoder sides of the network, the corresponding conv/pooling at each “scale” of the network are decoded.

Question 66

Convolutional layers cannot accept arbitrary input sizes?

Answer 66

False. This is one big advantage of conv layers over FC layers. This is able to be done because we’re striding the kernel over the entire input, so it doesn’t matter what the input size is.

Question 67

What type of network is popular for image-to-image type tasks?

Answer 67

Encoder/decoder style networks.

Question 68

What task is being performed in object detection?

Answer 68

Given an image, we want to output a list of bounding boxes with a probability distribution over classes per box.

Question 69

What two challenges arise in object detection tasks?

Answer 69

1. Variable number of bounding boxes possible

2. Need to determine candidate regions (position and scale) first

Question 70

What four numbers are important in object detection networks?

Answer 70

The bounding box, which can be defined as the location of the upper-left hand corner at (x, y), and the width W and height H of the box.

Question 71

Describe the multi-headed architecture used in an object detection network?

Answer 71

• One head that predicts distribution over class labels (classification problem)
• One head that predicts the location of the bounding box for each image region (regression problem)

Question 72

The two heads used in an object detection network use SEPARATE features? (True/False)

Answer 72

False. This is one advantage of deep learning. Both heads can SHARE features, and then be jointly optimized (by summing the gradients)

Question 73

For object detection networks, we feed in multiple gridded images (with different grid sizes for each image) to facilitate detection of objects at multiple scales? (True/False)

Answer 73

True. Redundant boxes are then combined using Non-Maximal Suppression (NMS).

Question 74

What is one advantage of the YOLO architecture for object detection?

Answer 74

It’s faster than many other alternatives because it looks for objects at a SINGLE SCALE.

Question 75

What is one type of metric that can be used for evaluating object detection tasks?

Answer 75

Mean Average Precision (MAP)

See 10:00 mark in Module 2 Lesson 9: Single Stage Object Detection for calculation details

Question 76

How does a two-stage object detection network work?

Answer 76

Instead of making dense predictions, we decompose the problem into two steps:

1. Find regions of interest with object-like things
2. Classify those regions (and refine bounding boxes)

Question 77

What is the key idea used in the Faster R-CNN algorithm?

Answer 77

The main idea is to also use the NN to generate the region proposals. It outputs an objectness score, and then the Top K regions are selected for classification.

Question 78

One challenge with the Faster R-CNN algorithm is that some parts (gradient w.r.t. bounding box coordinates) are not differentiable? (True/False)

Answer 78

True.

Question 79

In two-stage object detection, what are some ways we can generate region of interest (ROI) proposals?

Answer 79

1. Unsupervised learning (inefficient)
2. Map each ROI in image to corresponding feature maps (this way we don’t repeat conv for overlapping regions). Challenge with this is we end up with variable input sizes into FC layers (can be remedied using pooling to convert to fixed size)
3. Use the network to predict an region proposals with an “objectness score”, and take the Top K regions.

Question 80

What is the key idea used in the Fast R-CNN algorithm?

Answer 80

To REUSE computation by finding regions in FEATURE MAPS. One of the main challenges with this though is that you end up with variable input sizes to the FC layers. This can be remedied by using pooling to convert to a fixed grid size.

Question 81

At both steps in two-stage object detection networks, the bounding boxes are refined through regression? (True/False)

Answer 81

True.

Question 82

It is not possible to use a pre-trained backbone network trained on a basic classification task (i.e. transfer learning) for use in an object detection task? (True/False)

Answer 82

False.

Question 83

What are some of the main concepts to consider in the context of Machine Learning Bias and Fairness.

Answer 83

1. Fairness
2. Accountability
3. Transparency
4. Ethics
5. Safety/Security

Question 84

Consideration of the ethics of AI is separable from the pure science/engineering side of the field? (True/False)

Answer 84

False.

Question 85

Why can technology be considered political?

Answer 85

For many reasons, but one of the most salient being that technology can be used to exert power and control over other people.

Question 86

What is Goodhart’s Law?

Answer 86

“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”

In other words, if you incentivize optimization, you’ll get overfitting.

Question 87

What are two ways that Goodhart’s Law is applicable to ML in the context of fairness?

Answer 87

1. Use of benchmarks incentivizes the creation of algorithms that are well suited to those problems but might not generalize well outside that narrow domain (overfitting)
2. Incompatible (i.e. can’t fulfill one measure while also fulfilling another) and incommensurable (not even able to compare) fairness measures.

Question 88

What is the definition of a WELL-CALIBRATED classifier?

Answer 88

A classifier is well-calibrated if the probability of the observations with a given probability score of having a label is equal to the proportion of observations having that label.

Example: If a binary classifier gives a score of 0.8 to 100 observations, then 80 of them should be in the positive class.

Question 89

How is Group Calibration performed?

Answer 89

The scores for subgroups of interest are calibrated (or at least, equally mis-calibrated). This can be shown on a Reliability Plot (aka a Calibration Plot)

Question 90

Some models tend (empirically speaking) to have well-calibrated predictions, while other models tend to be overconfident? (True/False)

Answer 90

True. Logistic regression tends to be well calibrated. DL models (e.g. ResNet tend to be overconfident.

Question 91

What is Platt (Temperature) Scaling?

Answer 91

It’s a way of fitting overconfident models to be more well-calibrated. It’s done by taking a validation dataset and learning a transform that would make it well-calibrated.

Question 92

When performing Platt Scaling to improve model calibration, we use the same dataset as was used for original training? (True/False)

Answer 92

False. It’s very important to use a DIFFERENT validation dataset, although in practice it can be the same validation dataset that was used for other applications (e.g. during training).

Question 93

For a binary classifier, how many parameters must be learned for Platt Scaling?

Answer 93

Two, ‘a’ and ‘b’. During the Platt Process, these parameters are learned such that the output of our model through a sigmoid function parameterized by ‘a’ and ‘b’ matches a well-calibrated output (as determined by, for example, our reliability diagram).

q_hat = sigmoid(a*z_i + b)

Question 94

What parameter must be learned for multi-class Platt Scaling?

Answer 94

The Temperature ‘T’: q_hat = max_k[softmax(z_i/T)]

Question 95

What are the philosophical problems that arise from fairness techniques like Platt Scaling?

Answer 95

Platt scaling relies on segmenting people/things into groups. Ideally, we would like to treat people as individuals. Practically speaking, we have to consider groups to make the problem tractable, but how we select those groups (and what characteristics we will use to define them) will have inherent tradeoffs.

Question 96

What measure is used for “test fairness” for a binary classifier?

Answer 96

Positive Predictive Value (PPV): TP / (TP + FP)

Question 97

If we seek to equalize the FPR and FNR between two groups with DIFFERENT prevalences, the positive predictive value (PPV) will always be different? (True/False)

Answer 97

True. The PPV, FPR, FNR, etc. are all inter-related quantities based on the classification matrix. This is what gives rise to the “Fairness Impossibility Theorem(s)”.

Question 98

When does an instance of the “Fairness Impossibility Theorem” exist?

Answer 98

The impossibility theorem exists for ANY THREE (or more) measures of model performance derived (non degenerately) from the confusion matrix.

In a system of equations with three or more equations, the prevalence ‘p’ is determined uniquely. If groups have different prevalences, these quantities CANNOT be equal.

Question 99

What are the two central insights that are incorporated into The Markkula Center Framework for Ethical Decision-Making?

Answer 99

1. There are different ethical perspectives.

2. Process matters: do the moral math.

Question 100

What five steps comprise The Markkula Center Framework for Ethical Decision-Making?

Answer 100

1. Recognize an Ethical Issue
2. Get the facts
3. Evaluate options following different approaches/moral frameworks/value systems
4. Make a decision and test it
5. Act and reflect on the outcome

Question 101

What are operations are performed to calculate the gradient of the output of a layer calculated w.r.t. to the kernel values in a convolutional layer (dY/dK)?

Answer 101

It is the CROSS-CORRELATION between the UPSTREAM GRADIENT and the INPUT (until k1 x k2 output)

See 6:40/15:10 mark in Lesson 6.

Question 102

What are operations are performed to calculate the gradient of the loss w.r.t. to the input X in a convoluational layer (dL/dX)?

Answer 102

It is the CONVOLUTION between the UPSTREAM GRADIENT and the KERNEL (this can be implemented by simply flipping the kernel 180 degrees and performing cross-correlation)