ConvNeXt

Question 1

Explain in a few words the algorithm of MixUp augmentation

Answer 1

The user initiates the parameter alpha

Each time in training:
Takes 2 random images from the dataset
Randomise lambda from the Beta distribution with param alpha.
mixing the images and the labels with the lambda parameter

Question 2

Give the equation for mixing the images and the labels once lambda is drawn

Answer 2

lambdax_1 + (1-lambda)x_2

Question 3

In MixUp paper - what is the range of alpha values they recommend

Answer 3

alpha between 0.1 to 0.4
For bigger alpha values they found underfitting
They didn’t mention for smaller alpha values

Question 4

Explain the need for label smoothing in multiclass problem

Answer 4

In a multiclass problem, people usually use the cross entropy loss.
When a model uses cross-entropy loss to learn it creates a model that sometimes can be ‘overconfident’

Question 5

Explain how in label smoothing the model is not becoming ‘overconfident’

Answer 5

In label smoothing, the target probability has decreased to 0.9 for example (can be more) and the model would want the unnormalised logits to be closer to each other

Question 6

What are unnormalised logits

Answer 6

logits that are the last layer of the neural network

Question 6

How one normalises the logits

Answer 6

Softmax on the logits

Question 7

What is the difference between a Convolution operator to a Depth-wise convolution operator

Answer 7

That a convolution with f filters will have all the filters being performed on all the channels (& all the image).
In Depth-wise convolution, we have to have f=c which c is the number of channels. Then each filter only works on 1 channel.

Question 8

What is the problem with the normal convolution in a computational sense

Answer 8

Each kernel looks at all the channels, and it was shown to be redundant. Plus, the computations amount goes up quadratically to the input channel size.

Question 9

Describe the depthwise separable convolution

Answer 9

First take a depthwise convolution - which is to have the same amount of filters as channels, and each filter looks at only 1 channel.
Then take pointwise convolution - which is to have a 1x1 convolution that goes through all the channels.

Question 10

What is the computational cost of a normal convolution

Answer 10

h * w * c_in * c_out * k^2

Question 11

What is the computational cost of a separable depthwise convolution

Answer 11

h * w * c_input * (c_output + k^2)

Question 12

What is the difference in I/O between normal convolution and separable depthwise convolution with expansion layer

Answer 12

I/O of convolution is h * w * c^2 + k^2 * c^2
I/O of separable depthwise w expansion is:
h * w * 14 * c +12 * c^2 + k^2 * c ->
which is roughly 7 times bigger .

Question 13

What are FLOPs

Answer 13

FLoating-point OPerations

Question 14

What are FLOPS

Answer 14

FLoating-point OPerations per Second

Question 15

What happens when reducing the NN size with Depthwise separable convs followed by expansion layers.

Answer 15

We get an increase memory access and increase latency.

Question 16

What is the idea in mobileNet to combat accuracy drop by using DW separable conv instead of normal conv

Answer 16

To add expansion layers - they chose 6 times as the input layer.

Question 17

What is the idea that PConv is using

Answer 17

There is spatial redundancy in the channel domain which then PConv can only work on 1/4 of the channels and still not lose spatial information

Question 18

How does PConv is working

Answer 18

It is a normal 2d conv that is working on only part of the channels.

Question 19

What is the block of FasterNet

Answer 19

A PConv followed by 2 PWConvs for expansion and reduction.

Question 20

What were their reasons for putting BN and ‘activation’ only once in each block

Answer 20

Overuse of activations was shown in MobileNet to have destroyed information in the low-dimensional space. Plus choosing BN cause it can be combined with a conv2d.