DL-05b - Convolutional neural network - Architectures Flashcards
DL-05b - Convolutional neural network - Architectures
Who is behind LeNet?
Yann LeCun
DL-05b - Convolutional neural network - Architectures
When did Yann LeCun publish LeNet?
1998
DL-05b - Convolutional neural network - Architectures
What was LeNet used for?
Reading zip codes/digits.
DL-05b - Convolutional neural network - Architectures
Who created AlexNet?
Alex Krishevsky
DL-05b - Convolutional neural network - Architectures
When did Alex Krishevsky publish AlexNet?
2012
DL-05b - Convolutional neural network - Architectures
What network was published in 1998?
LeNet
DL-05b - Convolutional neural network - Architectures
What network was published in 2012?
AlexNet
DL-05b - Convolutional neural network - Architectures
What challenge did AlexNet attempt to solve?
ImageNet,
DL-05b - Convolutional neural network - Architectures
How many different labels does ImageNet have?
1,000
DL-05b - Convolutional neural network - Architectures
What’s the architecture of AlexNet?
5 conv layers, 3 FC layers, overlapping max pooling
DL-05b - Convolutional neural network - Architectures
How many parameters did AlexNet have?
~62 million
DL-05b - Convolutional neural network - Architectures
When was VGGNet published?
2014
DL-05b - Convolutional neural network - Architectures
Who published VGGNet?
Simonyan and Zisserman
DL-05b - Convolutional neural network - Architectures
How many layers did VGGNet have?
16, hence it’s often called VGG16.
DL-05b - Convolutional neural network - Architectures
What was the key innovation of VGGNet?
Depth of network is key to performance.
DL-05b - Convolutional neural network - Architectures
What were some drawbacks of VGGNet relative to its contemporaries? (2)
- more expensive to evaluate
- requires large memory and computation power.
DL-05b - Convolutional neural network - Architectures
Whwne was GoogLeNet published?
2014
DL-05b - Convolutional neural network - Architectures
Who published GoogLeNet?
Szegedy et al.
DL-05b - Convolutional neural network - Architectures
How deep was GoogLeNet?
22 layers
DL-05b - Convolutional neural network - Architectures
What key innovations did GoogLeNet have? (3)
- Reduced parameters with better performance
- Inception modules
- Average pooling instead of FC layers
DL-05b - Convolutional neural network - Architectures
What is the inception module?
Block that splits into multiple CNN blocks with different kernel sizes.
DL-05b - Convolutional neural network - Architectures
In inception modules, what do concatenation layers do?
Combines all the outputs and feature maps from the conv filters into one single output of the Inception module.
DL-05b - Convolutional neural network - Architectures
In inception modules, what solves the problem of high computation cost?
1x1 conv blocks to reduce number of input channels.
(192 -> 1x1-> 16) -> (16 -> 5x5 -> 32)
(See image)
DL-05b - Convolutional neural network - Architectures
Describe the full inception module.
(See image)
DL-05b - Convolutional neural network - Architectures
What is depicted? (See image)
A full inception module.
DL-05b - Convolutional neural network - Architectures
What is depicted? (See image)
A naive inception module.
DL-05b - Convolutional neural network - Architectures
What are the benefits of using inception modules? (3)
- Extracts features at varying scales using varying filter sizes
- minimizes computation load
- utilizes 1x1 conv filters for better cross-channel pattern learning
DL-05b - Convolutional neural network - Architectures
When was Inception 4 released?
2016
DL-05b - Convolutional neural network - Architectures
What tricks are introduced to make Inception better later on? (2)
Improvements to speed through:
- (5x5) factored into 2 x (3x3) convs
- (nxn) factored into (1xn) and (nx1).
DL-05b - Convolutional neural network - Architectures
When was ResNet published?
2015
DL-05b - Convolutional neural network - Architectures
Who was behind the ResNet paper?
He et al.
DL-05b - Convolutional neural network - Architectures
Which company published ResNet?
Microsoft.
DL-05b - Convolutional neural network - Architectures
How many layers did the original ResNet model have?
152
DL-05b - Convolutional neural network - Architectures
What idea did ResNet introduce?
Residual learning
DL-05b - Convolutional neural network - Architectures
What block is this? (See image)
A residual block
DL-05b - Convolutional neural network - Architectures
What intuition can we use for ResNets?
The residual blocks makes the layers comparable to ensembles of different models.
DL-05b - Convolutional neural network - Architectures
What are the “two modes” for how layers are trained in ResNets?
Layers are either trained or training is skipped for layers using skip connections.
DL-05b - Convolutional neural network - Architectures
When were Inception-ResNets published?
2017
DL-05b - Convolutional neural network - Architectures
Who were behind Inception-Resnets?
Szegedy et al.
DL-05b - Convolutional neural network - Architectures
What is SENet short for?
Squeeze and Excitation Networks
DL-05b - Convolutional neural network - Architectures
Who were behind SENet?
Hu et al.
DL-05b - Convolutional neural network - Architectures
When was SENet published?
2017
DL-05b - Convolutional neural network - Architectures
What does a squeeze-and-excitation block attempt to do?
Enables the network to perform dynamic channel-wise feature recalibration.
DL-05b - Convolutional neural network - Architectures
What is the name of the block that attempts to perform dynamic channel-wise feature recalibration?
Squeeze-and-excitation block
DL-05b - Convolutional neural network - Architectures
In a squeeze-and-excitation block, what is the purpose of squeezing?
squeeze global spatial information into a channel descriptor.
DL-05b - Convolutional neural network - Architectures
In a squeeze-and-excitation block, what is the purpose of excitation?
Excitation operation is used to fully capture channel-wise dependencies.
DL-05b - Convolutional neural network - Architectures
Squeeze-and-excitation block looks like?
(See image)
DL-05b - Convolutional neural network - Architectures
What is the technique used in EfficientNet called?
Compound scaling
DL-05b - Convolutional neural network - Architectures
Which network introduced Compound scaling?
EfficientNet
DL-05b - Convolutional neural network - Architectures
What is Compound scaling?
An equation for setting the width, depth and image resolution of a network.
DL-05b - Convolutional neural network - Architectures
What is the equation for compound scaling?
(See image)
DL-05b - Convolutional neural network - Architectures
What is MBConv short for?
Mobile inverted bottleneck
DL-05b - Convolutional neural network - Architectures
What is the shortened name of Mobile inverted bottleneck?
MBConv
DL-05b - Convolutional neural network - Architectures
What is the main idea behind MBConv?
They use an inverted structure for efficiency reasons.
DL-05b - Convolutional neural network - Architectures
What does MBConv blocks do? (3 steps)
- Increases number of channels with 1x1 conv
- Depth-wise convolutions
- Downsample channels to initial value using 1x1 conv
DL-05b - Convolutional neural network - Architectures
Who is behind EfficientNet V2?
Tan and Le.
DL-05b - Convolutional neural network - Architectures
When was EfficientNet V2 published?
2021
DL-05b - Convolutional neural network - Architectures
What does EfficientNet V2 use? (3) (BLR)
- Combines MBConv and Fused-MBConv blocks
- Uses progressive learning (changes image size dynamically)
- Uses adaptive regularization
DL-05b - Convolutional neural network - Architectures
How does ResNet tackle the error propagation challenge?
Through skip connections.
DL-05b - Convolutional neural network - Architectures
What makes error propagation efficient in a ResNet architecture?
Residual connections allow error to bypass layers, ensuring efficient error propagation and training.
DL-05b - Convolutional neural network - Architectures
What architecture is this? (See image)
MBConv
DL-05b - Convolutional neural network - Architectures
Label the masked parts.
(See image)
DL-05b - Convolutional neural network - Architectures
Describe the full architecture of MBConv.
(See image)
DL-05b - Convolutional neural network - Architectures
What architecture is this? (See image)
Fused MBConv
DL-05b - Convolutional neural network - Architectures
Label the masked parts.
(See image)
DL-05b - Convolutional neural network - Architectures
Describe what parts a fused-MBConv consists of.
(See image)