10_advanced deep learning concepts

Question 1

By which two factors is the performance of a model limited?

Answer 1

• architecture-driven limitations:
  limited model capacity
  improper model initialization
  appropriateness of architecture
  (inductive biases)
• data-driven limitations:
  limited amount of data
  data quality

Question 2

What is meant by “data quality”?

Answer 2

1) appropriateness
(eg highly pixelated image would not be good for image classification task)

2) cleanliness
(how accurate was the labeling done? outliers in the dataset?)

3) generalizability
(are there domain shifts? greyscale training images are useless to train RGB models)

Question 3

How can we improve the data quality?

Answer 3

• only use appropriate data
• clean data
• carefully check data for domain shifts

Question 4

Why do we need data augmentation?

Answer 4

to increase the size of training dataset synthetically

Question 5

What kinds of data augmentation are there?

Answer 5

• original
• horizontal flip
• vertical flip
• contrast variations
• image blocking

–> can also be combined

Question 6

How can models be pre-trained?

Answer 6

Through transfer learning

–> initialize model parameters with those from a model of the same architecture
that was previously trained on similar data

Question 7

How can the capacity of the model be improved?

Answer 7

• deeper models have more layers than others
• wider models have more neurons in a single layer

Question 8

What are issues when training large networks?

Answer 8

backpropagation gets more complicated for a large number of network layers:

• gradients can vanish (eg with sigmoid function, large positive numbers go to zero)
• gradients can explode

Question 9

How can we avoid vanishing gradients?

Answer 9

batch normalization (BatchNorm) on every layer

take outputs and normalize them, scale before going through the activation function

Question 10

How do we get rid of exploding gradients?

Answer 10

residual connections

only learn the residuals that typically have less extreme gradients –> learn the differences/delta gradients between expected outputs and the output of the layer

Question 11

What is ResNets?

Answer 11

takes advantage of residual connections as well as BatchNorm

–> are very deep! up to 101 layers

Question 12

How do most supervised tasks work?

Answer 12

discriminative (discriminate between different choices)

Question 13

How is the U-Net build?

Answer 13

encoder-decoder = autoencoder architectures

Question 14

What is the Code (= bottleneck) between encoder and decoder layers?

Answer 14

goal: a meaningful representation of the data

Question 15

What is one way to perform representation learning?

Answer 15

autoencoder

Question 16

What are autoencoders used for?

Answer 16

• representation learning
• data denoising
• anomaly detection

Question 17

What can you do with decoders if they are trained successfully?

Answer 17

use them to generate data from noise

–> standalone decoders can be called generators

Question 18

What are adversarial attacks?

Answer 18

stack a barely visible rgb noise image on top of another image and the model should not be confused by this

Question 19

What does GAN stand for?

Answer 19

generative adversarial network

Question 20

What is the general idea behind a GAN?

Answer 20

have generator G create fake samples and try to trick discriminator D into thinking they are real samples

–> two-player minimax game

Question 21

How do GANs work? (steps)

Answer 21

1) D tries to maximize objective function (succeeds in identifying real samples) - pursues classification task

2) G tries to minimize objective function (succeeds i generating seemingly real samples)

Training: iterate between training D and G (with backdrop) until D says 50% chance of being real or fake

Question 22

How do diffusion models work?

Answer 22

the generator (now acting as an encoder) is trained to make sense of increasingly noisy data

–> turn highly noisy latent representations into realistic images

the latent representation is created by a large language model

Question 23

What is meant by the concept “attention” for CNNs?

Answer 23

which parts of the input data are important?

Question 24

What does attention in NLP enable?

Answer 24

enable each element of the input sequence to attend to any element of the output sequence

–> transformer models implement this attention mechanism

Question 25

What are some of the most important learning paradigms?(5)

Answer 25

• supervised/unsupervised learning
• transfer learning
• semi-supervised/weakly supervised learning
• self-supervised learning
• continual learning

Question 26

What is transfer learning?

Answer 26

like supervised learning, but takes learned abilities from earlier tasks to the next

Question 27

What is semi-supervised learning?

Answer 27

combines a supervised learning process (with a small labeled dataset) with unsupervised methods

eg use clustering to label more data before training

Question 28

What is weakly supervised learning?

Answer 28

related to semi-supervised learning in that it learns to train based on weak labels eg noisy labels (add confidence score to a label)

Question 29

What is self-supervised learning?

Answer 29

getting data is becoming less & less expensive, but labelling it is still very expensive

–> goal is to learn basic representation that can be easily transferred to a given task

instead of learning a trivial task, the model is trained to decide whether two samples are identical or not

typically relies heavily on data augmentations

–> very efficient and time/money-saving to pretrain models!

Question 30

What is continual learning?

Answer 30

some models have to mitigate domain shifts in the data

–> one risk is catastrophic forgetting
can be minimized by experience replays!