DL-01 - Introduction (+ impl)

Question 1

DL-01a - Introduction

Who was behind Alexnet? (1 + 2)

Answer 1

Alex Krizhevsky in collaboration with Ilya Sutskever and Geoffrey Hinton.

Question 2

DL-01a - Introduction

What’s the formula for the sigmoid function?

Answer 2

(See image)

Question 3

DL-01a - Introduction

What formula is this? (See image)

Answer 3

Sigmoid

Question 4

DL-01a - Introduction

Why is the sigmoid activation function less used these days? (2)

Answer 4

• Vanishing gradient problem
• Non-zero centered output (range 0-1)

Question 5

DL-01a - Introduction

Why is the non-zero centered output of a sigmoid function a problem?

Answer 5

It causes zig-zagging dynamics during optimization.

Question 6

DL-01a - Introduction

What is one cause of vanishing gradients with the sigmoid function?

Answer 6

Saturation behavior at either end of the tails.

Question 7

DL-01a - Introduction

What formula is this? (See image)

Answer 7

Tanh.

Question 8

DL-01a - Introduction

What’s the formula for tanh?

Answer 8

(See image)

Question 9

DL-01a - Introduction

How is tanh related to sigmoid?

Answer 9

Question 10

DL-01a - Introduction

How does tanh and sigmoid compare? (3)

Answer 10

• Tanh solves the “zero centered” problem.
• Both gradients still saturate.
• Tanh is generally preferred over sigmoid.

Question 11

DL-01a - Introduction

What are the pros of the ReLU function? (4)

Answer 11

• No vanishing gradient
• Fast convergence
• Simple implementation
• Better convergence performance than sigmoid

Question 12

DL-01a - Introduction

What are the cons of the ReLU function? (1)

Answer 12

“Dying ReLU” problem with large gradient flows.

Question 13

DL-01a - Introduction

What is parametric/leaky ReLU?

Answer 13

ReLU with slope in negative region. (See image)

• Parametric: Has learnable param.
• Leaky: Pre-defined param.

Question 14

DL-01a - Introduction

What’s the formula for PReLU? Is alpha a param or a hyperparam?

Answer 14

Alpha is a param and is learned during training.

(See image)

Question 15

DL-01a - Introduction

What’s the formula for leaky ReLU? Is alpha a param or a hyperparam?

Answer 15

Alpha is a hyperparam, typically fixed at 0.01.

(See image)

Question 16

DL-01a - Introduction

What activation function is this? (See image)

Answer 16

Softmax

Question 17

DL-01a - Introduction

What’s the formula for softmax?

Answer 17

(See image)

Question 18

DL-01a - Introduction

How is momentum used in gradient descent?

Answer 18

Momentum is used in gradient descent to accelerate convergence by adding a fraction of the previous update to the current update.

Question 19

DL-01a - Introduction

What does momentum help with?

Answer 19

Overcoming local minima and oscillations.

Question 20

DL-01a - Introduction

How are momentum and Nesterov different?

Answer 20

• Momentum speeds up gradient descent
• Nesterov Momentum adds a corrective factor for better approximation.

Question 21

DL-01a - Introduction

What is RMSprop?

Answer 21

RMSprop is an adaptive learning rate optimization algorithm for gradient descent in deep learning models.

Question 22

DL-01a - Introduction

How does RMSprop work?

Answer 22

RMSprop works by adapting the learning rate for each weight based on the magnitudes of their gradients, using a running average of squared gradients.

Question 23

DL-01a - Introduction

How does ADAM work?

Answer 23

The ADAM optimizer works by adaptively adjusting the learning rate for each parameter using the first and second moments of the gradients.