Lecture 6:Deep Learning, CNNs

Question 1

What are the 3 major breakthroughs in deep learning?

Answer 1

1. Speech Recognition & machine translation(2010+)

2.Image Recognition & computer vision(2012+)

3.Natural language processing (2014+)

Question 2

How to compute input to hidden?

Answer 2

1. compute net activation net = x*W + b
   x–>inputs
   W–>weights
   b–>bias weights
   2.compute activation function : h=S(neth)

Question 3

How to compute hidden to output?

Answer 3

o=S(h*W + b)

Question 4

What are the three initial drawbacks?

Answer 4

1.Standard back propagation with sigmoid activation does not scale well with multiple layers
2.Overfitting
3.Multilayered ANNs need lots of labeled data

Question 5

What are the two types of problems when multiplying the gradients many times for each layer?

Answer 5

1. Vanishing gradient problem
   2.Exploding gradient problem

Question 6

What does the vanishing gradient problem consist of(4)?

Answer 6

-gradients shrink exponentially with nb of layers–>weight updates get smaller–>weights of early layers change very slowly –> learning very slow

Question 7

What does the exploding gradient problem consist of(4)?

Answer 7

-multiplying gradients makes them grow exponentially–>weight updates get larger and larger–>weights become so large as to overflow and result in NaN values

Question 8

What are the two solutions to initial drawback #1?

Answer 8

1.Use other activation functions
2.Do gradient clipping: set bounds on the gradients

Question 9

What is overfitting(second initial drawback)?

Answer 9

-Large network–> lots of parameters–>increased capacity to learn by heart

Question 10

What are the 2 solutions to overfitting?

Answer 10

1.Regularization
2.Dropout

Question 11

What does regularization consist of?

Answer 11

-modify the error function that we minimize to penalize large weights

Question 12

What does dropout consist of(2)?

Answer 12

-keep a neutron active with some probability p or setting it to 0 otherwise
-prevents the network from becoming too dependent on any one neuronal

Question 13

What is the problem with the third initial drawback?

Answer 13

Most data is not labeled

Question 14

What is the solution to the third initial drawback?

Answer 14

Pre-train the network with features found automatically using unsupervised data –> automatic feature learning

Question 15

What is Classic ML?

Answer 15

-Manual extraction of features

Question 16

What does classic ML require?

Answer 16

-Labeled data and hand-crafted features

Question 17

What are 3 cons of classic ML?

Answer 17

-Needs expert knowledge
-Time-consuming and expensive
-Dos not generalize to other domains

Question 18

What does automatic feature learning consist of?

Answer 18

Each layer learns more abstract features that are then combined/composed into higher-level features automatically

Question 19

What are 3 pros of automatic feature learning?

Answer 19

-We feed the network the raw data
-The features are learned by the network
-Features learned can be re-used in similar tasks

Question 20

What are 5 advantages of unsupervised feature learning?

Answer 20

-more unlabeled data available than labeled data
-Humans learn first from unlabeled examples
-less risk of over-fitting
-no need for manual feature engineering
-features are organized into multiple layers : each level creates new features from combinations of features from level below + more abstract than the ones below (hierarchy of features)

Question 21

What are the 2 steps of the general architectures of a deep network?

Answer 21

1.Unsupervised pre-training of neural network using unlabeled data eg.autoencoder
2.Supervised training with labeled data using features learned from above with a standard classifier eg.ANN

Question 22

What are 2 ways to learn a representation of the data(1st step)?

Answer 22

-Deep Belief Networks (mid 2000s)
-Autoencoders(2006)

Question 23

What is a CNN?

Answer 23

Convolutional Neural Network

Question 24

What does the convolutional layer consist of?

Answer 24

-Uses a filter/kernel that convolves on the image
-The filter is a small weight matrix to learn

Question 25

What is the objective of the convolutional layer?

Answer 25

The network learns the values of the filter(s) that activate when they see some visual feature that is useful to identify the object(final classification)

Question 26

What are the 2 convolution hyper-parameters?

Answer 26

1.Stride
2.Padding

Question 27

What is stride?

Answer 27

How many steps do you move the filter every time

Question 28

What is padding?

Answer 28

Filter should pick up high values surrounded by low values

Question 29

What are pooling layers used for(2)?

Answer 29

-to reduce the size of the activation maps
-so that we reduce the nb of parameters of the network and avoid overfitting

Question 30

What is max pooling?

Answer 30

Similar to the convolution step, but instead of performing a matrix multiplication, max pooling takes the maximum value within the window.

Question 31

What is average pooling?

Answer 31

Taking the average value over an input window for each channel of the input.

Question 32

What is the architecture of a CNN?

Answer 32

1.Stack:
-convolutional layers
-pooling layers
2.Finish off with a fully connected layer at the end for final classification

Question 33

What are 2 examples of successful CNN networks?

Answer 33

LeNet:
-first successful applications of CNNs
-1990s
-used to read zip codes

AlexNet:
-first work that made CNNs popular for computer vision

Question 34

History of AI…

Answer 34

Artificial Intelligence(1950s- 1980s)–> rules written by experts:(

Machine Learning(1980s-2010s)–>rules learns via the data BUT features identified by experts

Deep Learning(2010s-)–>rules AND features learned from the data