Week 1: principles of deep learning in artificial networks

Question 1

What is a deep network?

Answer 1

A learning network that transforms or extracts features using multiple nonlinear processing units arranged in multiple layers with hierarchical organisation and different levels of representation and abstraction

Question 2

How do complex outcomes emerge in deep networks?

Answer 2

From interactions between many simple steps

Question 3

What is a representation in a deep network?

Answer 3

The information the computer is given. Each representation is built from an earlier representation that can transfer the features and extract complex features from simpler features

Question 4

What are the 4 operations in a linear-nonlinear layer

Answer 4

Filter/convolve
Threshold/rectification
Pool
Normalise

Question 5

What is the purpose of the filter/convolve operation?

Answer 5

To determine how well each group of nearby pixels matches each of a group of filters

Question 6

How does the filter operation work?

Answer 6

The input is a pixel map

• convolution step looks for a pattern in a group of neighbouring pixels that corresponds to the convolution filter
• if source pixels follow the filter pattern the results is a high value, if input area is all same brightness result will be 0, if source pixels are opposite to the filter the result will be negative
• output is feature maps

Question 7

What is the purpose of the threshold/rectify operation?

Answer 7

Introduce nonlinearity by setting negative activations of units to zero (and maybe set a maximum activation)

Question 8

What is the goal of the ReLU activation function?

Answer 8

To only activate the output feature map if its value reaches a certain threshold

Question 9

What is the purpose of the pooling operation?

Answer 9

Downsample the units to improve computational efficiency

Question 10

How does the pooling operation work?

Answer 10

Typically takes the maximum of a square of 2x2 neighbouring units of the feature map

Question 11

What is the purpose of the normalise operation?

Answer 11

Rescale responses of each feature map to have mean 0 and standard deviation 1 so each feature map contributes similarly to classification

Question 12

Why is normalisation necessary?

Answer 12

The range could be different between feature maps, weighting some more than others

Question 13

What tasks are deep learning useful for?

Answer 13

Useful for achieving tasks that are difficult to describe formally. Tasks that are difficult for computers but intuitive for humans

Question 14

What is the final layer of the network

Answer 14

The final fully-connected layer links pattern of most abstracted, top-level features to required response
-the last feature map is flattened into a line of independent units where each is connected to all the others

Question 15

What does the softmax function do?

Answer 15

Determines the probability of the desired response

Question 16

How does the softmax function work?

Answer 16

Each input image has a score reflecting the match bewegen the top layer’s activation pattern. The score is converted to a probability that this input image falls into each category

Question 17

How are convolution filters and weights related?

Answer 17

The convolution filters form the weights of connections between the nodes in a neural network

Question 18

How does a network learn the weights of the connections?

Answer 18

Using back-propagation error

Question 19

What does back-propagation error do?

Answer 19

Adjusts the filter structure - the link between layers

Question 20

What is the basis for back-propagation error?

Answer 20

The match or conflict of expected and actual outputs

Question 21

Why do filters generally have a single set of weights for all positions in the feature map?

Answer 21

• if a feature is useful to compute at one position, it is probably useful at another position
• the filter values are weights that need to be learned, it is computationally demanding if the set is too large
• the convolution operation is a fast matrix function, if filters are not fixed the convolution operation cannot be used

Question 22

What is a definition of machine learning?

Answer 22

A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E