Lecture 8: Deep Neural Networks

Question 1

What is a neural network?

Answer 1

A neural network is a set of interconnected stimuli. It is able to learn new behaviours.

Question 2

Name 3 milestones in the history of NNs.

Answer 2

The electronic brain was the first advancement. It was not able to learn.

Hinton produced the multi-layered perceptron, which allowed for backpropagation.

Hinton then produced a deep neural network, which allowed pre-trained models.

Question 3

What are the 3 neuron activation functions? Is there another alternative?

Answer 3

Sigmoid
Tanh
ReLU

An alternative would be to apply a layer function like SoftMax.

Question 4

Give two examples of Neural Network topologies.

Answer 4

The first is perceptron, which has 2 layers: input and ouput.

The second is multi-layer perceptron, which is the same as perceptron but has an additional hidden layer.

Question 5

What is meant by the term “training parameters”?

Answer 5

Finding/changing weights

Question 6

Give three learning algorithms that are used in neural networks.

Answer 6

Batch gradient descent, which calculated the gradient at the end of each batch.

Stochastic gradient descent, which calculates gradient at each iteration/stimulus.

Mini batch gradient descent uses mini-batch, which is a subset of the total dataset.

Question 7

What is the difference between the training dataset and test dataset?

Answer 7

Training dataset refers to the example stimuli used, but test dataset is the test stimuli used to assess model performance.

Question 8

What is overfitting and underfitting?

Answer 8

Overfitting is when the model does not generalise well, i.e. when the test accuracy error is high.

Underfitting is when the model generalises almost too well, i.e. when the test accuracy error is almost zero.

Question 9

What are examples of optimisation methods for neural networks?

Answer 9

Adam
SGD
RMSProp

Question 10

How can overfitting be limited?

Answer 10

By tuning the regularisation strength and dropout probability

Question 11

Give an example of a benchmark dataset for neural networks.

Answer 11

MNIST consists of handwritten digits (10 categories). It has 60,000 training set and 10,000 test set.

CIFAR-10 consists of colour images (10 classes). It has 50,000 training images, 10000 test images.

CIFAR-100 has 100 classes with 600 each and 20 superclasses.

ImageNet is the most important and complex dataset.

Question 12

What is a convolutional neural network?

Answer 12

It is a neural network that converts human-defined features to learned features, called filters.

Question 13

What was the first convolutional network called?

Answer 13

LeNet

Question 14

How are convolutional feature maps used?

Answer 14

It is a sequential implementation of feature maps, where the window is slided over each feature map.

Question 15

How can the spatial resolution of feature maps be reduced?

Answer 15

By pooling/sub-sampling.

Question 16

Name the following two CNN topologies:

1. input -> convolution -> fully connected -> softmax
2. input -> (convolution -> pooling) -> fully connected -> softmax

Answer 16

1. Minimal topology
2. Convolution-pooling module

Question 17

What are the three types of challenges for deep learning in robotic vision? Give two examples of each.

Answer 17

Learning challenges (uncertainty estimation, identifying unknowns, incremental learning, class-incremental learning, active learning)

Embodiment challenges (temporal embodiment, spatial embodiment, active vision, manipulation for perception)

Reasoning challenges (object + scene semantics, object + scene geometry, semantics + geometry)