Introduction

Question 1

What is a perceptron?

Answer 1

Perceptron: network of threshold nodes for pattern classification.

Perceptron learning rule – first learning
algorithm

Question 2

State 3 differences between human brain and Von Neumann machine

Answer 2

Von Neumann:
One or a few high speed (ns) processors with considerable computing power;
One or a few shared high speed buses for communication;
Sequential memory access by address

Human Brain:
Large number (10^11) of low speed processors (ms) with limited computing power;
Large number (10^15) of low speed connections
Content addressable recall (CAM)

Question 3

How is human brain adaptive compared to a von Neumann machine?

Answer 3

Adaptation by changing the connectivity (topology & the thickness of connections)

Question 4

What is Hebbian rule of learning?

Answer 4

Hebbian rule of learning: increase the connection strength between neurons i and j whenever both i and j are activated increase the connection strength between nodes i and j whenever both nodes are simultaneously ON or OFF.

Question 5

Whose work was the origin of automata theory.

Answer 5

Pitts & McCulloch (1943)

Question 6

Which was the first mathematical model for biological neurons?

Answer 6

Pitts & McCulloch (1943)

Question 7

Why did the US government stop funding ANN research? State 3 reasons.

Answer 7

Single layer perceptron cannot represent (learn) simple functions such as XOR.

Multi-layer of non-linear units may have greater power but there was no learning algorithm for such nets.

Scaling problem: connection weights may grow infinitely

Question 8

What caused a renewed enthusiasm in ANN research in the 90’s?

Answer 8

– New techniques
• Backpropagation learning for multi-layer feed forward
nets (with non-linear, differentiable node functions)
• Physics inspired models (Hopfield net, Boltzmann
machine, etc.)
• Unsupervised learning (LVQ nets, Kohonen nets)

– Impressive applications (character recognition, speech
recognition, text-to-speech transformation, process control,
associative memory, etc.)

Question 9

What does generalisation in the context of ML mean?

Answer 9

we want the network to perform well on data that was not used during the training process!

Question 10

In the context of supervised learning, what does training mean?

Answer 10

a process of tweaking the parameters to minimize the errors of predictions (outputs should be close to targets)

Question 11

What does back propagation mean?

Answer 11

The backpropagation algorithm searches for weight values that minimize the total error of the network

Question 12

What are the two steps of back propagation?

Answer 12

– Forward pass: in this step the network is activated on one example
and the error of each neuron of the output layer is computed.

– Backward pass: in this step the network error is used for updating the
weights. Starting at the output layer, the error is propagated backwards
through the network, layer by layer, with help of the generalized delta
rule. Finally, all weights are updated.

Question 13

What is a downside of back propagation?

Answer 13

No guarantees of convergence, especially when learning rate is too large or too small. In case of convergence, it is the local minima or global minima.

Question 14

How do you solve the convergence problem of back propagation in practice?

Answer 14

try several starting configurations and learning rates

Question 15

Give three examples of MLP and describe in a line what they do

Answer 15

NetTalk: a network that reads aloud texts

ALVINN: a Neural network that drives a car

Falcon: a real-time system for detecting fraud with credit card transactions

Question 16

Briefly describe the revolution of NN?

Answer 16

In the 1940s, first mathematical model of biological neuron by Pitts & McMullcoch

1949 Hebbian rule

1950’s and 60’s single layer perceptron

1980’s and 90’s multi layer perceptron like NETtalk, ALVINN and Falcon

2006
– Restricted Boltzmann Machine as a building block for DNN
– Contrastive Divergence algorithm
– Combine supervised with unsupervised learning
– Deep Belief Networks, Stacked Auto-Encoders

2010 : AlexNet (ImageNet Challenge)

2013: DeepMind “solves” the Atari 2600 games

2016-: AlphaGo, AlphaGoZero, AlphaZero (Go, Chess, …?)

Question 17

What is the Perceptron convergence theorem?

Answer 17

Perceptron convergence theorem: “everything that can be represented by a perceptron can be learned”

Question 18

What are the enabling factors of the huge success of Neural networks?

Answer 18

• Availability of “Big Data”: the more data we have the
better we can train a network

• Powerful Hardware (GPU’s): the speedup of the training
the process by 100-1000 times reduces the training time
from years to hours

• New algorithms and architectures: leaving the MLP
standard behind …

Question 19

What are key deep learning architectures?

Answer 19

• Convolutional Networks: when adding layers enforce hidden nodes to learn “local features” - that reduces the number of parameters
• Recurrent Networks: networks for modeling sequential data
• Autoencoders: networks that learn intrinsic features of the data
• RBMs, VAEs, GANs: generative models of the data

Question 20

Give an example of a CNN application with very high accuracy

Answer 20

MNIST : 99.67%

Question 21

What is MNIST? Describe the architecture and features of the algorithm

Answer 21

training:
60.000 images

testing:
10.000 images

each image:
32x32 pixels

accuracy: 99.7%

Question 22

Which were the most misclassified digits by MNIST?

Answer 22

There were 33 misclassified digits, most confusion being between 7 and 9 and 3 and 5.