ANN Lecture 1 & 2 - Introduction and Multi Layer Perceptron

Question 1

Marr’s Level of Analysis

Answer 1

Computational Level:
What is the aim of the process?
Which problem is solved?
[Psychology]

Algorithmic Level:
Which algorithm is used?
How is the problem solved?
[Cognitive Science]

Implementational Level:
How is the algorithm realized
or implemented?
[Neuroscience]

Question 2

Explicit Knowledge

Answer 2

Can be formalized and can be implemented in an algorithm, so a computer will probably be better than a human.
e.g.: chess, integrating complex formulas, navigation

Question 3

Implicit Knowledge

Answer 3

Can’t be formalized. It is knowledge that a human has and can use, but cannot access verbally.
e.g.: walking, juggling, all other motor skills, object/speech recognition etc.

Question 4

Perceptron
and
XOR-Problem

Answer 4

Output =
Step-Activation-Function(Weighted input-vector + Bias)

A perceptron seperates a space into two parts since the perceptron is a hyperplane. But XOR is is not linearly seperable, so a perceptron can not learn it. A MLP (Multi-Layer-Perceptron) though is able to solve the XOR-Problem.

Question 5

Multi-Layer-Perceptron

Answer 5

• Feed-forward neural network.
• > Input-Layer -> Hidden Layers -> Output Layer
• Forward step
• Back propagation step

Question 6

Tasks for a MLP

Answer 6

Regression:
Finding the relation ship between two continuous variables.

Classification:
Predicting a label for a sample.

Question 7

One Hot Encoding

Answer 7

Vector-length of one-hot encoded vector is equal to the number of labels. Label can be interpreted as a discrete probability distribution. (Single one, remainder zeros)

Question 8

Output Function for Classification

Answer 8

Softmax Function:
Y_i = (e^d_i)/sum_j(e^d_j)
Takes an Vector as input and turns it into a discrete probability distribution.

Question 9

Loss Function for regression

Answer 9

• Sum squared error (approximate the real valued outputs as good as possible)
  Loss = 1/2 * SUM_i(T_i - Y_i)^2

Question 10

Loss Function for Classification

Answer 10

Cross Entropy (Similarity matters more than the distance between output and target)

Loss = - SUM_i(T_i * log(Y_i))

Question 11

MLP - Forward Step

Answer 11

Drive = Weight Matrix x Input-Vector + Bias-Vector
Activation = Sigmoid-Function(Drive)

Question 12

Activation-Functions

Answer 12

Heavisided Step Function:
(One or Zero)
Logistic Function:
(Soft transition between One and Zero, sigmoid)
TanH-Function:
(Soft transition between One and minus One, sigmoid)
ReLu-Function:
(Zero up to Zero, Linear slope from zero)

Question 13

Supervised Learning

Answer 13

• We have input data with corresponding targets (labels)
• We are trying to find the underlying mapping of targets to the input
• This mapping should explain all our data and generalized to unseen data.