Recurrent Neural Networks

Question 1

Was sind Recurrent Networks?

Answer 1

Question 2

Für welche Szenarien sind Recurrent Networks optimal?

Answer 2

Wenn vergangene Variablen gebraucht/behalten werden müssen.
Vor allem bei Problemen mit einem zeitlichen Faktor.

Question 3

Beschreib ein Simple Recurrent Network

Answer 3

Question 4

Was ist ein Kontext Layer?

Answer 4

kopiert Output eines hidden layers von t-1 und gibt ihn als Input in das Netzwerk

Question 5

Welche Alternative gibt es zu SRN, wenn man ein feedforward Netzwerk benutzen möchte?

Answer 5

Time Delay Neural Network

Question 6

Was sind Vorteile von Time Delay Neural Network?

Answer 6

• Feedforward architecture
  ▪ easy parallelization
  ▪ efficient learning
• No memory fading over
  the length of the context

Question 7

Was sind Nachteile von Time Delay Neural Network?

Answer 7

Drawbacks of TDNNs:
- Length of context is fixed
- For large contexts:
▪ large input dimensionality
▪ many weights/parameters

Question 8

Was ist ein Echo State Network?

Answer 8

• Variation von SRN
• verschiedene Historien eines SRN führen zu verschiede Aktivierungen, wenn recurrent/input weights zufällig sind
• also kann man auch nur die output weights trainieren

Question 9

Was ist die echo state property?

Answer 9

the largest
eigenvalue of W^rec must not be much larger than 1

Question 10

Was ist der Vorteil an ESN?

Answer 10

Das Optimierungsproblem kann durch Algebera gelöst werden, ohne iteratives trainieren

Question 11

Was ist der Nachteil an SRN?

Answer 11

Durch ihre einfache Struktur können die SRNs nicht hierarchisch angeordnet werden, es entwickelt sich also keine komplexere Struktur, da nur auf einer Ebene gelernt wird (Output Layer). Sie sind gut für basic classifier

Question 12

Was sind Extreme Learning Machines?

Answer 12

Question 13

Was ist der Vorteil von Extreme Learning Machines?

Answer 13

Schnelles Training

Question 14

Was ist der Nachteil von Extreme Learning Machines

Answer 14

Difficult / data-dependent setting of parameters, e.g. scaling
of weights and their sparseness

Question 15

Was ist ein Hopfield Network

Answer 15

• idea of Associative Memory

Question 16

Wie wird bei Hopfield Networks gelernt?

Answer 16

Gewichtsaktualisierung: Die Verbindungen (synaptische Gewichte) zwischen den Neuronen werden gemäß der Hebb’schen Regel aktualisiert.
w_ij = 1/n * Summe (über p Patterns) von x_i^p * x_j^p

Question 17

Hopfield Networks

How many patterns can be stored in memory?

Answer 17

Question 18

Hopfield Network

Was ist die Energy of the Network’s Activations?

Answer 18

Question 19

Was ist die Idee dahinter, die Hebb Rule zu erweitern, sodass 2N Patterns gemerkt werden können?

Answer 19

Question 20

Was sind Limitationen von Hopfield Networks?

Answer 20

• erlauben nur wenige Patterns
• erlauben nur statische Patterns, keine dynamischen

Question 21

Erkläre Conway’s Game of Life

Answer 21

Question 22

Which of the statements about Recurrent Neural Networks (RNNs) are true?
1. Considering an unlimited number of timesteps, a simple recurrent network can be translated into a feed-forward MLP.
2. In an echo-state network (ESN), only the weights between the hidden and the output layer are trained, thus the error function can be differentiated to a linear system.
3. Simple recurrent networks retain absolute knowledge of each previous input in the sequence.
4. RNNs can be used for sequence classification, but not for sequence prediction.

Answer 22

2

Question 23

Complete the following statement: The predictive performance of decision trees can be improved using…?
1. Random Trees
2. Bagging
3. Boosting

Answer 23

alle drei

Question 24

Which of the following ensemble learning techniques is more likely to yield better predictive performance than the other?
1. Bagging better than Boosting
2. Boosting better than Bagging
3. Both always show the same performance

Answer 24

2

Question 25

Which of the following statements about Dropout in neural networks are true?
1. Dropout can prevent a network from underfitting the training data.
2. The choice of neurons to be deactivated is made based on their prior reliability.
3. Dropout applied on the input level can be viewed as bagging of input features.
4. Dropout strongly resembles ensemble learning when using a network repeatedly with alternating deactivated neural units.

Answer 25

3 und 4

Question 26

Complete the following statement: During Bagging or Bootstrap Aggregation, …?
1. …subsets of the original training data are sampled with replacement.
2. …all models in the ensemble operate on the same resampled subsets of training data, but with different random initialization of model parameters.
3. …an input sample is classified by taking the vote of the most reliable predictor as final output label.

Answer 26

1