Unit 3 - Recurrent Neural Networks

Question 1

What is recurrent neural network?

Answer 1

Variation to feed forward network

Output from previous step are fed as input to the current step

Talk about traditional neural networks

Less complexity due to using same parameters for each input

Uses a hidden state that stores all the information about the previous steps or a sequence

Question 2

Example of RNN:

Answer 2

To predict the next word in the sentence, you need to know the previous words of the sentence

Question 3

Types of RNN:

Answer 3

One to one
One to many
Many to one
Many to many

Question 4

What is one to one RNN?

Answer 4

Single input and single output

Fixed input and output sizes so works like a traditional neural network

Example: image classification

Question 5

What is one to many RNN?

Answer 5

Gives multiple outputs for a single input

Fixed input size and gives sequence of data outputs

Example: music generation, image captioning

Question 6

What is many to one RNN?

Answer 6

Single output is required for multiple input units or sequence of them

Example: sentiment analysis

Question 7

What is many to many RNN with equal unit size?

Answer 7

Used to generate sequence of output data from a sequence of input units

In this case, both input output units are same in number

Example, name-entity recognition

Question 8

What is many to many RNN with unequal unit size?

Answer 8

Which net sequence of output data from sequence of input data

This case inputs, an outputs have different number of units

Example: machine translation

Question 9

What is LSTM?

Answer 9

Variation of RNN

Critical components - memory cell and gates

Contents of memory formulated by forget and input gates

If both gates closed, no change in memory contents

No vanishing gradient problem

Question 10

Three neural networks used in LSTM:

Answer 10

1. Forget network - info not needed is removed
2. Remember network - adds the info from input if needed to the state
3. Select network - presents the version of internal state as the output
4. State - stores the formulated input

Question 11

LSTM Gates:

Answer 11

1. Forget gate - input combined with previous output to generate a fraction 0 and 1 and then multiplies with previous state.
2. Input gate - which new info enters state of LSTM. Output of input gate(fraction 0 to 1) multiplied with tan h block and added to previous state.
3. Output gate - exposes the current memory state

Question 12

What is encoder decoder architecture?

Answer 12

Maps input domain to output domain in two stage network:

1. Encoder - compresses input into latent space representation.
2. Decoder - reconstructs from the latent representation to predict the output

Question 13

What is sequence to sequence model?

Answer 13

Map fixed length input to fixed length output. Length of input and output may differ.

English to Chinese example

Question 14

How Seq to Seq model works?

Answer 14

Uses encoder-decoder architecture

Consists of 3 parts:
1. Encoder - has several units of LSTM or GRU cells where each accepts an element of input sequence. Collects info from that element and forwards it .

2. Intermediate (encoder) vector - final hidden state from the encoder part of the model. Encapsulates info of all the input elements.
3. Decoder - like encoder, has several recurrent units. Predicts output from each unit at each time step. Takes hidden state from previous step and produces output

Question 15

Applications of Seq to Seq model:

Answer 15

1. Machine translation
2. Question and answer
3. Video captioning
4. Speech recognition

Question 16

What is a recursive neural network?

Answer 16

Ability to model hierarchical structure in the training datasets

Applying same set of weights recursively over tree-like structures

Generalise the RNN from chain-like struct to tree-like struct.

Question 17

Types of Recursive Neural Network:

Answer 17

1. Recursive autoencoder - learns to reconstruct input. In NLP, reconstructs the contexts. In supervised learning, learns likelihood of certain labels
2. Recursive Neural Tensor Network -