Sequence Models

Question 1

What is a one-line summary of Sequential Learning?

Answer 1

One data item is dependent on those that come before or after it, not independently and identically distributed

Question 2

What are some common applications of Sequence Learning?

Answer 2

Speech/Voice Recognition
Weather forecasting
Language translation
DNA Sequence Analysis

Question 3

What are the four types of application scenarios in regards to Recurrent Neural Networks?

Answer 3

One-to-one
One-to-many
Many-to-one
Many-to-many

Question 4

What is a one-line summary of One-to-one in the context of Recurrent Neural Networks?

Answer 4

Classical feed forward neural network with one input and one output e.g. image classification

Question 5

What is a one-line summary of One-to-many in the context of Recurrent Neural Networks?

Answer 5

Input image and output words with variable length e.g. Image Captioning

Question 6

What are some common applications of Many-to-one in the context of Recurrent Neural Networks?

Answer 6

Sentiment Classification
Share Price Predictions

Question 7

What are some common applications of Many-to-many in the context of Recurrent Neural Networks?

Answer 7

Language Translation - Input and output are with variable length

Video Clip Classification - Input and output have the same length

Question 8

What is the equation for a Basic Recurrent Neural Network unit?

Answer 8

Current state = activation function * (Previous state and the input vector at present time step)

Question 9

What challenge do Recurrent Neural Networks face?

Answer 9

It suffers from the vanishing/exploding gradient problem

Question 10

What are Long Short Term Memory Networks?

Answer 10

They are altered variations of Recurrent Neural Networks, which are specifically designed to capture long term dependencies.

Question 11

What are Long Short Term Memory networks better at doing compared to Recurrent Neural Networks?

Answer 11

They are better at back propagating the gradient much more efficiently than the standard Recurrent Neural Network

Question 12

What does a Long Short Term Memory consist of in regards to its two main components?

Answer 12

Long term memory cell states, which are non-learnable
Short term memory hidden states, which are learnable.

Question 13

What connects the two main blocks of a Long Short Term Memory Network, and determines whether information passes through to either side?

Answer 13

Multiple sigmoid functions act as gates that switch on and off the information passing areas.

Question 14

What three types of gates exist within the Long Short Term Memory Network Unit?

Answer 14

Forget Gate
Input Gate
Output Gate

Question 15

What does the Forget Gate determine in LSTMs?

Answer 15

The forget gate determines how much long term memory is retained (% amount)

Question 16

What does the Input Gate determine in LSTMs?

Answer 16

The Input Gate determines how much of the short-term information should be contributed to the long-term memory (% amount)

Question 17

What does the Output Gate determine in LSTMs?

Answer 17

The Output Gate determines how much of the new long-term memory should be contributed to the current output. (% amount)

Question 18

What is the challenge in regards to LSTMs?

Answer 18

LSTMs are able to capture long term memory, but not for long enough for it to be useful

Question 19

What is a Transformer especially good at?

Answer 19

Transformers are able to capture dependencies between each input word and between all input words to outputs in a sentence

Question 20

What is the overall architecture of a Transformer?

Answer 20

It is modelled as an Encoder-Decoder architecture:
Encoder - The output is a continuous vector representation of the inputs
Decoder - Takes the continuous vector from the Encoder and generates an output one by one

Question 21

What are some common applications of Transformers?

Answer 21

Language Translation
Chatbot

Question 22

What is the overall pipeline for a Transformer?

Answer 22

Starts with two chains:
- Encoder self-attention -> Feed-forward network
- Decoder self-attention

These pipelines combine into a single pipeline, which is as follows:
- Decoder-Encoder Attention -> Feed-Forward Network

Question 23

What are the details surrounding the Input Embedding step in Transformers?

Answer 23

Each word is mapped to a vector
We then add a positional encoding to the vector using sine/cosine functions

Question 24

What are the details surrounding the Encoder Training area of the Transformer?

Answer 24

The multi-head attention calculates the relationship between all pairwise inputs
The Feed-Forward network maps the attention vectors so that something can be fed to the Transformer decoder.

Question 25

What are the details surrounding the Multi-Head Attention area of the Transformer?

Answer 25

Input - Each word has a Query, Key and Value assigned to it
Scaled Dot-Product Attention is then fed into the Scale, which is fed into the Softmax function.

For each input, compute multiple attention vectors and use the weighted average as the final attention vector for each word.

Output the vector as the Dot Product between attention weight and value. Multiple words can be processed in parallel.

Question 26

What are the details surrounding the Decoder Training area of the Transformer?

Answer 26

During training, the target sentence is input to the Masked Multi-Head Attention, which masks out relationships to future words.

Then another Multi-Head Attention learns the interactions between input words and target words

The output is one-hot encoding, such as a 1000-word vector

Question 27

What does Model Inference mean in regards to the Transformer?

Answer 27

To estimate the second word and onwards, the model uses the whole input sentence and all previously generated target words to infer the result.