RNN & Attention

Question 1

Neural Nets

What is a Feedforward Neural Net?

Answer 1

n features -> W matrix (dxn) -> non linear func -> d hidden units -> V matrix (Cxd) -> softmax -> C probabilities

Representing text by summing/averaging word embeddings

Question 2

Neural Nets

Word Embeddings in NNs

Answer 2

Similar input words get similar vectors

Question 3

Neural Nets

W Matrix in NNs

Answer 3

Similar output words get similar words in softmax matrix

Question 4

Neural Nets

Hidden States in NNs

Answer 4

Similar contexts get similar hidden states

Question 5

Neural Nets

What problems ARE handled by NNs? (vs count-based LMs)

Answer 5

• Can share strength among similar words and contexts
• Can condition on context with intervening (interconnected) words

Question 6

Neural Nets

What problems ARE NOT handled by NNs?

Answer 6

Cannot handle long-distance dependencies

Question 7

Recurrent Neural Networks

Sequential Data Examples

Answer 7

• Words in sentences
• Characters in words
• Sentences in sample

Question 8

Recurrent Neural Networks

Long Distance Dependencies Examples

Answer 8

• Agreement in number, gender
• Selectional preference (determine meaning of rain/reign using clouds/queen)

Question 9

Recurrent Neural Networks

Recurrent Neural Networks

Answer 9

• Retains information from previous inputs due to the use of hidden states
• Designed to process sequential data, where the order of the data matters (FFNN treats inputs independently)
• At each step, RNN takes in current input with the hidden state from the previous step and updates the hidden state (“remembers”)

Question 10

Recurrent Neural Networks

Unrolling in Time

Answer 10

• RNN updates hidden vector upon each input,
• Unrolling means breaking down “cell” into multiple copies
• Make it easier to see how network processes sequence step-by-step

Question 11

Recurrent Neural Networks

What can RNNs do?

Answer 11

• Sentence classification
• Conditional generation
• Language modeling
• POS tagging

Question 12

Recurrent Neural Networks

Sentence Classification

Answer 12

Read whole sentence and represent it

Question 13

Recurrent Neural Networks

Conditional Generation

Answer 13

Use sentence representation to make prediction

Question 14

Recurrent Neural Networks

Language Modeling

Answer 14

Read context up until a point and represent the context

Question 15

Recurrent Neural Networks

POS Tagging

Answer 15

Use sentence and context representation to determine the POS of a word

Question 16

RNN Training

How to update hidden state?

Answer 16

h_t = tanh( Wx_t + Vh_t + b_h )

Question 17

RNN Training

How to compute output vector?

Answer 17

y_t = tanh( Uh_t + b_y )

Question 18

RNN Training

How to calculate the loss?

Answer 18

• Loss/label is calculated each time the hidden state is updated (at each input)
• Add up all losses to get total loss

Question 19

RNN Training

Unrolling

Answer 19

Unrolled graph (of sum of loss) is a well-formed computation graph that we can use to run backpropagation through time

Question 20

RNN Training

Parameter Tying

Answer 20

• All parameter calculation are tied together because we are summing the loss together
• This is how we are able to determine context, POS, and long-distance dependencies
  *

Question 21

Recurrent Neural Networks

Bi-RNNs

Answer 21

• Runs the RNN in both directions
• Need two different models to run the input through
• Helpful with context as context goes in both directions

Question 22

RNN Long Short-Term Memory

Vanshing Gradient

Answer 22

• Cccurs during backpropagation
• Gradients (used to update the network’s weights) shrink exponentially as they are propagated backward through time
• Makes it hard to determine long-distance dependencies

Question 23

RNN LSTM

Long Short-Term Memory (LSTM)

Answer 23

• Overcomes vanishing gradients and helps to learn long-term dependencies
• Has memory cell and gates that regulate the flow of information
• Allows network to retain or forget information over long sequences

Question 24

Attention

Why do we need attention?

Answer 24

• In normal RNN, as the input sequence becomes longer, the model struggles to compress all relevant information
• Results in poor performance

Question 25

Attention

Attention

Answer 25

• Attention computes weighted combination of all hidden states at every time step of the output sequence
• Allows the model to focus more on relevant parts of the input when generating the output
• Higher attention when query is closer to key (every time query is close to key (feature), increase the attention for that key)

Question 26

Attention

Calculating Attention

Answer 26

1. Use “query” vector (decoder) and “key” vectors (encoder) to calculate weight and normalize using softmax
2. This weight indicate how much attention the model should pay to each input position
3. Combine the attention weights into a vector
4. This creates the context vector, which summarizes the important parts of the input for generating the next output (use to pick next word)

Question 27

Attention

Self-Attention

Answer 27

• Normal attention involves attention between two different sequences
• Self-attention computes attention within a single sequence, each position to attends to all other positions in the sequence
• Each word forms “query” which computes attention over each other word

Question 28

Attention

Multi-Head Self-Attention

Answer 28

• Used in transformer
• Captures different relationships between tokens by performing multiple attention operations (called “heads”) in parallel
• Each “head” focuses on different parts of the input/different dependencies, giving model richer understanding of input