09 - Recurrent Neural Networks Flashcards
What is a RNN (roughly explained?)
Recurrent neural network.
RNNs are used to process sequential data x^t={x^1,…,x^τ} , where t is the time/dimension step index
- Even though other nets without sequence based spcialization could process this type of data input as well, RNNs can scale much longer sequences than other nets, just like CNNs can process larges images.
- They can also process sequences of variable length (2d data as well)
RNN Parameter Sharing.
Diff between multilayer nets & RNNs: parameter sharing across different parts of the model
- Enabeling the model to be applied to inputs with different lengths and generalizing across them
- enables extracting information that can occur at different positions in the input sequence
- example: “I went to Nepal in 2009” vs “In 2009, I went to Nepal”
- A traditional fully-connected feedforward net that processes fixed length sentences would have needed to learn all possibilities.
The way it works:
- each output member is a function of the previous output member & produced using the same update rule applied to the previous outputs
Unfolding Computational graphs
classical dynamical system: s(t) = f(s(t-1);theta), s - state of the system
- we call this system recurrent, bc the definition of state s at time t refers back to the same definition one time t-1
A graph can be unfolded for a finite nr of steps tau, by applying the definition tau-1 times. Eg the equation above for tau=3: s(3)=f(s(2);theta)=f(f(s(1);theta);theta)
Example of system driven by external signal: h(t) = f(h(t-1);x(t),\theta)
What are the advantages of unfolding computational graphs?
The unfolding process allows to factorize g into repeated application of function f. This gives following advantages:
- The input size for the model is always the same, regardles of the sequence length, bc it is specified in the terms of transition from state to state, not in terms of a variable-length history of states
- The same transition function f with the same parameters can be applied at every step
The model is now generalized to sequence length not given in the training set, which allows for far fewer training examples
What RNN designs/output modes are there?
Sequence to Sequence (A prototypical RNN)
Sequence to Single Output
Single Input to Sequence
Explain the Sequence to Sequence RNN Design
Both the input x and the output o are sequences. Such an output could be a classification vector with c elements for c classes. For example if we have a dictionary with 100 words, we get a 100dim vector. There are recurrent connections between hidden units.
Draw the RNN with following:
x->h->o->L->Y
U - Input to hidden (like feedforward FCNs or CNNs)
W - Hidden to Hidden, defining a transition to the next state
V - Hidden to Output, like regular NNs, transforms low/high dim codes to correct output type (eg classification or regression)
NOTE: loss and y are only used for training
For each time step from t= 1 to t=τ, following update equations are used:
- Pre-nonlinear activation: $a^t = Wh^{t-1}+Ux^t+b$ (so like the other nets just with one extra term (W?)
- Non-Linearity (get state): $h^t = tanh(a^t)$
- Transition to output: $o^t = Vh^t+c$ (c is an optional extra bias, normally you wanna use it, o are our logits)
- E.g if building classifier: $\hat{y}^t=softmax(o^t)$
NOTE: the update equations are
- differentiable (backprop)
- applied at every time step
Explain the Sequence to Single Output RNN Design
Recurrent connections between hidden units, that read the entire sequence and then produce a single output from that. The output (a tensor) can be seen as the summary of the input sequence.
Draw the RNN with following:
x->h->o->L->Y
U - Input to hidden (like feedforward FCNs or CNNs)
W - Hidden to Hidden, defining a transition to the next state
V - Hidden to Output, like regular NNs, transforms low/high dim codes to correct output type (eg classification or regression)
Explain the Single Input to Sequence RNN Design
We are not getting an input at each time step, but only one input (eg at the start). It can be used at each step like on the graph but mostly it is just applied at the start
True generation of data
The recurrence is both the feedback connection from the output to the next steps hidden layer and from the pas hidden layer. In training it is the y not o that is gven to the next hidden layer.
Straw hat example
The tokens are often <bos> (beginning of sentence) or <eos> (end of sentence)</eos></bos>
RNN - Backpropagation Through Time
To compute the gradient in a RNN, the generalized back-propagation algorithm (section 6.5.6) is simply applied to the unrolled computational graph.
So, for each node N, the gradient $∇_NL$ is computed recursively, based on the gradient computed for nodes that follow in the graph
- Like usually, we start with the delta at the output (following is the componentwise notation from the book, Anders does not use that). It’s the simple numerical diff between the output and target ∇_{o(t)}L_i=y_hat(t) -y(t)
- Now go one step back to get the hiddens (just back as in calculation steps, not time step) ∇{h(t)}L= V^T ∇{o(t)}L
- Now we can iterate back through time to back-propagade gradients from t=τ-1 to t=1. So h(t) will descent from o(t) and h(t+1).
Using sequence to sequence design in generative networks.
Most generative networks do not only take the input but also the last output (if guessing the next word, it makes sense to know what was sair before). The connection between the hiddens are still there.
Encoder-Decoder RNNs
A structure in which both input and output can have variable lengths. This is used in LLMs, for example chatGPT is decoder only, BERT is encoder only.
Encoder only nets are often used for eg classifying, while decoder only nets are used for generation of human like content
Bidirectional RNNs
ses forward and backward stream of hidden states.
For example, it could make sense to read sentences backward and forward. This does not always make sense but sometimes it gives a little boost.
Transformers take all directions always (LMMs are a subclass of transformers)
Name strategies to make RNNs deeper
- add more hidden states that connect to each ther over time
- add non-recurrent, regular layers
- adding skip connections to strategy 2 to avoid problems with backpropagation
Why use LSTM? Pros and Cons?
Normal RNNs suffer from long term dependency problems, meaning over time, as more and more information piles up, the less effective at learning new things the nets become
→ LSTM is a solution to that problem and allows a net to remember the things it needs to keep ahold of the context but forget things that no longer matter.
LSTMs are way more computationally heavy, because each gate is coupled to the current input and previous state with their own weights and biases: U,W,U^g,W^g,U^f,W^f,U^o,W^o,b,b^g,b^f,b^o
LSTMs are much better for long sequences.
Normal RNNs are so deep, a long row of vector multiplications, that there is a risk of the values exploding or converging to 0. Especially backprop is hard.
Training is faster and better with LSTMs, so even though we have many more parameters (x4) it is worth it
Explain LSTMs.
The LSTM cell includes three gates:
1. Forget gate: what is no longer important for context and can be forgotten
2. Input gate: what new information should be added or updated
3. Output gate: what should be output
All gates are values between 0 and 1 (following a sigmoid)
- 1 - let everything in
- 0 - full shut down
- For each time step, we put the input and previous state into 4 sublayers
Input:
- current input x and previous state h and are squashed by tanh
- tanh function is standart but any non-linearity can be used
State:
- function of the previous state
- much like momentum →current value is accumulated as a running average of the past history
- BUT the momentum coefficient is dynamic and implemented in the forget gate
