Quiz #5

Question 1

What is ‘attention’?

Answer 1

Weighting or probability distribution over inputs that depends on computational state and inputs.

Question 2

What does attention allow us to do?

Answer 2

It allows information to propagate between “distant” computational nodes while making minimal structural assumptions.

Question 3

What is one of the most popular tools used for building attention mechanisms?

Answer 3

Softmax

Question 4

Softmax is differentiable? (True/False)

Answer 4

True

Question 5

What are the inputs to softmax attention?

Answer 5

A set of vectors {u1, u2, …u_} and a query vector ‘q’. What we want to do is select the most similar vector to q via p = softmax(U.q)

Question 6

What is the difference between softmax applied to the final layer of an MLP and softmax attention?

Answer 6

• When Softmax is applied at the final layer of an MLP:
  
  • q is the last hidden state, {u1, u2, … u_n} is the embedding of the class labels
  • Samples from the distribution correspond to labelings (outputs)
• In Softmax attention:
  
  • q is an internal hidden state, {u1, u2, … u_n} is the embeddings of an “input” (e.g. the previous layer)
  • Samples from the distribution correspond to a summary of {u1, u2, … u_n}

Question 7

What was the biological inspiration for attention mechanisms?

Answer 7

Saccades (basically rapid, discontinous eye movements between salient objects in the visual field).

Question 8

At a conceptual level, what is it that are visual attention mechanisms trying to do?

Answer 8

Given the current state/history of glimpses, where and what scale should we look at next?

Question 9

The representational power of a softmax attention layer (or more generally, any attention layer) decreases as the input size grows? (True/False)

Answer 9

False. It increases. This is because the size of the hidden state grows with the size of the input.

Question 10

How is the similarity of query vector ‘q’ typically computed compared to the set of hidden state vectors {u1, u2, … u_n}?

Answer 10

Cosine similarity (i.e. inner product of q with each of the u vectors}

Question 11

What is the ‘controller state’ in a Softmax Attention layer?

Answer 11

Initially, it’s just the input query vector q0. We use q0 to compute the next controller state (i.e. hidden state) q1, then use q1 to compute q2, …so on.

Question 12

What are transformer models?

Answer 12

Models that are made up of multiple attention layers.

Question 13

What are three important architectural distinctions that result in the superior performance of transformer models compared to previous attention based architectures?

Answer 13

1. Multi-query hidden state propagation (“self-attention”)
2. Multi-head attention
3. Residual connections, LayerNorm

Question 14

What is self-attention?

Answer 14

It uses a controller state (i.e. query/hidden state) for every single input). So the size of the controller state grows with the size of the input, giving it even more representational power than traditional attention networks (remember that the representational power of an attention network grows proportionally to its input size).

Question 15

What is multi-head attention?

Answer 15

Multi-head attention combines multiple attention ‘heads’ being trained in the same way on the same data - but with different weight matrices, thus yielding different values.

Each of the ‘L’ attention heads yields values for each token - these values are then multiplied by trained parameters and added.

(To me this seems kind of similar to the idea in convnets of using multiple “filters” for each convolutional layer so we can learn different feature representations. )

Question 16

What are some of the major reasons why machine translation is difficult?

Answer 16

1. Language is ambigous
2. Language depends on context
3. Languages are very different (e.g. structure, what is implicit vs. explicit, etc.)

Question 17

Translation is often modeled as a conditional language model? (True/False)

Answer 17

True. Typically Prob(tokens | source)

Question 18

The probability of each output token is estimated together based on the source material in a machine translation model? (True/False)

Answer 18

False. They are estimated separately from left-to-right.

Question 19

In a machine translation model, we calculate the probability of each output token estimated separately (left-to-right) based on what two things?

Answer 19

1. Entire input sequence (encoder outputs)
2. All previously predicted tokens (decoder “state”)

Question 20

In the context of machine translation models, the argmax[p(t | s)] is intractable? (True/False)

Answer 20

True.

Question 21

In the context of machine translation models, why is argmax[p(t | s)] impossible to find, and what technique do we use to remedy this?

Answer 21

• The problem: Exponential search space of possible sequences
• Remedy is to use beam search (typical beam size of 4 to 6)

Question 22

What does the beam search algorithm allow us to do?

Answer 22

To search exponential space in linear time.

Question 23

How does the beam search algorithm work for machine translation?

Answer 23

We explore a limited number of hypotheses ‘k’ at a time. At each step, we extend each of ‘k’ elements by one token. Top ‘k’ overall then become the hypotheses for the next step.

Question 24

Each beam elements has a different _____? For transformer/decoder this is _________ input for _______ steps?

Answer 24

1. State
2. Self-Attention
3. Previous steps

Question 25

Total computation scales linearly with beam width? (True/False)

Answer 25

True

Question 26

Computation over the beam is difficult to parallelize on GPUs when using the beam search algorithm? (True/False)

Answer 26

False. It is highly parallelizable over the beam.

Question 27

What are the three reasons inference can be computational expensive for machine translation models?

Answer 27

1. Step-by-step computation (auto-regressive inference)
2. Output projection: theta(len_vocab * output_len * beam_size)
3. Deeper models

Question 28

What are three strategies we can use to overcome some of the inherit inefficiencies associated with machine translation inference?

Answer 28

1. Use smaller vocabularies
2. More efficient computation

Reduce depth / increase parallelism

Question 29

To improve computational efficiency for machine translation models, what is one good reason why it’s reasonable to use smaller vocabularies?

Answer 29

Because while a vocabulary may be huge, for any given input sequence, the likely outputs will generally be constrained to a fairly small set.

IBM alignment models use statistical techniques to model the probability of one word being translated into another. Alternatively, lexical probabilities can be used to predict most likely output tokens for a given input. Using these approaches can achieve up to 60% speedup.

Question 30

What is one way we can overcome the challenge of how to model rare or unseen words?

Answer 30

Model most frequent words as their own token, less frequent words get broken up into their constituent parts. One popular algorithm for doing this is byte-pair encoding.

Question 31

How does byte-pair encoding work?

Answer 31

BPE comes from the idea of compression, where the most frequent adjacent pair is iteratively replaced.

Example:

Consider the string “abcdeababce”

Step 1: Replace most frequent pair “ab” with “X” (and add replacement rule)

“XcdeXXce”

X=”ab”

Step 2: Replace next most frrequent pair (here including the replacement byte)

“YdeXYe”

X=”ab”

Y=”Xc”

Question 32

When using parallel computation to speedup machine translation inference, what is the major bottleneck?

Answer 32

Autoregressive inference time dominated by decoder

Question 33

For machine translation models, average efficiency can be increased by translating multiple inputs at once? (True/False)

Answer 33

True (however, may not be practical for real-time systems)

Question 34

What is the biggest slowdown typically the result of in machine translation models?

Answer 34

It comes from the requirement that for the decoder we need to predict each token in the output sequence one at a time. (i.e. autoregressive)

Question 35

Neural Machine Translation (NMT) systems have an _______ learning curve with respect to the amount of training data, resulting in _______ quality in _________ settings, but better performance in ___________ settings.

Answer 35

1. Steeper
2. Worse
3. Low-resource
4. High-resource

Question 36

What is one of the primary challenges of Neural Machine Translation (NMT) models?

Answer 36

Data Scarcity. There is lots of data available available, for example, English —> French, but what about Pashto –> Swahili? This is a big challenge.

Question 37

Besides data scarcity, what are some of the other challenges associated with machine translation?

Answer 37

1. Language similarity - many (most) languages are very different than English
2. Domain - the training data that is available might not be closely related enough to your task of interest. (for example, Facebook trying to develop models for newsfeed using Ubuntu training manuals)
3. Evaluation - no access to a test set (this is one of main blockers to research in low-resource settings)

Question 38

What are some of the most powerful techniques for translation in low-resource settings?

Answer 38

1. Multi-lingual training (i.e. exploiting the relatedness between languages)
2. Backtranslation - using two languages (typically need to be fairly high-resource languagues) to train an intermediate model that can then be used to bootstrap a better model.
3. Language Agnostic Sentence Representations - using a model to take a low-resource language into a high-resource language (done using an Bi-Directional Recurrent Encoder-Decoder style network