Quiz #4

Question 1

What is an embedding?

Answer 1

A mapping between objects to vectors through a trainable function. Generally, we want that function to create a map such that similar objects are grouped together. Examples: Word Embeddings: Word –> Vector Graph Embeddings: Node –> Vector

Question 2

How is a graph embedding learned?

Answer 2

We optimize the objective that connected nodes have more similar embeddings than unconnected nodes via gradient descent.

Question 3

When representing structured information, what three things are important?

Answer 3

1. State: compactly representing all the data we have processed thus far 2. “Neighborhoods”: What other elements to incorporate? (e.g. spatial, part-of-speech, etc.) * Can be seen as selecting from a set of elements * Typically use some similarity measure or attention 3. Propagation of Information: How to update information given selected elements.

Question 4

In a fully connected network the weights that are applied to the input are data-dependent? (True/False)

Answer 4

False. In a FCN, the weights are learned and applied to the input regardless of the input values. This is an important driver behind the use of non-local style neural networks. The idea is that instead of outputting a simple dot product of the weights and the input, we actually use a similarity function ‘f’ (for instance, the exponentiated dot product exp(xi.T, xj)) and use that to modulate a representation of input element j, such as Wg*xj. This is a powerful concept because it allows us to make the WEIGHTS of the network DATA-DEPENDENT, since we’re modulating our feature representation by the similarity representation of two features. It allows the network to LEARN for each piece of data, what is SALIENT. This is really the main idea behind ATTENTION MECHANISMS. See 14:00/16:19 mark in Module 3 Lesson 11 “Structures and Representations” for review of this concept.

Question 5

What is a conditional language model?

Answer 5

It’s just like the standard language model (i.e. the probability of word given all the previously occurring words) but conditioned on an extra context ‘c’. Examples: * Topic-aware language model * Text summarization * Machine Translation * Image Captioning * Optical Character Recognition * Speech recognition

Question 6

What are four problems that arise if you try to use MLPs/FC networks for modeling sequential data?

Answer 6

1. Cannot easily support variable sized sequences as inputs or outputs 2. No inherent temporal structure 3. No practical way of holding state 4. The size of the network grows with the maximum allowed size of the input or output sequences.

Question 7

The lower the PERPLEXITY score, the better a model is? (True/False)

Answer 7

True.

Question 8

What are language models fundamentally used for?

Answer 8

To estimate the probability of a sequences of words given all the preceding words.

Question 9

What is Masked Language Modeling?

Answer 9

It is an auxiliary task, different from the final task we’re interested in, but which can help us achieve better performance by finding good initial parameters for the model.

Question 10

A recurrent network unfolded in time is really just a very deep feedforward network with shared weights? (True/False)

Answer 10

True. (Bengio, 1994)

Question 11

Gradient descent becomes increasingly inefficient when the temporal span of the dependencies increases? (True/False).

Answer 11

True. See Bengio, 1994 for good discussion of the problems associated with training NN with long term dependencies.

Question 12

What are the four main components of an LSTM network?

Answer 12

1. Input gate: This decides what new information we’re going to store in the cell state. This has two parts. First, a sigmoid layer called the “input gate layer” decides which values we’ll update. Next, a tanh layer creates a vector of new candidate values, C~t, that could be added to the state. 2. Forget gate: responsible for deciding what information is to be thrown away or kept from the last step. This is done by the first sigmoid layer. 3. Cell State: Essentially the memory of an LSTM, and the key that makes them much more performant at long sequences than vanilla RNNs. At each time-step the previous cell state (C_t-1) combines with the forget gate to decide what information is to be carried forward which in turn combines with the input gate (i_t and c~t) to form the new cell state or the new memory of the cell. 4. Output gate: The final output of the LSTM cell. The cell state obtained from above is passed through a hyperbolic function called tanh so that the cell state values are filtered between -1 and 1. https://towardsdatascience.com/lstm-gradients-b3996e6a0296

Question 13

What is the ‘Cell State’ in an LSTM network?

Answer 13

Essentially the memory of an LSTM, and the key that makes them much more performant at long sequences than vanilla RNNs. The cell state acts as a transport highway that transfers relative information all the way down the sequence chain. You can think of it as the “memory” of the network. The cell state, in theory, can carry relevant information throughout the processing of the sequence. So even information from the earlier time steps can make it’s way to later time steps, reducing the effects of short-term memory. As the cell state goes on its journey, information get’s added or removed to the cell state via gates. The gates are different neural networks that decide which information is allowed on the cell state. The gates can learn what information is relevant to keep or forget during training. https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21

Question 14

What is the output range of the tanh function?

Answer 14

[-1, 1]

Question 15

What is the output range of the sigmoid function? Why is this output range significant in the context of recurrent style NNs?

Answer 15

[0, 1]. It can be useful, for example, as part of the “forget gate” structure. If the output of the sigmoid is 0, then we could multiply it with some other input vector in order to zero out the elements in the vector, i.e. “forgetting” it.

Question 16

The price recurrent networks pay for their reduced number of parameters is that optimizing the parameters may be difficult? (True/False)

Answer 16

True. See page 379 of DL book.

Question 17

What are the four major components of an RNN?

Answer 17

1. Input 2. Hidden State 3. Weights/Parameters 4. Output

Question 18

Why is the use of fully connected layers/MLPs problematic for sequential/time-series data?

Answer 18

Since each of the weights and biases in a fully-connected network is INDEPENDENT, there’s no real way of maintaining the structure and order in the data. You could in theory make the network so large that it would have the capacity to memorize the order/structure information, but this would be so brittle and prone to overfitting that it doesn’t work in any practical setting.

Question 19

What role does the hidden state h(t) play in an RNN, and what is it a function of?

Answer 19

It’s a contextual vector at time t that acts as a “memory” of the past state(s) of the network. It is calculated as a function of the current input x(t) and the previous hidden state h(t-1).

Question 20

The hidden state and the inputs use the same weights and biases in an RNN? (True/False)

Answer 20

False. This is really a trick question. RNNs DO share weights/biases, but they are shared across time. However, the hidden state and the input weights aren’t just copies of the same weights, they each use their own set. More concretely, say we have the input weight matrix U and the hidden state weight matrix V. if we have the sentence fragment “The quick brown fox.”, we would feed each word in one at a time, and U and V would be applied to each word. Then in the backprop update, the weights for both U and V would be updated based on the gradient.

Question 21

RNNs share the same weights across time? (True/False)

Answer 21

True. This parameter sharing is one of the primary benefits of RNNs, as it allows the model to retain information about structure and temporal ordering.

Question 22

What is the “Distributional Semantics” concept?

Answer 22

It is the idea that if you understand the context that a word is used in, then it means you must have some understanding of the word itself. “You shall know a word by the company it keeps.” (Firth, 1957) It is one of the most successful ideas of modern statistical NLP.

Question 23

What is the CONTEXT of a word ‘w’ that appears in a text?

Answer 23

It is the words that appear nearby, i.e. within some fixed-size window. We can use the many different contexts that w appears in to build up a representation of the word.

Question 24

What is the idea behind Collobert & Weston vectors?

Answer 24

A word and its context comprise a POSITIVE training sample; a random word inserted in that sample context comprises a NEGATIVE training sample.

Question 25

What is the idea behind Word2Vec (i.e. the skip-gram model)?

Answer 25

Using words to predict their context words. The context here is a fixed window of size 2m. It creates a probability distribution for each of the words within the context window from -2m to +2m conditioned on the center word at position ‘t’.

Question 26

What is the objective function of the Word2Vec model?

Answer 26

Average negative log likelihood (where the likelihood is calculated by simply multiplying all the probabilities within the context window conditioned on the center word at position ‘t’ and the paramters theta that are being optimized.

Question 27

Word2Vec is fast to compute compared to earlier NN based models.? (True/False)

Answer 27

True. This is because it is only optimizing the word vectors, unlike the earlier NN based models

Question 28

How many sets of vectors are used for each word in the vocabulary for the Word2Vec model?

Answer 28

Two. U_w when w is a center word, and V_o when o is a context word. To measure how likely the word w appears with context word o, we use the inner product between U_w and V_o (with a softmax formulation so that the output is a probability score in range [0,1])

Question 29

How many parameters are used in the Word2Vec model (in terms of their center word and context word vectors U_w and V_o)?

Answer 29

The total parameters would the set of all center word vectors {U_w} and set of all context word vectors {V_o}.

Question 30

Why is SGD expensive to compute for the Word2Vec model (when using the basic softmax formulation)? What are two approaches for dealing with this?

Answer 30

Because the size of our entire vocabulary is huge. Two approaches mentioned that can be used to mitigate this: 1. Hierarchical Softmax 2. Negative Sampling

Question 31

What are the two main ways of evaluating word embeddings?

Answer 31

1. Intrinsic * Evaluation on a specific/intermediate subtask * Fast to compute * Helps to understand the system * Not clear if really helpful unless correlation to real task is established. 2. Extrinsic * Evaluation on real task (i.e. text classification) * Can take a long time to compute * Difficult to debug: unclear if subsystem is the problem or its interaction * If exactly replacing exactly one subsystem with another improves accuracy –> Winning!

Question 32

What is the idea behind ‘intrinsic’ word embedding evaluation?

Answer 32

Evaluate word vectors by how well their cosine distance after addition captures intuitive semantic and syntactic analogy questions. Example: Man:Woman::King:?

Question 33

Graph embeddings can be thought of as a generalization of word embeddings? (True/False)

Answer 33

True. In a graph context, an embedding is a learned map from entities to vectors of numbers that encodes similarity. This is similar to word embeddings, except that instead of mapping from word –> vector we map from node –> vector, and we want similar nodes to have similar vectors.

Question 34

Graph embeddings are a form of unsupervised learning on graphs? (True/False)

Answer 34

True.

Question 35

What are three reasons graph embeddings are useful?

Answer 35

1. They are task-agnostic entity representations. 2. Features are useful on downstream tasks without much data. 3. Nearest neighbors are semantically meaningful.

Question 36

The CBOW architecture predicts the current word based on the context, and the Skip-gram predicts surrounding words given the current word? (True/False)

Answer 36

True. See page 5 of https://arxiv.org/pdf/1301.3781.pdf for a very useful diagram of this.

Question 37

What is t-SNE?

Answer 37

t-Distributed Stochastic Neighbor Embedding. It’s a dimensionality reduction tool that is useful for visualizing high-dimensional datasets.

Question 38

How does t-SNE differ from PCA?

Answer 38

t-SNE differs from PCA by preserving only small pairwise distances or local similarities whereas PCA is concerned with preserving large pairwise distances to maximize variance.

Question 39

t-SNE is good for clustering?

Answer 39

I would say this is more False than True, although it isn’t totally black and white. While t-SNE might appear to be a good candidate for clustering because of it’s frequent use in dimensionality reduction to facilitate visualization of high dimensional data, it’s important to remember that t-SNE is a non-linear transformation that DOES NOT preserve distance nor density. See this link for a good discussion of this: https://stats.stackexchange.com/questions/263539/clustering-on-the-output-of-t-sne

Question 40

How is t-SNE performed?

Answer 40

The t-SNE algorithm calculates a similarity measure between pairs of instances in the high dimensional space and in the low dimensional space. It then tries to optimize these two similarity measures using a cost function (KL Divergence). https://towardsdatascience.com/an-introduction-to-t-sne-with-python-example-5a3a293108d1

Question 41

What is a language model?

Answer 41

Fundamentally, it’s a way of estimating the probability of a sequence occurring, for instance: P(“I ate an apple”) intuitively has a much higher probability of occurring than P(“I ate an Hawaii”). The ability to perform comparisons like this is one of the most important reasons we use language models! Mathematically, we could say something like: Prob(sequence) = product(Prob(next word | history))

Question 42

What is one main reason vanilla RNNs are difficult to train?

Answer 42

Vanishing/exploding gradient problems. For backprop the derivative of the output at time t with respect to to t0 is going to be proportional to the weight vector raised to to the power of t. This repeated application of the weights means that if the magnitude of W is > 1, we end up with exploding gradients, and if magnitude of W is < 1 we get vanishing gradients.

Question 43

What problem do LSTMs aim to remedy that are inherent to vanilla RNNs?

Answer 43

Vanishing gradient issues.

Question 44

What is the update rule for the forward pass of vanilla RNNs?

Answer 44

https://towardsdatascience.com/recurrent-neural-networks-rnns-3f06d7653a85

Question 45

What are the update equations for an LSTM?

Answer 45

Question 46

What is teacher/student forcing?

Answer 46

Teacher forcing is a strategy for training recurrent neural networks that uses ground truth as input, instead of model output from a prior time step as an input.

https://machinelearningmastery.com/teacher-forcing-for-recurrent-neural-networks/

Question 47

What is knowledge distillation?

Answer 47

Knowledge distillation refers to the idea of model compression by teaching a smaller network, step by step, exactly what to do using a bigger already trained network. The ‘soft labels’ refer to the output feature maps by the bigger network after every convolution layer. The smaller network is then trained to learn the exact behavior of the bigger network by trying to replicate it’s outputs at every level (not just the final loss).

https://towardsdatascience.com/knowledge-distillation-simplified-dd4973dbc764