Word2Vec

Question 1

Is Word2Vec a dense or sparse embedding?

Answer 1

Dense

Question 2

Are dense or sparse embeddings better?

Answer 2

Dense embeddings work better - not sure why

Question 3

What does static embedding mean?

Answer 3

It means that the words are single embeddings that do not change

Question 4

How does contextual embedding differ from static embedding?

Answer 4

The vector for the word will vary depending on its context

Question 5

What does self-supervision mean?

Answer 5

It means that a large tagged dataset is not needed

Question 6

What does Word2Vec do in simple terms?

Answer 6

Trains a binary classifier to predict if Word A is likely to show up near word B - we then use the embedding weights

Question 7

What model is used by Word2Vec?

Answer 7

The skip-gram model

Question 8

What is the skip-gram model?

Answer 8

It takes a set of target words and neighbouring context words (positive examples), random samples are taken to create negative examples, logistic regression is used to train classifier and the learned weights provide the embeddings

Question 9

What does a skip-gram model store?

Answer 9

A target embedding for matrix W for target words and a context embedding matrix C for context and noise words

Question 10

How do we avoid a bias towards common words when selecting noise words?

Answer 10

We use a weighted unigram sample frequency to select the noisy words

Question 11

How are Word2Vec embeddings learnt?

Answer 11

They are learnt by minimising a loss function using stochastic gradient descent (SGD)

Question 12

What does the loss function do in Word2Vec?

Answer 12

Aims to maximise the probability that the target word is close to the positive examples and maximise the probability that the target word is not close to the negative examples

Question 13

Visually, what are we trying to do with Word2Vec?

Answer 13

We are trying to move the weights such that there is an increased association between the target word and positive examples, and a decrease when we have the negative examples

Question 14

How are the target matrix and context matrix initialised?

Answer 14

They are randomly initialised, typically with Gaussian noise

Question 15

How is the word embedding matrix initialised?

Answer 15

Target Matrix (W) + Context Matrix (C)

Question 16

What are some other static embeddings?

Answer 16

Fasttext and Global Vectors (GloVe)

Question 17

What words appear in a context window when it is small?

Answer 17

Words that are similar, such as those within a list

Question 18

What words appear in a context window when it is larger?

Answer 18

Words that are more associated rather than similar, so it will capture longer distance topical relationships

Question 19

What do analogies mean in relation to Word2Vec?

Answer 19

It looks at how A is to B as C is to …

Question 20

How are analogies computed using Word2Vec?

Answer 20

We compute the vector that takes you from the embedding space from A to B, and apply the offset to word C and find what words are similar

Question 21

When using Word2Vec with analogies, what needs to be excluded?

Answer 21

Morphological variants of the target word (i.e We don’t want potato → potato | potatoes, but we do want potato → brown)

Question 22

What are the problems with bias and embeddings?

Answer 22

Bias of the time will be included, which can be problematic when we consider how the world constantly changes

Question 23

What is allocation harm?

Answer 23

it is where bias in algorithms result in unfair real world outcomes (e.g. credit check that results in denial due to some underlying bias)

Question 24

What is bias amplification?

Answer 24

It is where embeddings exaggerate patterns making encodings even more bias than the original training resource. Implicit bias can be captured (racial, sexist, ageist bias)

Question 25

What is representational harm?

Answer 25

It is where harm is caused by a system demeaning or ignoring some social groups

Question 26

What is debiasing?

Answer 26

It is a way of manipulating embeddings to remove unwelcome stereotypes, it may reduce bias but will not eliminate it

Question 27

When do two words have first-order co-occurrence?

Answer 27

When they are typically near each other

Question 28

When do two words have second-order co-occurrence?

Answer 28

When they have similar neighbours