Word Embeddings

Question 1

How does a count based word embedding differ from the term-term matrix of distributional semantics?

Answer 1

we use a term-document matrix
the meaning of a word is then
[count(u,d1), count(u,d2)…]

Question 2

what do we get if we read a term-document matrix column wise

Answer 2

represents a document

Question 3

what do we get if we read a term-document matrix row wise

Answer 3

represents a word

Question 4

why do we use tf-idf

Answer 4

some words are very common but arent important. this allows us to perform weighting

Question 5

how do we calculate tf

Answer 5

log_10( 1 + count(t,d)

Question 6

how do we calculate idf

Answer 6

log_10(N / df(t))
N is number of documents
df(t) is the number of documents where t appears

Question 7

how do we calculate tf idf

Answer 7

tf * idf

Question 8

what is laplace smoothing and why do we use it

Answer 8

add 1 to every entry as a pseudo count

this ensures that if a word pair occurs >0 then it does not become 0 due to sparsity

Question 9

what are the benefits of embedding words using term-document matrix (3)

Answer 9

simple and intuitive
dimensions are meaningful
easy to debug and interpret

Question 10

what are the drawbacks of embedding words using term-document matrix (2)

Answer 10

sparse- there may be lots of terms and documents

word meaning may be different in a particular context

Question 11

Formalise the challenge of knowing the meaning of a word within its contect

Answer 11

we have words represented by vectors
we have the probability of a word within context Pr(w | u1, u2 …)

our task is to find a vector for a word that will maximise pr(w| …) within its context

the loss function is
the sum for each word and context in train: log(p(w|u1….)

Question 12

How does the bengio model work?

Answer 12

Predict the next word using m-1 previous words:

p(Wt | Wt-1 …. Wt-m)

Question 13

How does the CBOW model work?

Answer 13

Predict a word from the context of m words before and m words after

Question 14

Give the architecture of the cbow model

Answer 14

multiple input nodes- one for each word in window
take the average
apply softmax

to get p(w|context)

Question 15

Give the architecture of the Bengio model

Answer 15

multiple input nodes- one for each word in window
concat
tanh
softmax

to get p(w|context)

Question 16

How does the skipgram model work?

Answer 16

Predict the context of m words before and m words after given a target word

Question 17

Give the architecture of the skipgram model

Answer 17

w(t) as input
softmax
multiple input nodes- one for each word in window as output

Question 18

How can we go from cbow to skipgram

Answer 18

bayes

Question 19

What is Word2Vec?

Answer 19

A skipgram model that can capture linear relational meanings like analogy and shows semantic change over time

Question 20

What are the problems with Word embeddings? (3)

Answer 20

• bias
• unknowns
• can only capture one context

Question 21

How can we capture more than one context of a word?

Answer 21

• make f(w,c) a continuous function, like elmo or bert

- make f(w,c) discrete with respect to c. Word sense disambiguation