Word Embeddings

Question 1

TF-IDF

Answer 1

term frequency - inverse document frequency

tf-idf(t, d, D) = tf(f,d) * idf(t,D)

tf-idf(t, d, D) = f_{t,d} * log(|D| / |{d \in D: t \in d}|)

Question 2

Continuous Bag-of-Words (CBOW)

Answer 2

Predicts target word w_t from context words:

w_{t-j} … w_{t-2} w_{t-1} ? w_{t+1} w_{t+2} … w_{t+j}

Question 3

Skip-gram model

Answer 3

Predicts context words from the target word

? ? ? w_t ? ? ?

Question 4

Skip-gram model target function

Answer 4

1/T \sum_t \sum_{-c < j < c} log(p(w_{t+j} | w_t; \theta))

find argmax \theta (learned embedding vector)

Question 5

softmax (NLP context)

Answer 5

p(w_{t+j} | w_t) = exp(dot(v’w{t+j}, v_w_t)) / \sum_{w’=1}^W exp(dot(v’_w’, v_w_t))

v'_w_{t+j} = vector presentation of the context word
v_w_t = vector representation of the input word