C3

Question 1

vector space model

Answer 1

dimensions are the words, documents are vectors: shows which words a document contains

Question 2

2 problems vector space model

Answer 2

synonymy: many ways to refer to the same object (bike and bicycle)

polysemy: many words have more than one distinct meaning

Question 3

word embeddings model

Answer 3

represent words in a continuous vector space

• relatively low dimensional vector space
• semantically and syntactically similar words are mapped to nearby points (distributional hypothesis)

Question 4

feedforward network

Answer 4

multilayer network in which the units are connected with no cycles

• three kinds of nodes: input, hidden, output
• each layer is fully connected (in standard architecture)

Question 5

classification with feedforward network

Answer 5

Binary: single output node

multi-class: output node for each category, output layer gives probability distribution

Question 6

word2vec

Answer 6

train a neural classifier on a binary prediction task: is word w likely to show up near the word bicycle? => Take the learned classifier weights on the hidden layer as the word embeddings

computationally efficient predictive model for learning word embeddings from raw text

Question 7

training word2vec

Answer 7

supervised learning problem on unlabeled data: self-supervision

1. treat target word and a neighbouring context word as positive examples
2. randomly sample other words in the lexicon to get negative samples
3. train a classifier to distinguish those two cases
4. learned weights are the embeddings

• maximize similarity of (target word, context word) pairs drawn from positive examples
• minimize similarity of (w,c_neg) pairs from negative examples
• each word + context is a classification problem (adjust current vector or not)

=> weights get updated while the model processes the collection (minimize and maximize dot-products)

Question 8

3 advantages of word2vec

Answer 8

• it scales: can be trained on billion word corpora in limited time + possibility of parallel training
• pre-trained word embeddings trained by one can be used by others
• incremental training: train one piece, save results, continue later

Question 9

4 text mining tasks that benefit from using word2vec

Answer 9

• synonym detection
• richer word and context representation for named entity recognition
• document similarity
• finding word associations/clusters

Question 10

tf-idf

Answer 10

w_t,d = tf_t,d * idf_t

tf = term frequency of word t in document d = 1 + log_10(count(t,d))

idf = inverse document frequency = log_10(N/df_t) with N the total number of documents and df_t the number of documents t occurs in

Question 11

PMI

Answer 11

Pointwise Mutual Information

PMI(w,c) = log_2(P(w,c)/P(w)P(c))

estimate of how much more the two words co-occur than we expect by chance

Question 12

first-order co-occurrence

Answer 12

when two words are typically nearby each other (“wrote” and “book”)

Question 13

second-order co-occurrence

Answer 13

when two words have similar neighbours (“wrote” and “said”)

Question 14

distibutional hypothesis

Answer 14

words that occur in similar contexts tend to be similar (the context of a word defines its meaning)

Question 15

proposed neural architectures for computing word vectors

Answer 15

• word2vec
• GloVe
• FastText
• ELMo
• BERT