Word Embeddings

Question 1

Embeddings

Word Embedding

Answer 1

• Continuous vector representation of words
• Can represent features of syntax and/or semantics
• Can introduce more complex mappings

Question 2

Representations

Distributional Representations

Answer 2

• Words are similar if they appear in similar contexts
• In contrast: non-distributional representations are created from lexical resources, such as WordNet

Question 3

Embeddings

WordNet

Answer 3

Large database of word embeddings, including parts of speech, semantic relationships, etc.

Question 4

Representations

Distributed Representations

Answer 4

• Each item is represented by vector of values
• Each feature in the vector represents distinct attribute
• In contrast: local representations are represented using discrete symbols

Question 5

Types of Embeddings

Two types of embeddings

Answer 5

1. Count-based
2. Prediction-based

Question 6

Types of Embeddings

Count-Based Embeddings

Answer 6

• Represent a word using the normalized counts of its context words
• Sparse vectors

Question 7

Types of Embeddings

Prediction-Based Embeddings

Answer 7

• Represent a word using a small-dimension continuous vector
• Vectors are learned by training a classifier to predict the word (unsupervised)
• Word emdeddings are the byproduct
• Dense vectors

Question 8

Count-Based Embeddings

How to create word-context count matrix

Answer 8

• count # of co-occurences of word/context
• rows as words
• columns as contexts
• Co-occuring words like “they” and “the” are low-importance and shouldn’t mean anything

Question 9

Count-Based Embeddings

tf-idf

Answer 9

w_(t,d) = tf_(t, d) * idf_t

• Weight value for word t in document d
• words like “the” and “it” have very low idf

Question 10

Count-Based Embeddings

PMI

Answer 10

PMI(w1, w2) = log( p(w1, w2) / ( p(w1)*p(w2) ) )

• Pointwise mutual information
• See if words like “good” appear more often with “great” than we would expect by chance

Question 11

Count-Based Embeddings

How to measure the closeness?

Answer 11

Measure the closeness using cosine similarity

Question 12

Prediction-Based Methods: WordVec

CBOW

Answer 12

• Continuous bag of words
• NLP model
• Predict word based on sum of surrounding embeddings
• Used in Word2Vec
  Input: context of words, fixed num of words surrounding target
  Output: target word context is associated with

Question 13

Prediction-Based Methods: WordVec

Skip-gram

Answer 13

• NLP model
• Predict each word in the context given the word
• Used in Word2Vec
  Input: target word in the middle of a context
  Output: surrounding context words that are likley to appear around the context word

Question 14

Embeddings

How to learn word embeddings?

Answer 14

1. Pre-train on unsupervised task and use byproduct embeddings
2. Randomly initialize embeddings then train jointly with the task
3. Pre-train on unsupervised task (e.g. POS tagging) and test on another (e.g. parsing) and use byproduct embeddings

Question 15

Contexts

Small Context Window

Answer 15

Creates syntax-based embeddings

Question 16

Contexts

Large Context Window

Answer 16

Creates more semantics-based embeddings

Question 17

Contexts

Context based on syntax

Answer 17

More functional embeddings, w/ words from same inflection grouped

Question 18

Count-Based Embeddings

tf_(t, d)

Answer 18

tf_(t,d) = (num of times term t appears in doc d) / (total num of terms in doc d)

Question 19

Count-Based Embeddings

idf_t

Answer 19

idf_t = log( (total num of of docs) / (num of documents containing term t) )

Question 20

Prediction-Based Methods

Two Prediction-Based Methods

Answer 20

1. WordVec (CBOW, Skip-gram)
2. Neural Language Models

Question 21

Embeddings

How to use word embeddings?

Answer 21

People release pre-trained word embeddings as resources. Two ways to use:
* Initialize
* Concatinate

Question 22

Embeddings

How to initialize word embeddings?

Answer 22

Initialize with pre-trained embeddings

Question 23

Embeddings

How to create concatenated word embeddings?

Answer 23

Concatenate pre-trained embeddings with learned embeddings. Makes larger embeddings.

Question 24

Embeddings: Evaluating Embeddings

Two ways to evaluate word embeddings

Answer 24

1. Intrinsic vs Extrinsic
2. Qualitative vs Quantitative

Question 25

Embeddings: Evaluating Embeddings

Intrinsic vs Extrinsic

Answer 25

Intrinsic: How good is it based on its features?
Extrinsic: How useful is it for the downstream tasks?

Question 26

Embeddings: Evaluating Embeddings

Qualitative vs Quantitative

Answer 26

Qualitative: Examine the characterisitics of examples
Quantitative: Calculate statistics

Question 27

Embeddings: Evaluating Embeddings

Semantic-Syntatic Word Relationship Test

Answer 27

• 5 types of semantic questions
• 9 types of syntatic questions
• Evaluation: accuracy on retrieving correct word as the closest word

Question 28

Embeddings

Visualization of Embeddings

Answer 28

Reduce high-dimensional embeddings into 2/3D for visualization

Question 29

Embeddings

Non-Linear Projection

Answer 29

Groups things that are close in high-dimension space

Question 30

Embeddings

Limitations of Embeddings

Answer 30

• Sensitive to superficial differences (dog/dogs)
• Insensitive to context (financial bank, bank of river)
• Non necessarily coordinated with knowledge across language, just based on training data
• Not interpretable
• Can encode bias