Week 7 - Distributional Semantics

Question 1

Semantic Processing

Answer 1

The computer needs to “understand” what words mean in a given context

Question 2

Distributional Hypothesis

Answer 2

The hypothesis that we can infer the meaning of a word from the context it occurs in

Assumes contextual information alone constitutes a viable representation of linguistic items, in contrast to formal linguistics and the formal theory of grammar

Question 3

Distributional Semantic Model

Answer 3

Generate a high-dimensional feature vector to characterise a linguistic item

Subsequently, the semantic similarity between the linguistic items can be quantified in terms of vector similarity

Question 4

Linguistic Items

Answer 4

words (or word senses), phrases, text pieces (windows of words), sentences, documents, etc…

Question 5

Semantic space

Answer 5

The high-dimensional space computed by the distributional semantic model, also called embeding space, (latent) representation space, etc…

Question 6

Vector distance function

Answer 6

Used to measure how dissimilar two vectors corresponding linguistic items are

Question 7

Vector similarity function

Answer 7

Used to measure how similar two vectors corresponding linguistic items are

Question 8

Examples of vector distance/similarity function

Answer 8

Euclidean Distance
Cosine Similarity
Inner Product Similarity

Question 9

Euclidean Distance

Answer 9

Given two d-dimensional vectors p and q:

sqrt( sum(pi-qi)^2 for i=0->d) )

Question 10

Inner Product Function

Answer 10

Given two d-dimensional vectors p and q:

sum(pi*qi) for i=0->d

Question 11

Cosine Function

Answer 11

Given two d-dimensional vectors p and q:

sum(pi*qi) for i=0->d

divided by

sqrt( sum(pi^2) for i=0->d )
* sqrt( sum(qi^2) for i=0->d )

Question 12

Vector Space Model

Answer 12

count based

Algebraic model for representing a piece of text object (referred to as a document) as a vector of indexed terms (e.g. words, phrases)

In the document vector, each feature value represents the count of an indexed term appearing in a relevant piece of text

By collecting many document vectors and storing them as matrix rows (or columns), it results in the document-term matrix.

Might treat the context of a word as a mini-document

Question 13

VSM term weighting schemes

Answer 13

Binary Weight

Term Frequency (tf)

Term Frequency Inverse Document Frequency (tf-idf)

Question 14

VSM binary weighting

Answer 14

Each element in the document-term matrix is the binary presence (or absence) of a word in a document

Question 15

VSM Term Frequency Weighting

Answer 15

Each element in the document-term matrix is the frequency a word appears in a document, called term frequency (tf)

Question 16

Inverse Document Frequency

Answer 16

Considers how much information the word provides, i.e. if it’s common or rare across all documents

idf(k) = log(M / m(k))

Where:
M - total number of documents in the collection
m(k) - Number of documents in the collection that contain word k

Question 17

Term Frequency Inverse Document Frequency

Answer 17

For document i and word k
t(i,k) = tf(i,k) * idf(k)

idf(k) = log(M / m(k))

Where:
M - total number of documents in the collection
m(k) - Number of documents in the collection that contain word k

idf considers how much information the word provides, i.e. if it’s common or rare across all documents

Question 18

VSM for word similarity

Answer 18

Construct two vectors using the VSM (Vector space model)

Use cosine (or inner product) similarity to compute the similarity between the word vectors

Two approaches for getting word vectors:
Based on documents
Based on local context

Question 19

Context based word similarity

Answer 19

Instead of using a document-term matrix, use a word-term matrix, populating it with:

the co-occurence of a term and a word within the given context windows of the term, as observed in a text collection.

Question 19

Context Engineering

Answer 19

How to choose the context for context based word similarity

Options:
- Whole document that contains a word
- All words in a wide window of the target word
- content words only (no stop words) in a wide window of the target word.
- Content words only in a narrower window of the target word
- Content words in a narrower window of the target word, which are further selected through using some lexical processing tools

Question 20

Document-Term Matrix

Answer 20

A matrix with columns of terms, and rows of documents

Question 21

Context Window Size

Answer 21

In the context of Context Engineering and Context based word similarity:

• Instead of entire document, use smaller contexts, e.g. paragraph, window of +- 4 words

Shorter window (1-3 words) - focus on syntax
Longer window (4-10 words) - capture semantics

Question 22

Benefits of low-dimensional dense vectors

Answer 22

-Easier to use as features in machine learning models
- Less noisy

Question 23

Latent Semantic Indexing

Answer 23

Mathematically is a singular value decomposition

Using SVD results computes:
document vectors: UD
term vectors: VD

Between document similarity: U * D^2 * UT
Between term similarity: V * D^2 * VT

The dimension k is normally set as a low value e.g. 50-1000 given 20k-50k terms

Question 24

Singular Value Decomposition

Answer 24

Decomposing a vector into three matrix components S, V, and D.

If there are m documents and n terms, and X is an mxn matrix:
X = U D V^T

Each row of V is a k-dimensional vector related to a term, it has n columns

Each row of U is a k-dimensional vector related to a document

The dimension k is normally set as a low value e.g. 50-1000 given 20k-50k terms

Question 25

Truncated SVD

Answer 25

Choose a number of k and truncate the SVD

Each row of UD provides a k-dimensional feature vector to characterise the row object

Each row of VD provides a k-dimensional feature vector to characterise the column object

Can be applied to any data matrix, not necessarily only the document-term matrix

Question 26

Predictive word embedding models

Answer 26

Perform prediction tasks based on word co-occurence information. E.g:
- Whether a word appears in a context of a target word
- How many times a word appears in the context texts of a target word

Examples for training are words and their context in a text corpus

Include:
- (general) continuous bag-of-words model
- skip-gram model
- GloVe model
- …

Question 27

Continuous-bag-of-words model

Answer 27

Assuming there are V words in the vocabulary we are dealing with a V-class classification task.

The input of each sample contains C context words

Objective is to learn a word embedding matrix of V rows and N columns (N is a hyperparameter) called W

Objective is to learn a word embedding for the vocabulary

Inputs to the model are one-hot encodings of the vocabulary where the non-zero element represents the context word being input.

Feature extraction component (h1 - output of hidden layer):
Copies the word embedding vectors for the context words from the rows of the embedding matrix, and averages them

Multi-class classification component:
Takes hi as the feature input and assigns it to one of the word classes in the vocabulary using logistic regression (a linear classification model trained using cross-entropy loss)

W’ is a matrix NxV which denotes the multi-class classification weight matrix of the logistic regression model

Like skip gram, based on wether two-words appear in each other’s context, doesn’t directly take into account the number of times two words appear in each other’s context.

Question 28

Skip-gram model

Answer 28

Like a continuous bag of words model but flipped, predicting the context of target word, instead of predicting the target word from the context

Like CBOW, based on wether two-words appear in each other’s context, doesn’t directly take into account the number of times two words appear in each other’s context.

Question 29

GloVe Model

Answer 29

Different from CBOW and Skip-gram utilises the frequency that a word appears in another word’s context in a given text corpus

Question 30

What to do with a semantic space

Answer 30

Clustering
- grouping similar words together

Data visualisation
- mapping the semantic space to a two (or three) dimensional space

Support other NLP tasks
- To be used as the input of machine learning models. E.g. Neural networks for solving NLP tasks.

Question 31

Advantages and disadvantages of distributional semantics

Answer 31

Advantages:

• Very practical in terms of processing
• Effective in capturing word meaning and relations, and support the training of neural language models

Disadvantages:

• It is still an open issue whether statistical co-occurrences alone are enough to address deeper semantic questions
• Semantic similarity is still a vague notion. For instance, the association between “car” and “van” is different from that between “car” and “wheel” (semantic similarity vs semantic relatedness)
• What type of semantic information can be captured from context, and what part of the meaning of words remains unreachable without complementary knowledge?

Question 32

Disadvantages of distributional semantics