W5 L2 distrubutional semantics

Question 1

watch a video on what feedforward neural networks are

Answer 1

yuh

Question 2

watch a video on binary logsitc regression as a one layer network

Answer 2

yuh

Question 3

watch video

Answer 3

Multinomial Logistic Regression as a
1-layer Network

Question 4

watch a video on softmax

Answer 4

yuh

Question 5

watch a video on two layer network with scalar output

Question 6

what are word emeddings

Answer 6

short vectors with dimenisons ranging from 50-1000

its faster because vectors are dense

Question 7

what is word2vec

Answer 7

two architectrures
continous bag of words model
and contiuous skip gram model
in a software package called word2vec

its for word emedding

Question 8

what do you train a classifier to solve in word2vec

Answer 8

you trian it to solve the binary task

is word c likely to show up near apicot

c is the correct answer to the question

Question 9

what are we actually interested in in w2v skip gram neg sampling

Answer 9

not the answer to the question is word c likely to show up near apricot but

the weights the classifer learns in order to answer this question

these weights are used in the embedding for the target word appricot

Question 10

what is the producure for word2 vec

Answer 10

In essence, word2vec performs binary classification training a logistic regression classifier. Here is the
procedure that the skip-gram model follows:

1. It treats the target word w and a neighbouring context word c as positive examples.
2. Then, it randomly samples other words in the vocabulary and uses them as negative samples.
3. Next, it uses logistic regression to train a classifier aimed at distinguishing between positive and
   negative samples.
4. Finally, it uses the learned weights as the word embeddings

Question 11

what is pca

Answer 11

principal component analysis applies a linear transform to the data so those dimensions which have the most variance (principle componets) can be easily identified

Question 12

how can we find principle componets

Answer 12

we can use singular value decomposition to find principal components

Question 13

what is the problem with using singular value decpomosition to find principle components

Answer 13

its expensive

Question 14

instead of using expnesive singular value decomposition what can we do

Answer 14

we can use word2vec

Question 15

what is the goal of the classiffer in word2vec

Answer 15

given a tuple (w, c) for the target word w paired with a
candidate context word c (e.g., (apricot, jam) and (apricot, aardvark)), return the probability that c is a real
context word (which should return True for jam, and False for aardvark). The probability for the actual
context word c is defined as:

P(+|w,c)

the chance that c is not a real context of the target word is
P(-|w,c) = 1 - P(+|w,c)

Question 16

how does the word2vec classfier estimate the probability p

Answer 16

the skip-gram model uses
embeddings similarity: a context word c is likely to occur near the target word w if its embedding is similar
to the target embedding. To compute this similarity, the skip-gram model uses dot product

similar(c,w) = c * w

The higher the dot product, the more similar the two embeddings are (note that cosine is a length normalised dot product).

Question 17

What simplifying assumption does the skip-gram model make?

Answer 17

The skip-gram model assumes context words are independent, similar to assumptions in Naïve Bayes and Markov models. This allows the model to multiply individual probabilities.

Question 18

What does the skip-gram model do?

Answer 18

It trains a probabilistic classifier that, given a target word
𝑤 and a context window of
𝐿 words
𝑐1:𝐿
c 1:L, assigns a probability based on how similar each context word is to the target word.

Question 19

How does the skip-gram model calculate probabilities?

Answer 19

It applies the logistic (sigmoid) function to the dot product of the target word’s embedding and each context word’s embedding.

Question 20

What embeddings does the skip-gram model learn?

Answer 20

The model learns:

Target word embeddings.
Context word embeddings.

Each word has two embeddings: one for when it’s a target word and another for when it’s a context word.

Question 21

What are the two matrices learned in the skip-gram model?

Answer 21

W: Contains embeddings for target words.
C: Contains embeddings for context words.

Question 22

what are the two emdeddings word2vec stores per word

Answer 22

one as target word and one as context word

Question 23

what is the process for learning embeddings

Answer 23

start with random initalization for the word emeddings

iterartivly update emeddings so that the embedding for the word w gets closer to the embedding of words that occur nearby and further wawy from the emebddings of words that do not co-occcur

Question 24

what are negative samples in skip gram

Answer 24

Since the model trains a binary classifier, apart from positive it also needs some negative instances. The
skip-gram model uses negative sampling, which means that it randomly selects a number of negative
examples with the ratio of negative-to-positive examples defined by the parameter k: in other words, for
each (w, cpos) training instance, the algorithm randomly selects k “noise” words from the vocabulary,

Question 25

What is the goal of the skip-gram algorithm?

Answer 25

: To adjust embeddings so that: Similarity of (w, c_pos) pairs (target and real context words) is maximized. Similarity of (w, c_neg) pairs (target and negative/noise words) is minimized.

Question 26

What does the skip-gram loss function ( L CE ) represent?

Answer 26

The loss function has two terms: First term: Maximizes the probability of the real context word c pos ​being a neighbor of w. Second term: Minimizes the probability of negative words c neg ​being neighbors of w.

Question 27

Why are the loss function terms multiplied?

Answer 27

The skip-gram model assumes word independence, so probabilities of context words can be multiplied

Question 28

How does the skip-gram model evaluate similarity?

Answer 28

By the dot product of the target word embedding 𝑤 w with: c pos ​ (real context words) to maximize similarity. c neg ​ (negative samples) to minimize similarity.

Question 29

How is the skip-gram loss function minimized?

Answer 29

Using stochastic gradient descent (SGD), which is commonly used for optimizing models like logistic regression and neural networks.

Question 30

What does the skip-gram learning process aim to achieve in one step?

Answer 30

It adjusts embeddings to bring the target word w closer to its true neighbors c pos ​ and farther from noise words c neg ​

Question 31

What is a core assumption in building word embeddings?

Answer 31

Word embeddings can capture semantic similarity between words, making it essential to evaluate them on this ability.

Question 32

How is semantic similarity tested without context?

Answer 32

By comparing word similarity scores (e.g., cosine similarity of word vectors) with human-assigned ratings using datasets like: WordSim-353: 353 noun pairs with similarity ratings (e.g., "plane, car" = 5.77). SimLex-999: Adjective, noun, and verb pairs with ratings. TOEFL dataset: 80 questions with a target word and 4 choices, one synonymous

Question 33

How is semantic similarity tested in context?

Answer 33

Using datasets like: SCWS: Stanford Contextual Word Similarity. WiC: Word-in-Context dataset. These test models on human judgments of word use in sentential contexts.

Question 34

What is the semantic textual similarity task?

Answer 34

It focuses on sentence-level similarity, where pairs of sentences are labeled with human-assigned similarity scores.

Question 35

How is compositionality tested in semantic representations?

Answer 35

By evaluating their ability to perform paraphrasing tasks, such as: Fantasy world → Fairyland. Dog house → Kennel. Fish bowl → Aquarium.

Question 36

How is relational meaning tested in semantic models?

Answer 36

Using analogy tasks like: "Apple is to tree as grape is to ___ (vine)." Solved with vector arithmetic: 𝑏 ∗ =𝑎 ∗ − 𝑎 +𝑏 where the closest vector to the result is selected.

Question 37

How does Word2Vec solve analogy tasks

Answer 37

By using vector arithmetic, such as: "King - Man + Woman" → Queen. "Queen - King + Kings" → Queens. These demonstrate linguistic regularities in word embeddings