NLP

Question 1

General methods to create Word embeddings?

Answer 1

1. One Hot Encoding
2. Bag of Words
3. N-Grams
4. Tf-Idf
5. Integer Encoding

Question 2

OHE : Advantages & Disadvantages

Answer 2

Advantages - Easy & Intuitive
Disadvantages -
1. Sparse vectors
2. Out of Vocabulary words
3. No semantic meaning
4. Fixed Size

Question 3

BOW : Advantages & Disadvantages

Answer 3

Advantages - Easy & Intuitive
Disadvantages -
1. Sparse vectors
2. Out of Vocabulary words
3. No semantic meaning

Question 4

N-grams : Advantages & Disadvantages

Answer 4

Advantages - Easy & Intuitive; Captures semantic meaning to an extent
Disadvantages -
1. Out of Vocabulary words
2. Computationally expensive

Question 5

What is Tf-Idf

Answer 5

Term Frequency-Inverse Document Frequency

TF measures the frequency of a word in a document.
TF(t,d)= Totalnumberoftermsindocumentd/ Numberoftimestermtappearsindocumentd

IDF measures how often a term appears in a document wrt whole corpus.
IDF(t)=log(Numberofdocumentscontainingtermt/
Totalnumberofdocuments)

Question 6

Why do we use Log in IDF?

Answer 6

This is because for words that has occurred in very few documents, their IDF value will be too high. And, therefore contribution of TF value will be neglected.

Question 7

Advance methods to create Word embeddings?

Answer 7

1. Word2Vec
2. GloVe
3. fastText
4. Transformers

Question 8

What are Word Embeddings?

Answer 8

Vector representation of words.
Embeddings captures the semantic meaning of the information.

Ex - Embedding of the sentence “today is a sunny day” and “the weather is nice today” will be similar.

Question 9

What is Word2Vec & GloVe?

Answer 9

Words that appear in similar contexts have similar embeddings.
However, these technique will struggle when a word has multiple meanings, because Glove and Word2Vec have fixed representation of a word.

Question 10

Word2Vec

Answer 10

By Google

Trained on a very large corpus by training a shallow neural network.
1. Predict the surrounding words given the center word. - Skip-Gram
2. Predict the center word given the surrounding words. - CBOW

Question 11

GloVe

Answer 11

By Stanford

Trained by looking at the co-occurrence matrix of words (how often words appear together within a certain distance) and then using that matrix to obtain the embeddings.

Question 12

fastText

Answer 12

By Facebook

Question 13

Embeddings using Transformers

Answer 13

Transformers learn embeddings in the context of their task.

For example, BERT learns word embeddings in the context of masked language modeling (predicting which word to fill in the blank) and next sentence prediction (whether sentence B follows sentence A).

Question 14

CLS token

Answer 14

Classification Token - Represents the entire sentence

Question 15

SEP token

Answer 15

Separator Token - Separates sentences

Question 16

Word embedding for a word that is split in more than one token

Answer 16

It can be found out by using pooling strategy in which we obtain the embedding of each token and then average them to obtain the word embedding.

Question 17

What are Sentence Embeddings?

Answer 17

Vector representation of a sentence.
Sentence embeddings are inherently compressions of the information in a sequence of text (words) and compressions are inherently lossy. This implies that sentence embeddings are representations with a lower level of granularity.

Three ways: -
1. CLS Pooling
2. Max Pooling
3. Mean Pooling

Question 18

CLS Pooling

Answer 18

Embedding corresponding to the [CLS] token.
CLS token capture the idea of the entire sentence

Used when the transformer model has been fine-tuned on a specific downstream task that makes the [CLS] token very useful.

Question 19

BERT vs RoBERTa

Answer 19

BERT performs NSP in its original training process so a pre-trained BERT produces meaningful CLS representations out-of-the-box.

RoBERTa does not use NSP in its pre-training process nor does it perform any other task that tunes the CLS token representation. Therefore, it does not produce meaningful CLS representations out-of-the-box.

Question 20

Why do we need sentence embeddings over word embeddings?

Answer 20

Sentence embeddings are trained for tasks that require knowledge of the meaning of a sequence as a whole rather than the individual tokens.
Some concrete examples of such tasks are:
👉 Sentiment analysis
👉 Semantic similarity
👉 NSP (in BERT’s pre-training)

Question 21

Cosine Similarity

Answer 21

Cosine of the angle between the embeddings.
It compare how similar two vectors are regardless of their magnitude.

If the angle between the embeddings is small, the cosine will be close to 1 and as the angle grows, the cosine of the angle decreases.

Question 22

Mean Pooling

Answer 22

Averaging all the word embeddings of the sentence

More effective on models that have not been fine-tuned on a downstream task. It ensures that all parts of the sentence are represented equally in the embedding.

Question 23

Max Pooling

Answer 23

Taking the maximum value of each dimension of the word embeddings.

Useful to capture the most important features in a sentence. This can be very useful if particular keywords are very informative, but it might miss the subtler context.

Question 24

Is polling strategy better than what to Word2Vec or GloVe?

Answer 24

No, the reason is that the [CLS] token is not trained to be a good sentence embedding. It’s trained to be a good sentence embedding for next-sentence prediction!

Question 25

Sentence Transformer

Answer 25

Also called as SBERT.

It uses Mean or Max Pooling.
Other step is to perform normalization which ensures that the embedding vector has a unit length (Magnitude = 1).

Question 26

Why do we need Normalization?

Answer 26

This is particularly helpful when we want to compare vectors.

The magnitude of the embeddings is not relevant when computing the cosine similarity, but it is required for other distance measures.

Question 27

Lexical search system

Answer 27

Looks at exact matches of the input question

Question 28

Fuzzy search system

Answer 28

Looks at approximate matches of the input question

Question 29

Statistical search system

Answer 29

Looks at the frequency of words in the input question

Question 30

Embedding dimensions

Answer 30

all-MiniLM-L6-v2 - 384
OpenAI - 1536

Open Source Models - 384 to 1024
Closed Source Models - 384 to 4096

Embedding dimension is a trade-off.
Very large embeddings, will potentially give better results, but is also costlier for hosting and inference.
For vector databases, will have to pay more for storage.

Question 31

Formula of Cosine similarity

Answer 31

Cosine similarity is defined as the dot product of the vectors divided by the product of their magnitudes.

Question 32

Dot Product

Answer 32

When the magnitude of the vector is important.
Dot Product measures how much one vector extends into the direction of another vector. It focusses on both magnitude and angle.

Cosine similarity is a normalized dot product.

Question 33

Euclidean Distance

Answer 33

Distance between two vectors by measuring a straight line between them.

Question 34

Temperature Scaled Mixing

Answer 34

The temperature parameter allows you to weigh certain examples more than other.

Question 35

Stemming vs Lemmatization

Answer 35

Stemming is a process that stems or removes last few characters from a word, often leading to incorrect meanings and spelling.

Lemmatization considers the context and converts the word to its meaningful base form, which is called Lemma.

Question 36

POS tagging

Question 37

N-gram

Answer 37

Sequence of N-words

Question 38

Perplexity

Answer 38

• Perplexity is a measure of the complexity in a sample of texts.
• Perplexity can be related to entropy, which measures uncertainty.

Perplexity is used to tell us whether a set of sentences looks like they were written by humans rather than by a simple program choosing words at random. A text that is written by humans is more likely to have lower perplexity, whereas a text generated by random word choice would have a higher perplexity.

Question 39

BERT

Answer 39

BERT employs an encoder-only architecture. The decision to use an encoder-only architecture in BERT suggests a primary emphasis on understanding input sequences rather than generating output sequences.

Traditional language models process text sequentially, either from left to right or right to left. This method limits the model’s awareness to the immediate context preceding the target word. BERT uses a bi-directional approach considering both the left and right context of words in a sentence, instead of analyzing the text sequentially, BERT looks at all the words in a sentence simultaneously.

Question 40

BERT undergoes two steps:
Pre-training & Fine tuning

Explain Pre-training

Answer 40

During pre training, BERT is trained on large amount of unlabeled data to learn contextual embeddings. It has two primary objective:

1. Masked Language Modelling (MLM)
2. Next Sentence Prediction

The model aims to minimize the combined loss function of the Masked LM and Next Sentence Prediction

Question 41

BERT undergoes two steps:
Pre-training & Fine tuning

Explain Fine tuning

Answer 41

BERT is fine-tuned using labeled data specific to the downstream tasks of interest like sentiment analysis, question-answering, named entity recognition.

Question 42

Masked Language Model (MLM)

Answer 42

In BERT’s pre-training process, a portion of words (around 15%) in each input sequence is masked (with a special symbol like [Mask]) and the model is trained to predict the original values of these masked words based on the context provided by the surrounding words.

Question 43

Next Sentence Prediction (NSP)

Answer 43

In the pre-training process, BERT learns to understand the relationship between pairs of sentences, predicting if the second sentence follows the first in the original document.
50% of the input pairs have the second sentence as the subsequent sentence in the original document, and the other 50% have a randomly chosen sentence.

Question 44

Why to train MLM and NSP together?

Answer 44

MLM helps BERT to understand the context within a sentence and NSP helps BERT grasp the connection or relationship between pairs of sentences.
Hence, training both the strategies together ensures that BERT learns a broad and comprehensive understanding of language, capturing both details within sentences and the flow between sentences.

Question 45

BERT architecture

Answer 45

The architecture of BERT is a multilayer bidirectional transformer encoder. A transformer architecture is an encoder-decoder network that uses self-attention on the encoder side and attention on the decoder side.

1. BERTBASE has 12 layers in the Encoder stack while BERTLARGE has 24 layers in the Encoder stack. These are more than the Transformer architecture described in the original paper (6 encoder layers).
2. BERT architectures (BASE and LARGE) also have larger feedforward networks (768 and 1024 hidden units respectively), and more attention heads (12 and 16 respectively) than the original Transformer architecture (512 hidden units and 8 attention heads).
3. BERTBASE contains 110M parameters while BERTLARGE has 340M parameters.