Week 8: Clustering and Text Mining Flashcards
1
Q
What are other types of embeddings in R, after word2vec?
A
GloVe, fastText
2
Q
What type of Gaussian model mixture is k-means?
A
all prior class proportions 1/K, EII model, all posteriors are either 0 or 1
3
Q
What are embeddings?
A
Short, dense vectors that can be used to represent words?
4
Q
What is the intuition of skip-gram/ word2vec?
A
- Treat the target word and a neighbouring context word as positive examples
- Randomly sample other words in the lexicon to get negative samples
- Use logistic regression to train a classifier to distinguish those two cases
- Use the learned weights as the embeddings
5
Q
What is case folding?
A
a kind of normalisation that is mapping everything to lowercase
6
Q
What is the purpose of tf-idf?
A
Raw frequency is not the best measure of association between words because words like “the” and “good” occur frequently and aren’t informative. tf-idf gives weight to words that appear in fewer documents
7
Q
What are some popular lexicons for sentiment analysis? (3)
A
AFINN, NRC, bing
8
Q
Code to turn a document term matrix into a dataframe in R?
A
tidy()
9
Q
How do you calculate tf, idf and tf-idf?
A
term frequency: the frequency of the word t in document d / the total number of terms in the document
tf(t,d)= count(t,d)/total number of terms in the document
inverse document frequency: number of documents / number of documents the term occurs in
idf = log10(N/df(t))
tf-idf = tf*idf
10
Q
How do you create a vocabulary of words in R?
A
it = itoken(wordvector) #create index tokens full\_vocab = create\_vocabulary(it) #create full vocabulary
11
Q
What is stemming?
A
naive version of morphological analysis that consists of chopping of word-final affixes
12
Q
What is a term-document matrix?
A
- each row represents a word in the vocabulary
- each column represents a document
- each cell represents the number of times a particular word occurs in a particular document
13
Q
What assumption does skip-gram make?
A
all context words are independent
14
Q
How do you create a token co-occurrence matrix in R?
A
create_tcm(it, vectorizer, skip_grams_window = 5)
15
Q
How does the skip-gram algorithm adjust during training?
A
adjust initial embeddings to maximise the similarity (dot product) of the (w, cpos) pairs drawn from the positive examples and minimise the similarity (dot product) of the (w, cneg) pairs from the negative examples
16
Q
What are other names for LCA and LPA
A
LCA = binomial mixture model LPA = gaussian mixture model
17
Q
What does the likelihood tell us?
A
How well the model fits the data
18
Q
Code to create a corpus in R?
A
Corpus(textsource)
19
Q
How do you use grep to match and count matches from data? (3 ways)
A
grep(“expression”, data) #returns indices of all matching expressions from data
grep(“expression”, data, value = TRUE) #returns all the actual matching expressions from data. gives whole string, not just matching part
length(grep(“expression”, data)) #returns number of matches of expression in data
20
Q
What is the assumption of gaussian mixture modelling?
A
The data within each cluster is normally distributed
21
Q
What are vector semantics for words?
A
represent a word as a point in multidimensional semantic space that is derived from the distributions of word neighbours
22
Q
What are the 2 things that ^ represents in RE
A
Caret ^ matches the start of a line
or negates the contents of [] if it is [^…]
23
Q
How do you use regexpr and gregexpr and regmatches to match data?
A
r = regexpr(“expression”, data) #gives index of each match and length of each match (only for first match of each index)
r = gregexpr(“expression”, data) #gives index of each match and length of each match (all matches)
regmatches(data, r) #returns the actual matches obtained from regexpr or gregexpr
24
Q
What is lemmatisation?
A
the task of determining that two words have the same root, despite their surface differences
25
What are the terms of tf-idf
```
tf = term frequency
idf = inverse document frequency
```
26
What are some forms of text representation? (4)
- Bag-of-words: word count or word proportion for each word
- Time-series: label each token and put in order
- Tf-idf, embeddings
27
What is a document-term matrix (DTM)?
Each row is a document, each column is a word
28
What are 3 types of text normalisation?
- tokenising (segmenting) words
- normalising word formats
- segmenting sentences
29
What is the distributional hypothesis?
words that occur in similar contexts tend to have similar meanings
30
What is the kleene \* and knleene + in RE?
\* zero or more occurrences of the immediately previous character or regular expression
+ one or more occurrences of the immediately previous character or regular expression
31
Code to create a document term matrix in R?
DocumentTermMatrix(corpus)
32
What is a clitic?
A clitic is a part of a word that can’t stand on its own, can only occur attached to another word. E.g. we’re
33
What is gaussian mixture modelling trying to find?
Hidden groups within the data that are not recorded
34
What is Zipf's law?
the frequency that a word appears is inversely proportional to its rank
35
What is information retrieval?
the task of finding the document d from the D documents in some collection that best matches a query q
36
What does the value of the cosine metric represent?
The cosine value ranges from 1 for vectors pointing in the same direction, through 0 for orthogonal vectors, to -1 for opposite vectors
37
How do you specify a range in RE?
"-" e.g. [2-5]
38
Code for implementing mclust in R?
Mclust(data, G=2, modelNames = “E”)
39
What is a regular expression?
an algebraic notion for characterising a set of strings
40
What type of vector representations come from a term-term matrix?
Sparse because most are 0
41
What is word normalisation?
the task of putting words/tokens in a standard format
42
How do you calculate the cosine similarity in R?
sin2(matrix1, matrix2, method=“cosine”, norm=“l2”)
43
What are \b and \B in RE?
\b matches word boundary
\B matches non-word boundary
44
What is the precedence hierarchy in RE?
1. Parenthesis ()
2. Counters \* + ? {}
3. Sequences and anchors ^ $
4. Disjunction |
45
What do {n} and {m,n} mean in RE?
n occurrences of the previous char or expression
from n to m occurrences of the previous char or expression
46
What is the classifier to train for skip-grams?
train a classifier such that a given tuple (w,c) of a target word w paired with a candidate/context word c, it will return the probability that c is a real context word P(+|w,c)
47
What is a token learner and a token segmenter?
- A token learner takes a raw training corpus and induces a vocabulary, a set of tokens
- A token segmenter takes a raw test sentence and segments it into the tokens in the vocabulary
48
What does Penn Treebank tokenisation do?
It separates out clitics (doesn’t becomes does n’t), keeps hyphenated words together, separates out all punctuatio
49
What are two properties of word embeddings?
- analogies: look at the analogies in vector space el.g. King - man + woman = queen
- Bias: semantics derived automatically from language corps contain human-like biases
50
In multivariate model based clustering with 2 observed features, what do the mean and standard deviation become?
Mean becomes a vector of 2 means
Standard deviation becomes a 2x2 variance covariance matrix determining the shape of the cluster
51
What is the likelihood, defined by the statistical model and the assumption?
p(data|paramaters) = p(y|theta)
52
When you have fitted an Mclust model to fit\_mclust in R, what information can you obtain from the result?
fit\_mclust$paramaters #gives means, variances, proportions, modelName
fit\_mclust$bic #gives the bic for each option and shows which models have the lowest. These are negative in the mclust package, take absolute value
fit\_mclust$loglik #gives log likelihood used to calculate bic manually
fit\_mclust$classification #gives the cluster classification vector
fit\_mclust$uncertainty #gives uncertainty of each point
plot(fit\_mclust, “density”) #gives density plot
53
What is ? in RE?
the preceding character or nothing
e.g. colou?r
54
What is inflectional stemming and stemming to root?
- Inflectional stemming: remove plurals, normalise verb tenses, remove other affixes
- Stemming to root: reduce word to most basic element
55
How can you tokenise text in R?
library(tidytext)
unnest\_tokens(data, outputcolumn, inputcolumn) #takes one term per row and automatically removes punctuation
56
What is an advantage of word2vec/skip grams?
Self-supervised learning. Don't need labels
57
What is the most sophisticated way of lemmatisation?
complete morphological parsing of the word
ie. takes cats and parses it into the two morphemes cat and s
58
What is the period . in RE?
A wildcard expression that matches any single character
59
What is the basis of information criteria (BIC) formula?
BIC = -2.log(l) + m.log(n)
```
l = likelihood = p(data|theta)
-2.log(l) = deviance = reconstruction loss = fit
m = number of parameters
n = number of observations/examples
m.log(n) = file size = complexity
```
60
What are the stringr options for matching strings?
str\_view(“string", “expression”) #only matches the first expression
str\_detect(“string”, “expression”) #returns true/false depending on whether the string matches the expression
str\_extract(“string”, “expression”) #extracts the first match
str\_extract\_all(“string”, “expression”) #extracts all matches into a vector
str\_match\_all(“string”, “expression”) #similar to str\_extract\_all except output is matrix with column for each
61
What are the problems with sentiment analysis? (4)
- Not great with longer tasks
- negation
- context-dependency
- You need partly labelled data
62
What do LCA and LPA stand for? what are they types of?
```
latent class analysis
latent profile analysis
types of model based clustering
```
63
What is morphology?
Morphology is how words are build up from stems, the central morpheme of the word and affixes
64
What is a corpus and a lexicon?
Corpus = a collection of documents (our whole dataset)
Lexicon: set of all unique words in a corpus
65
How does the byte-pair encoding algorithm work for tokenisation?
- it is a token learner
- starts with vocabulary of all characters
- chooses the two symbols that are most frequently adjacent, adds a new merged symbol to the vocabulary and replaces every adjacent pair in the corpus with the new merged symbol
- continues to count and merge, creating new longer and longer character strings, until k merges have been done creating k novel tokens
66
What are the steps of the expectation maximisation (EM) algorithm?
0. Guess the parameters
1. Work out the posterior probability of being M/F assuming normality (E step)
2. Update the parameters (M step)
\* repeat steps 1 and 2 until parameters stop changing
67
What are \d and \D in RE?
\d is any digit
\D is any non-digit
68
What is the probability that c is a context word P(+|w,c)?
P(+|w,c) = c.w = 1/1+exp(-c.w)
where . is the dot product
69
What is $ in RE?
matches the end of a line
70
What is the overall sentiment in sentiment analysis?
Sentiment = total positive words - total negative words
71
What is the basic workflow for text analysis? (5 steps)
1. Get some text
2. Organise text into corpus
3. Pre-process
4. Create representation
5. Perform analysis as usual
72
What is sentiment analysis?
task of classifying the polarity of a given text (ie is it a good/bad review)
73
When will two column vectors be similar in a term-document matrix?
When the documents contain similar words
74
What is tokenisation?
Tokenisation is the task of segmenting running text into words
75
What are some typical steps in text pre-processing? (8)
- Stemming (running -\> run)
- Lemmatisation (were -\> is)
- lower casing
- Stop word removal
- Punctuation removal
- Number removal
- Spell correction
- Tokenisation
76
What is the formula for the cosine similarity measure?
cosine(v,w) = (v.w)/(|v||w|) (where . is dot product)
77
How can you remove stop words in R?
anti\_join(data, stop\_words) #removes stop words on an unnest\_tokens object
78
Where does Zipf's law often deviate?
There are often deviations at high ranks, as a corpus often contains fewer rare words than predicted by a single power law
79
What are \n and \t in RE?
\n is newline
\t is tab
80
What does [] mean in RE?
anything within the brackets can be included e.g. colo[ou]r means colour or color
81
When will two row vectors be similar in a term-document matrix?
Similar rows mean that the words are similar because they occur in similar documents
82
How does mclust select a model in R?
mclust fits all the models with up to specified number of clusters, computes the BIC of each model and chooses the model with the best BIC
83
what does data %\>% pull(variable) do in R?
same as data$variable
84
How does latent class analysis work?
LCA lets the variables follow any distribution, as long as they are unrelated to each other (independent) within classes.
85
How do you get the dot product of 2 vectors?
v.w = v1w1 + v2w2 + … + vNwN
86
What are the number of parameters in a multivariate Gaussian mixture model:
pi(k,X): number of classes -1 : K-1
mu(k): K\*p (p is number of features)
var: K\*p (or just p when variances are equal over classes)
covariances: K\*p\*(p-1)/2 (or p\*(p-1)/2 when covariances are equal over classes) (or 0 when variables are uncorrelated, spherical clusters)
m = (K-1) + Kp + Kp + Kp(p-1)/2
87
What do sub and gsub do?
#replace matching expression with replacement
sub(“expression”, “replacement” data) (only first match of each index)
gsub(“expression”, “replacement” data) (all matches)
88
What is the overall probability curve made up of?
The weighted sum of two normal curves
89
How do you map words to indices in R? ie create a vectorizer
vectorizer = vocab\_vectorizer(small\_vocab)
90
What is sentence segmentation?
segmenting a text into sentences
91
What is a term-term matrix or word co-occurrance matrix?
Columns and rows both represent words
- each cell records the number of times the row (target) word and the column (context) word co-occur in some context in some training corpus
- the context could be a document or smaller such as a window around the word
92
What are all the parameters learned in skip-gram?
two matrices W and C each containing an embedding for every one of the |V| words in the vocabulary V
93
How do you do an escape in R?
\\
94
How do you get the tf-idf in R?
bind\_tf\_idf(tidytext dataset, tokens, documents, counts) #takes a tidytext dataset as input with one row per token per document
95
What are 3 alternatives to the BIC?
AIC: Akaike information criterion - same as BIC but penalty is m
AIC3: same as AIC but penalty is 3m/2
ICL: Integrated information criterion - same as BIC but reconstruction loss includes the assigned clusters
96
How can you say or in RE?
pipe |
97
What is the gaussian mixture model formula?
```
p(x) = pi(1,x) Normal(mean1, var1) + (1-pi(1,x)) Normal(mean2, var2)
where pi(1,x) is the probability that variable x takes on value 1 (e.g. probability that person is man)
ie proportion of values expected in each cluster
```
98
What are the identifiers for each parameterisation of a GMM and what to they measure?
E for equal, V for variable, I for identity matrix
- Volume (size of clusters in data space)
- Shape (circle or ellipse)
- Orientation (the angle of the ellipse)
E.g. VVE model has variable volume, variable shape, equal orientation
99
What does reducing the error rate of an RE involve?
- Increasing precision: minimising false positives (strings that were incorrectly matched)
- Increasing recall: minimising false negatives (strings that were incorrectly missed)
100
How do you get part of a string and reference back to that part in an RE?
Use capture group to store the expression in memory
the (.\*)er they were, the \1er they will be
101
How can you identify clusters that are not ellipses using the GMM?
start with the usual Gaussian mixture solution, merge similar components to create non-Gaussian clusters
102
How is the posterior probability calculated based on the current estimates of mean and sd?
pi(man,x) = point on male curve/point on total curve
103
What is the aim of the BIC?
tradeoff between complexity and file size
the lower the better
104
What is the code for merging components of clusters in R?
clustCombi(data=x)
