09. Text Analytics

Question 1

What is Lemmatisation

Answer 1

Lemmatisation – reduces the inflected words by finding the correct dictionary base/root word that belongs to the language.
In Lemmatisation, root word is called Lemma.
For example, runs, running, ran are all forms of the word run, therefore run is the lemma of all these words.

Question 2

What is stemming

Answer 2

Stemming is the process of reducing inflection in words to their root forms such as mapping a group of words to the same stem even if the stem itself is not a valid word in the Language.
Stem (root) is the part of the word to which you add inflectional (changing/deriving) affixes such as (-ed, -ize, -s, -de, mis).
Stems are created by removing the suffixes or prefixes used with a word, which is called Suffix/Prefix Stripping.

Sometimes called porters stemming

Question 3

What is the dimensionality in text analysis

Answer 3

It is the number of unique terms in the document. Various methods try to reduce this dimensionality to make the analysis simpler.

Question 4

What is case folding

Answer 4

It means you ignore the difference between capitals and standard text

Question 5

What does tokenizing doing

Answer 5

Tokenization is the task of separating (also called tokenizing) words from the body of text. Raw text is converted into collections of tokens after the tokenization, where each token is generally a word.

You need to define how you wish to break the text apart by i.e. punctuation

Question 6

What does parsing mean

Answer 6

Parsing: reading an unstructured text and converting it into a formatted data. This normally involves adding structure to the data.

Question 7

In text analysis what is meant by search and retrieval

Answer 7

Search and retrieval: search specific words/phrases, topics or entities like names of people and organisations into documents in a corpus.

Question 8

What is text mining

Answer 8

Text mining: this involves applying analysis methods to discover relationships and patterns in large text collections

Question 9

What is topic modelling

Answer 9

A topic consists of a cluster of words that frequently occur together and share the same theme. i.e. fluffy, meeow, purr, paw = the topic of cats. You need to refer to a corpus which would contain pre-labelled topics.

Question 10

What does RSS mean

Answer 10

Real Simple Syndication

Question 11

What are Regular Expressions

Answer 11

A method for defining parameters used for text mining i.e. $ is the symbol used to indicate the end of a text string

Question 12

What is Zipf’s Law

Answer 12

Vaguely holds true the ith word occurs 1/ith word

1st ranked = 1/1, 2nd ranked = 1/2, 3rd ranked = 1/3

Question 13

What is case folding

Answer 13

It ignores the capital letters / lower case detail of text

Question 14

What is information content of words

Answer 14

“Stop” words have basically no information content (i.e. the, and etc) these should be removed to improve text analysis

Question 15

What is TF

Answer 15

Term frequency
TF1(t,d)=SUMf(t,ti)
It is a count of the number of times that term appears in the corpus of documents

Question 16

What is the IDF

Answer 16

Inverted Document Frequency

The document frequency is the number of documents in the corpus that contain the term, hence

The inverted document frequency is the inverse of that

Question 17

What is the TFIDF

Answer 17

TFIDF (t,d) = TF(t,d) x IDF(t)

If this is higher the better. I high number means that this word is an important word.

Question 18

What is sentiment analysis

Answer 18

Looking for opinions, often uses classifiers (niave bayes) and often has a binary result)

Question 19

What is a word cloud

Answer 19

An image of the words found in a document with the more common words being bigger. First having removed the stop words.

Question 20

What is part of speech tagging (POS)

Answer 20

Changing the words out to the corresponding noun verb etc

Question 21

List some common regular expressions

Answer 21

means or
*matches zero or more instances of the previous letter
+ matches one or more instances of the previous letter
{2,4} matches two to four instances of the previous letter
^ means starts with
$means ends with

Question 22

What does bag of words mean

Answer 22

All the words in the text but order of words is not preserved