B03 Text Analytics II

Question 1

Text Pre-Processing Steps

Answer 1

-Tokenize Text -Normalize Spelling -Remove Stop Words -Stem Words -Normalize case -Tag Parts of Speech -Detect Sentence Boundaries

Question 2

Pre-Processing Definitions: Tokenization

Answer 2

-This involves breaking up text data into individual units or tokens. In other words, creating a bag of words from a document. -Tokens are most often a single word, but can also be an n-gram, sentence, or paragraph.

Question 3

Pre-Processing Definitions: n-gram

Answer 3

An n-gram is a contiguous sequence of n words within text. “The cow jumped over the moon;” “the cow” “cow jumped” “jumped over” “over the” “the moon”

Question 4

Pre-Processing Definitions: Part-Of-Speech (POS) Tagging

Answer 4

-This involves labeling a word in text as belonging to a particular part of speech such as nouns, verbs, adjectives, etc. -Part-of-speech tagging is also known as ‘grammatical tagging’ or ‘word-category disambiguation’.

Question 5

Pre-Processing Definitions: Stop Words

Answer 5

-This involves removing words that occur a lot but have very little analytic impact. -Common examples of stop words include: “a”, “an”, “and”, “but”, “by”, “if”, “it”, “that”, “the”, etc.

Question 6

Pre-Processing Definitions: Stemming

Answer 6

-The process of normalizing related word tokens into a single form. -This typically includes the identification and removal of prefixes, suffixes, and inappropriate pluralizations.

Question 7

Pre-Processing Definitions: Lemmatization

Answer 7

-A more advanced form of stemming that attempts to group words based on their core concept or lemma. -It uses both the context surrounding the word and additional grammatical information such as part of speech to determine the lemma.

Question 8

Pre-Processing Definitions: Spelling Normalization

Answer 8

This involves resolving spelling mistakes or eliminating spelling variations. Approaches include using: 1. A dictionary-based approach. 2. Fuzzy matching algorithms. 3. Word clustering and concept expansion techniques.

Question 9

Pre-Processing Definitions: Sentence Boundary Detection

Answer 9

-The process of breaking down entire documents into individual grammatical sentences. -For English text, it is almost as easy as finding every occurrence of punctuations like “.”; “?”; or “!”.

Question 10

Pre-Processing Definitions: Case Normalization

Answer 10

-This involves converting the entire document to either completely lower case or completely upper case characters. -While mixed case text may be helpful to humans in order to differentiate between nouns and proper nouns, they are not always useful for algorithms.

Question 11

Define Vector-space Model

Answer 11

-In the vector-space model, the rows represent documents and the columns represent tokens (vectors). -The elements of the model represent the occurrence of tokens within the text. -A commonly used vector-space model in text analytics is the document-term-matrix.

Question 12

Define the ‘bag-of-words assumption’

Answer 12

The vector-space model makes an implicit assumption that the order of words (or tokens) in a document does not matter. This is known as the bag-of-words assumption.

Question 13

The Vector-space Model can take on 3 forms, these are:

Answer 13

1. binary representation 2. frequency count 3. float-valued weighted vector.

Question 14

Ex. Binary Representation

Answer 14

Question 15

Ex. Frequency Count

Answer 15

Question 16

Define Weighted Vector in the Context of Vector Space Model

Answer 16

-Beyond simply using word frequency, the perceived
importance of the word is sometimes also considered and
applied as a weight.
-This requires choosing a weighting scheme.
-One of the most popular schemes is the tf-idf weighting
approach.

Question 17

Explain TF-IDF

Answer 17

-TF stands for term frequency. It is the proportion of times a
term appears in a document.
-IDF stands for inverse document frequency. The document
frequency for a term is the proportion of documents in
which that term appears. The inverse of this is the .

Question 18

Inverse document frequency is calculated as:

Answer 18

Question 19

The TF-IDF for a for a term is calculated as:

Answer 19

Question 20

Ex. of Weighted Vector

Answer 20

Question 21

The Tidy Text Format

The tidy data format requires that:

Answer 21

• Each variable is a column
• Each observation is a row
• Each type of observational unit is a table

Applied to text data, this requires that tidy text tables have
one token per row.

Question 22

The Tidy Text Format

Tidy data sets allow data to be
manipulated with a standard set
of “tidy” tools and functions.

For instance:

Answer 22

• dplyr
• broom
• ggplot2
• tidyr

Question 23

The TidyText Package

Answer 23

• Developed by Julia Silge and David Robinson.
• Provides functions and supporting data sets for text analysis.
• Provides support to tidy objects from other popular text mining packages such as tm and quanteda.

Question 24

What does unnest_tokens() function do?

Answer 24

• Removes punctuations.
• Removes whitespace.
• Converts text to lowercase.
• Puts a single token per row (default is one word per row).

Question 25

What does the anti_join() function do?

Answer 25

-Returns all records from table 1 where there are no matching
values in table 2.
-Keeps only columns from table 1.
-Used to remove stop words.

Question 26

What does bind_tf_idf() function do?

Answer 26

-Binds the tf_idf of a tidytext dataset to the original data.
-The dataset must have more than one document and it must
have one row per document-term combination.