Text Process Basics 1.2

Question 1

What is tokenization?

Answer 1

Tokens correspond to words and numeric

sequences separated by white-space characters or punctuation.

Question 2

What is sentence tokenization?

Answer 2

For long documents, we may not be interested in words but instead in
sentences therein:
• Check whether a sentence’s sentiment is positive or negative
• Check whether a sentence contains propaganda content
• Check the grammatical correctness of a sentence
• etc.

This is known as sentence tokenization.

Question 3

What is Stemming and give an example of it?

Answer 3

Stemming is the process of reducing inflection in words to their root
forms, such as mapping a group of words to the same stem, even if
the stem itself is not a valid word in the language.

E.g. Before to befor or sanitize to sanit

Question 4

When a language contains words that are derived from another word
as their use in the speech changes what is it called?

Answer 4

Inflected Language - Stemming and Lemmatization help to find the root forms from the inflected words.

Question 5

How are stems created?

Answer 5

Stems are created by removing the suffixes or prefixes used with a word.

Question 6

What are NLTK’s main stemmers?

Answer 6

Porter Stemmer and
Lancaster Stemmer (a.k.a Paice-Husk Stemmer) (more aggressive)

Question 7

What is Lemmatization?

Answer 7

Lemmatization reduces the inflected words
properly ensuring that the root word belongs to the language.
In Lemmatization the root word is called Lemma.
A lemma (plural lemmas or lemmata) is the canonical form, dictionary form, or citation form of a set of words.

Question 8

Pros and Cons of Stemming?

Answer 8

Pro: Quick to run, because it is based on simple rules; suitable for processing a large amount of text
Con: Output can be meaningless; no one-fits-all rule

Question 9

Pros and Cons of Lemmatization?

Answer 9

Pro: The derived root word is meaningful
Con: more expensive (and hence slow) to run

Question 10

What are stopwords?

Answer 10

Stop words, such as “am”, “the”, “to” and “are”, support words and sentences.
They help us to construct grammatical sentences, but they mostly do not affect the meaning of the sentence
NLTK has a stopwords list; we use it to find and remove the stopwords

Question 11

What are the typical text cleaning steps?

Answer 11

Raw text > Tokenization > Stopwords Removal > Stemming or Lemmatization > Normalized Text

Question 12

Give an example of when you should not perform Stemming or Lemmatization?

Answer 12

If you need to check the whether a word is used in the singular or plural form, you should not perform stemming or lemmatization