NLP Python Tutorial

Question 1

Function to download nltk corpora

Answer 1

nltk.download()

Question 2

Import to tokenize sentences

Answer 2

from nltk.tokenize import sent_tokenize

Question 3

Import to tokenize words

Answer 3

from nltk.tokenize import word_tokenize

Question 4

Function to tokenize sentences

Answer 4

print(sent_tokenize())

Question 5

Function to tokenize sentences example

Answer 5

print(sent_tokenize(text))

Question 6

Function to tokenize words

Answer 6

print(word_tokenize())

Question 7

Function to tokenize words example

Answer 7

print(word_tokenize(text))

Question 8

Import for stop words

Answer 8

from nltk.corpus import stopwords

Question 9

Turning stop words into a variable

Answer 9

stop_words = set(stopwords.words(‘english’))

Question 10

Turning tokenized text into a variable

Answer 10

word_tokens = word_tokenize()

Question 11

Turning tokenized text into a variable example

Answer 11

word_tokens = word_tokenize(example)

Question 12

Function for removing stop words

Answer 12

sentence = [w for w in word_tokens if not w in stop_words]

Question 13

Stemming import

Answer 13

from nltk.stem import PorterStemmer

Question 14

Turning stemmer into a variable

Answer 14

ps = PorterStemmer()

Question 15

Function for stemming

Answer 15

for w in:

print(ps.stem(w))

Question 16

Function for stemming example

Answer 16

for w in example_words:

print(ps.stem(w))

Question 17

Stemming an entire sentence steps

Answer 17

1. Tokenize

2. Stem

Question 18

Stemming an entire sentence example

Answer 18

words = word_tokenize(new_text)

for w in words:
print(ps.stem(w))

Question 19

Importing an nltk text

Answer 19

from nltk.corpus import

Question 20

Importing an nltk text example

Answer 20

from nltk.corpus import udhr

Question 21

Import for the PunktSentenceTokenizer

Answer 21

from nltk.tokenize import PunktSentenceTokenizer

Question 22

Training the PunktSentenceTokenizer

Answer 22

custom_sent_tokenizer = PunktSentenceTokenizer(train_text)

Question 23

Run sample text through PunktSentenceTokenizer

Answer 23

tokenized = custom_sent_tokenizer.tokenize(sample_text)

Question 24

Tag each tokenized word with a part of speech steps

Answer 24

1. For loop
2. Tokenize words
3. Tag words
4. Print tagged words

Question 25

Tag each tokenized word with a part of speech example

Answer 25

for i in tokenized[:5]: words = nltk.word_tokenize(i) tagged = nltk.pos_tag(words) print(tagged)

Question 26

Chunking 3 steps

Answer 26

1. chunkgram = 2. chunkParser = 3. chunked =

Question 27

chunkGram

Answer 27

chunkGram = r”””Chunk: { }”””

Question 28

chunkGram example

Answer 28

chunkGram = r”””Chunk: {**+?}”””

Question 29

chunkParser

Answer 29

chunkParser = nltk.RegexpParser()

Question 30

chunkParser example

Answer 30

chunkParser = nltk.RegexpParser(chunkGram)

Question 31

chunked

Answer 31

chunked = chunkParser.parse(tagged)

Question 32

Print the nltk tree

Answer 32

for subtree in chunked.subtrees(filter=lambda t: t.label() == ‘Chunk’): print(subtree)

Question 33

Draw chunks with nltk

Answer 33

chunked.draw()

Question 34

namedEnt definition

Answer 34

Marks proper nouns as organizations or people or money, etc

Question 35

namedEnt steps

Answer 35

1. Tokenize 2. pos Tagging 3. namedEnt

Question 36

namedEnt example

Answer 36

namedEnt = nltk.ne_chunk(tagged, binary=false)

Question 37

chunkGram for chinking

Answer 37

chunkGram = r”””Chunk: {+} | }+{“”””

Question 38

Build a list of documents

Answer 38

documents = [list( .words(fileid)), category) for category in .categories() for fileid in .fileids(category)]

Question 39

Build a list of documents example

Answer 39

documents = [list(movie_reviews.words(fileid)), category) for category in movie_reviews.categories() for fileid in movie_reviews.fileids(category)]

Question 40

Shuffle the documents

Answer 40

random.shuffle(documents)

Question 41

Print the number of documents

Answer 41

print(‘Number of Documents {}’.format(len(documents)))

Question 42

Print the first review

Answer 42

print(‘First Review: {}’.format(documents[0]))

Question 43

Write all the words in the reviews

Answer 43

all_words = [] for w in movie_reviews.words(): all_words.append(w.lower())

Question 44

Create a FreqDist

Answer 44

all_words = nltk.FreqDist(all_words)

Question 45

Find the most common words

Answer 45

print(‘Most common words: {}’.format(all_words.most_common(15)))

Question 46

Find how many times the word “happy” is used

Answer 46

print(‘the word happy: {}’.format(all_words[“happy”]))

Question 47

Find how many words are in the text

Answer 47

print(len(all_words))

Question 48

Use the 4000 most common words as features

Answer 48

word_features = list(all_words.keys())[:4000]

Question 49

Find features steps

Answer 49

1. define function 2. prep 3. for loop 4. return features

Question 50

Define find features

Answer 50

def find_features(document):

Question 51

Find features prep

Answer 51

``` words = set(document) features = {} ```

Question 52

Find features for loop

Answer 52

for w in word_features: | features[w] = (w in words)

Question 53

Find features example steps

Answer 53

1. prep 2. for loop 3. print(features)

Question 54

Find features example prep

Answer 54

features = find_features(.words(‘ ’))

Question 55

Find features example prep example

Answer 55

features = find_features(movie_reviews.words(‘neg/cv000_29416.txt’))

Question 56

Find features example for loop steps

Answer 56

1. for loop 2. if clause 3. print(keys)

Question 57

Find features example for loop

Answer 57

for key, value in features.items():

Question 58

Find features example if clause

Answer 58

if value == True: | print(key)

Question 59

Find features for all documents

Answer 59

featuresets = [(find_features(rev), category) for (rev, category) in documents]

Question 60

Import model selection

Answer 60

from sklearn import model_selection

Question 61

Define a seed for reproducibility

Answer 61

seed = 1

Question 62

Split featuresets into training and testing datasets

Answer 62

training, testing = model_selection.train_test_split(featuresets, test_size = 0.25, random_state = seed)

Question 63

Import sklearn classifier

Answer 63

from nltk.classify.scikitlearn import SklearnClassifier

Question 64

import SVC

Answer 64

from sklearn.svm import SVC

Question 65

Define the SVC model

Answer 65

model = SklearnClassifier(SVC(kernel = ‘linear’))

Question 66

Train the model on the training data

Answer 66

model.train(training)

Question 67

Find the accuracy of the SVC model steps

Answer 67

1. define accuracy | 2. print

Question 68

Define accuracy

Answer 68

accuracy = nltk.classify.accuracy(model, testing)

Question 69

Print accuracy

Answer 69

print(‘SVC Accuracy: {}’.format(accuracy))