Chapter2

Question 1

What is the topic for chapter2

Answer 1

Question 2

What is text corpus?

Answer 2

a text corpus is a large body of text. Many corpora are designed to contain a careful balance of material in one or more genres

Question 3

What is Gutenberg Corpus?

Answer 3

○ NLTK includes a small selection of texts from the Project Gutenberg electronic text archive, which contains some 25,000 free electronic books, hosted at http://www.gutenberg.org/

Question 4

Importing words from NTLK using diff python import

Answer 4

import nltk >>> nltk.corpus.gutenberg.fileids() >>> emma = nltk.corpus.gutenberg.words(‘austen-emma.txt’) (Let’s pick out the first of these texts which is emma.txt)

Question 5

Python provides another version of the import statement, as follows?

Answer 5

>>> from nltk.corpus import gutenberg >>> gutenberg.fileids() [‘austen-emma.txt’, ‘austen-persuasion.txt’, ‘austen-sense.txt’, …] >>> emma = gutenberg.words(‘austen-emma.txt’)

Question 6

write program to display other information about each text, by looping over all the values of fileid corresponding to the gutenberg file and compute statistics.?

Answer 6

>>> for fileid in gutenberg.fileids(): … num_chars = len(gutenberg.raw(fileid)) … num_words = len(gutenberg.words(fileid)) … num_sents = len(gutenberg.sents(fileid)) … num_vocab = len(set(w.lower() for w in gutenberg.words(fileid))) … print(round(num_chars/num_words), round(num_words/num_sents), round(num_words/num_vocab), fileid) 5 25 26 austen-emma.txt 5 26 17 austen-persuasion.txt 5 28 22 austen-sense.txt 4 34 79 bible-kjv.txt 5 19 5 blake-poems.txt 4 19 14 bryant-stories.txt ○ This program displays three statistics for each text: average word length, average sentence length, and the number of times each vocabulary item appears in the text on average (our lexical diversity score). Observe that average word length appears to be a general property of English, since it has a recurrent value of 4. (In fact, the average word length is really 3 not 4, since the num_chars variable counts space characters.) By contrast average sentence length and lexical diversity appear to be characteristics of particular authors.

Question 7

What does row() function does?

Answer 7

The raw() function gives us the contents of the file without any linguistic processing.or example, len(gutenberg.raw(‘blake-poems.txt’)) tells us how many letters occur in the text, including the spaces between words

Question 8

What is sent() function ?

Answer 8

The sents() function divides the text up into its sentences, where each sentence is a list of words:

Question 9

What is Brown corpus?

Answer 9

The Brown Corpus was the first million-word electronic corpus of English, created in 1961 at Brown University. This corpus contains text from 500 sources, and the sources have been categorized by genre, such as news, editorial, and so on . b. We can access the corpus as a list of words, or a list of sentences (where each sentence is itself just a list of words). We can optionally specify particular categories or files to read:

Question 10

how to find categories at Brown corpus?

Answer 10

>>> from nltk.corpus import brown >>> brown.categories()

Question 11

In which case Brown university corpus important?

Answer 11

ii. The Brown Corpus is a convenient resource for studying systematic differences between genres, a kind of linguistic inquiry known as stylistics.

Question 12

WRITE program to compare genres in their usage of modal verbs

Answer 12

>>> from nltk.corpus import brown >>> news_text = brown.words(categories=’news’) >>> fdist = nltk.FreqDist(w.lower() for w in news_text) >>> modals = [‘can’, ‘could’, ‘may’, ‘might’, ‘must’, ‘will’] >>> for m in modals: … print(m + ‘:’, fdist[m], end=’ ‘) … can: 94 could: 87 may: 93 might: 38 must: 53 will: 389 We need to include end=’ ‘ in order for the print function to put its output on a single line.

Question 13

If you want use FreqDistr u need to import what?

Answer 13

from nltk import FreqDist

Question 14

What is Reuters Corpus ?

Answer 14

The Reuters Corpus contains 10,788 news documents totaling 1.3 million words. b. The documents have been classified into 90 topics, and grouped into two sets, called “training” and “test”; thus, the text with fileid ‘test/14826’ is a document drawn from the test set.

Question 15

Inaugural Address Corpus

Answer 15

we looked at the Inaugural Address Corpus, but treated it as a single text . However, the corpus is actually a collection of 55 texts, one for each presidential address. An interesting property of this collection is its time dimension:

Question 16

Example Inaugural corpus

Answer 16

>> from nltk.corpus import inaugural >>> inaugural.fileids() [‘1789-Washington.txt’, ‘1793-Washington.txt’, ‘1797-Adams.txt’, …] >>> [fileid[:4] for fileid in inaugural.fileids()] [‘1789’, ‘1793’, ‘1797’, ‘1801’, ‘1805’, ‘1809’, ‘1813’, ‘1817’, ‘1821’, …]

Question 17

Annotated Text Corpora for teaching and learning

Answer 17

Many text corpora contain linguistic annotations, representing POS tags, named entities, syntactic structures, semantic roles, and so forth. NLTK provides convenient ways to access several of these corpora, and has data packages containing corpora and corpus samples, freely downloadable for use in teaching and research

Question 18

Some of the corpora

Answer 18

1. Movie Reviews Pang, Lee 2k movie reviews with sentiment polarity classification SentiWordNet Esuli, Sebastiani sentiment scores for 145k WordNet synonym sets Wordlist Corpus OpenOffice.org et al 960k words and 20k affixes for 8 languages WordNet 3.0 (English) Miller, Fellbaum 145k synonym sets

Question 19

Loading your own Corpus

Answer 19

f you have your own collection of text files that you would like to access using the above methods, you can easily load them with the help of NLTK’s PlaintextCorpusReader

3. The second parameter of the PlaintextCorpusReader initializer
   can be a list of fileids, like [‘a.txt’, ‘test/b.txt’], or a pattern that matches all fileids, like ‘[abc]/.*.txt’

Question 20

Example of importing file

Answer 20

>>> from nltk.corpus import PlaintextCorpusReader >>> corpus_root = ‘/usr/share/dict’

>>> wordlists = PlaintextCorpusReader(corpus_root, ‘.*’)

>>> wordlists.fileids() [‘README’, ‘connectives’, ‘propernames’, ‘web2’, ‘web2a’, ‘words’] >>> wordlists.words(‘connectives’) [‘the’, ‘of’, ‘and’, ‘to’, ‘a’, ‘in’, ‘that’, ‘is’, …]

Question 21

Generating Random Text with Bigrams Eaxample?

Answer 21

The bigrams() function takes a list of words and builds a list of consecutive word pairs. Remember that, in order to see the result and not a cryptic “generator object”, we need to use the list()function:

>>> import nltk
>>> sent = [‘In’, ‘the’, ‘beginning’, ‘God’, ‘created’, ‘the’, ‘heaven’]
>>> list(nltk.bigrams(sent))
[(‘In’, ‘the’), (‘the’, ‘beginning’), (‘beginning’, ‘God’), (‘God’, ‘created’), (‘created’, ‘the’), (‘the’, ‘heaven’)]
>>>

Question 22

Advise on testing code with an editor?

Answer 22

• It is often convenient to test your ideas using the interpreter, revising a line of code until it does what you expect
Once you’re ready, you can paste the code (minus any >>> or … prompts) into the text editor, continue to expand it, and finally save the program in a file so that you don’t have to type it in again later

Question 23

How to name file in python?

Answer 23

Give the file a short but descriptive name, using all lowercase letters and separating words with underscore

Question 24

Functions

Answer 24

A function is just a named block of code that performs some well-defined task

• We define a function using the keyword def followed by the function name and any input parameters, followed by the body of the functio

Question 25

Simple function of Plural ?

Answer 25

def plural(word):

if word.endswith(‘y’): return word[:-1] + ‘ies’ elif word[-1] in ‘sx’ or word[-2:] in [‘sh’, ‘ch’]: return word + ‘es’ elif word.endswith(‘an’): return word[:-2] + ‘en’ else: return word + ‘s
® >>> plural(‘fairy’) ‘fairies’ >>> plural(‘woman’) ‘women’

Question 26

endswith() method or function

Answer 26

• The endswith() function is always associated with a string object (e.g., word in 3.1). To call such functions, we give the name of the object, a period, and then the name of the function. These functions are usually known as methods.

Question 27

What is module?

Answer 27

• A collection of variable and function definitions in a file is called a Python module.

Question 28

What is Package ?

Answer 28

• A collection of related modules is called a package.
NLTK’s code for processing the Brown Corpus is an example of a module, and its collection of code for processing all the different corpora is an example of a package. NLTK itself is a set of packages, sometimes called a library.

Question 29

What is lexicon?

Answer 29

A lexicon, or lexical resource, is a collection of words and/or phrases along with associated information such as part of speech and sense definitions

Question 30

What is lexical resource?

Answer 30

○ Lexical resources are secondary to texts, and are usually created and enriched with the help of texts. For example, if we have defined a text my_text, then vocab = sorted(set(my_text)) builds the vocabulary of my_text, while word_freq = FreqDist(my_text)counts the frequency of each word in the text. Both of vocab and word_freq are simple lexical resources

Question 31

What is lexical entry?

Answer 31

○ A lexical entry consists of a headword (also known as a lemma) along with additional information such as the part of speech and the sense definition. Two distinct words having the same spelling are called homonyms.

Question 33

What is lexical enrty? explain diagramatically

Answer 33

Question 34

Wordlist Corpora

Answer 34

NLTK includes some corpora that are nothing more than wordlists. The Words Corpus is the /usr/share/dict/words file from Unix, used by some spell checkers. We can use it to find unusual or mis-spelt words in a text corpus

Question 35

Stop word corpus

Answer 35

There is also a corpus of stopwords, that is, high-frequency words like the, to and also that we sometimes want to filter out of a document before further processing.

Stopwords usually have little lexical content, and their presence in a text fails to distinguish it from other texts.

Example: ○ >>> from nltk.corpus import stopwords >>> stopwords.words(‘english’) [‘i’, ‘me’, ‘my’, ‘myself’, ‘we’, ‘our’, ‘ours’, ‘ourselves’, ‘you’, ‘your’, ‘yours’, ‘yourself’, ‘yourselves’, ‘he’, ‘him’, ‘his’, ‘himself’, ‘she’, ‘her’, ‘hers’, ‘herself’, ‘it’, ‘its’, ‘itself’, ‘they’, ‘them’, ‘their’, ‘theirs’, ‘themselves’, ‘what’, ‘which’, ‘who’, ‘whom’, ‘this’, ‘that’, ‘these’, ‘those’, ‘am’, ‘is’, ‘are’, ‘was’, ‘were’, ‘be’, ‘been’, ‘being’, ‘have’, ‘has’, ‘had’, ‘having’, ‘do’, ‘does’, ‘did’, ‘doing’, ‘a’, ‘an’, ‘the’, ‘and’, ‘but’, ‘if’, ‘or’, ‘because’, ‘as’, ‘until’, ‘while’, ‘of’, ‘at’, ‘by’, ‘for’, ‘with’, ‘about’, ‘against’, ‘between’, ‘into’, ‘through’, ‘during’, ‘before’, ‘after’, ‘above’, ‘below’, ‘to’, ‘from’, ‘up’, ‘down’, ‘in’, ‘out’, ‘on’, ‘off’, ‘over’, ‘under’, ‘again’, ‘further’, ‘then’, ‘once’, ‘here’, ‘there’, ‘when’, ‘where’, ‘why’, ‘how’, ‘all’, ‘any’, ‘both’, ‘each’, ‘few’, ‘more’, ‘most’, ‘other’, ‘some’, ‘such’, ‘no’, ‘nor’, ‘not’, ‘only’, ‘own’, ‘same’, ‘so’, ‘than’, ‘too’, ‘very’, ‘s’, ‘t’, ‘can’, ‘will’, ‘just’, ‘don’, ‘should’, ‘now’]

Question 36

define a function to compute what fraction of words in a text are not in the stopwords list:

Answer 36

>>> def content_fraction(text): …

stopwords = nltk.corpus.stopwords.words(‘english’) … content = [w for w in text if w.lower() not in stopwords] … return len(content) / len(text) … >>> content_fraction(nltk.corpus.reuters.words()) 0.7364374824583169

Question 37

Names corpus,

Answer 37

One more wordlist corpus is the Names corpus, containing 8,000 first names categorized by gender. The male and female names are stored in separate file

Question 38

Let’s find names which appear in both files, i.e. names that are ambiguous for gender:?

Answer 38

>>> names = nltk.corpus.names >>> names.fileids() [‘female.txt’, ‘male.txt’] >>> male_names = names.words(‘male.txt’) >>> female_names = names.words(‘female.txt’) >>> [w for w in male_names if w in female_names] [‘Abbey’, ‘Abbie’, ‘Abby’, ‘Addie’, ‘Adrian’, ‘Adrien’, ‘Ajay’, ‘Alex’, ‘Alexis’, ‘Alfie’, ‘Ali’, ‘Alix’, ‘Allie’, ‘Allyn’, ‘Andie’, ‘Andrea’, ‘Andy’, ‘Angel’, ‘Angie’, ‘Ariel’, ‘Ashley’, ‘Aubrey’, ‘Augustine’, ‘Austin’, ‘Averil’, …]

It is well known that names ending in the letter a are almost always female.Remember that name[-1] is the last letter of name below

Question 39

Comparative Wordlists

Answer 39

• NLTK includes so-called Swadesh wordlists, lists of about 200 common words in several languages. The languages are identified using an ISO 639 two-letter code.
example

○ >>> from nltk.corpus import swadesh >>> swadesh.fileids() [‘be’, ‘bg’, ‘bs’, ‘ca’, ‘cs’, ‘cu’, ‘de’, ‘en’, ‘es’, ‘fr’, ‘hr’, ‘it’, ‘la’, ‘mk’, ‘nl’, ‘pl’, ‘pt’, ‘ro’, ‘ru’, ‘sk’, ‘sl’, ‘sr’, ‘sw’, ‘uk’] >>> swadesh.words(‘en’) [‘I’, ‘you (singular), thou’, ‘he’, ‘we’, ‘you (plural)’, ‘they’, ‘this’, ‘that’, ‘here’, ‘there’, ‘who’, ‘what’, ‘where’, ‘when’, ‘how’, ‘not’, ‘all’, ‘many’, ‘some’, ‘few’, ‘other’, ‘one’, ‘two’, ‘three’, ‘four’, ‘five’, ‘big’, ‘long’, ‘wide’, …]

Question 40

What is Shoebox and Toolbox Lexicons?

Answer 40

○ Perhaps the single most popular tool used by linguists for managing data is Toolbox, previously known as Shoebox since it replaces the field linguist’s traditional shoebox full of file cards. Toolbox is freely downloadable from http://www.sil.org/computing/toolbox/.

Question 41

WordNet

Answer 41

○ WordNet is a semantically-oriented dictionary of English, similar to a traditional thesaurus but with a richer structure. NLTK includes the English WordNet, with 155,287 words and 117,659 synonym sets. We’ll begin by looking at synonyms and how they are accessed in WordNet.

Question 42

Summary

Answer 42

• A text corpus is a large, structured collection of texts. NLTK comes with many corpora, e.g., the Brown Corpus, nltk.corpus.brown.
• Some text corpora are categorized, e.g., by genre or topic; sometimes the categories of a corpus overlap each other.
• A conditional frequency distribution is a collection of frequency distributions, each one for a different condition. They can be used for counting word frequencies, given a context or a genre.
• Python programs more than a few lines long should be entered using a text editor, saved to a file with a .py extension, and accessed using an import statement.
• Python functions permit you to associate a name with a particular block of code, and re-use that code as often as necessary.
• Some functions, known as “methods”, are associated with an object and we give the object name followed by a period followed by the function, like this: x.funct(y), e.g., word.isalpha().
• To find out about some variable v, type help(v) in the Python interactive interpreter to read the help entry for this kind of object.
• WordNet is a semantically-oriented dictionary of English, consisting of synonym sets — or synsets — and organized into a network.
Some functions are not available by default, but must be accessed using Python’s import statement

Question 43

Set () what it is used in python?

Answer 43

To find unique words

Question 44

Why do we normalize text and how to do it?

Answer 44

Always normalize text using w.lower because using All and all will be counted diff in text processing

Question 45

Can you combine functions with object using . ? how?

Answer 45

len(set(a.lower().split(‘ ‘)))

Question 46

What are string operation?

Answer 46

s. lower() , s.upper() , s.titlecase()
s. split(t) , s.joint(t) , s.strip(t) ,s.rstrip() , s.find(t) , s.rfind(s) , s.replace(u,v)

Question 47

Two ways to get character from word?

Answer 47

1. using list function
2. using [w for w in word]

Question 48

If you have text with space and tabs, how can u get exact list of words in the text without those tabs and space?

Answer 48

use strip function first to remove the space and tabs then use split function to split the text with space

The strip() method returns a copy of the string with both leading and trailing characters removed (based on the string argument passed).

The strip() removes characters from both left and right based on the argument (a string specifying the set of characters to be removed).

The syntax of strip() is:

string.strip([chars])

strip() Parameters
chars (optional) - a string specifying the set of characters to be removed.
If the chars argument is not provided, all leading and trailing whitespaces are removed from the string

Question 49

w.find() , w.rfind() , w.replace( old, new)

Answer 49

All this find string , find string in reverse and find and replace string

Question 50

File operations

Answer 50

f = open(filename, mode)

f.readline() , f.read() , f.read(n)

for line in f : do something(line)

f. skeek()
f. write()
f. close()
f. closed to check if a particular file is really close.

Question 51

What is on eprobelm with reading text ? hw to dealt with it?

Answer 51

Question 52

How to create mutiple lines in python?

Answer 52

1. s = “”” this is a very
   long string if I had the
   energy to type more and more …”””

but will have(\n) at end this is a very\n long string if I had the\n energy to type more and more …’

2. s = (“this is a very”
   “long string too”
   “for sure …”
   )

This will not include extra space

3.

Breaking lines by \ works for me.

longStr = “This is a very long string “ \
“that I wrote to help somebody “ \
“who had a question about “ \
“writing long strings in Python”

4.

string = “"”This is a
very long string,
containing commas,
that I split up
for readability””“.replace(‘\n’, ‘ ‘ )

Question 53

Find a word that start with # in senetence?tweet = “@nltk Text analysis is awesome! #regex #pandas #python

Answer 53

print([word for word in tweet.split() if word.startswith(‘#’)])

Question 54

Advise on using text

Answer 54

Advice on using text. Always split ur text into words before working on each words

Question 55

How to find specific word in python?

Answer 55

[w for w in (knowntext) if condition(on w)]

Question 56

In regular expression what is the function of [] , [‘abc] , a|b,\d , \D

Answer 56

[] means matches one of the character inside,

[‘abc] …matches any charcter that is not abc

a|b…matches a or b

\d means any digit [0-9]

\D any non digit [^0-9]

Question 57

What is \s , \S , \w

Answer 57

\s….means any whitespace

\S any no-whitespace

\w…….means alpha numerice character

\W ..non alpha numeric character

Question 58

Metacharacter repitions ?

Answer 58

* : matches zero or more occurance

+ : matches one or more occurances

?: matches zero or one more occurance

{n} exactly n repitions

{n , } atleast n repition

{,n} at most n repition

{m,n} atleast n repition and atmost n repition

Question 60

Example of searching string

Answer 60

a = “usman @gwammaja @bayan masqa akwai spa”c

In [72]: [w for w in b.split() if re.search(‘@[A-Za-z0-9]+’,w)]
Out[72]: [‘@gwammaja’, ‘@bayan’]

Using Alpha numeric character ?

[w for w in b.split() if re.search(‘@\w+’,w)]
Out[73]: [‘@gwammaja’, ‘@bayan’]

Question 61

What is r infront in regular expression?

Answer 61

The ‘r’ in front tells Python the expression is a raw string. In a raw string, escape sequences are not parsed. For example, ‘\n’ is a single newline character. But, r’\n’ would be two characters: a backslash and an ‘n’.

Question 62

re.search() , re.march() , re.

Answer 62

re. match(), re.search(), re.findall() with Example
https: //www.guru99.com/python-regular-expressions-complete-tutorial.html

Question 63

live as if your were to die tomorrow , learn……

Answer 63

“Live as if you were to die tomorrow. Learn as if you were to live forever,” as Mahatma Gandhi

Question 64

How mant text corpus do we have in NLTK?

How do we access the sentences in a particular text? forxample snetence na 13

Answer 64

8 , sent2 , sent13

Question 65

how to calculate first ten words in sentence?

Answer 65

list(sent(sent7))[:10]

Question 66

Why do we call list sequence type?

Answer 66

Since lists in Python store ordered collections of items or objects, we can say that they are sequence types, exactly because they behave like a sequence. Other types that are also considered to be sequence types are strings and tuples.

Question 67

What is the relationship between sequence type and iterable?

Answer 67

You might wonder what’s so special about sequence types. Well, in simple words, it means that the program can iterate over them! This is why lists, strings, tuples, and sets are called “iterables”.

Question 68

Lists Versus Tuples? difference, when to use one?

Answer 68

Tuples are used to collect an immutable ordered list of elements. This means that:

You can’t add elements to a tuple. There’s no append() or extend() method for tuples,
You can’t remove elements from a tuple. Tuples have no remove() or pop() method,
You can find elements in a tuple since this doesn’t change the tuple.
You can also use the in operator to check if an element exists in the tuple.

So, if you’re defining a constant set of values and all you’re going to do with it is iterate through it, use a tuple instead of a list. It will be faster than working with lists and also safer, as the tuples contain “write-protect” data.

Question 69

List vs Dictionary?

Answer 69

A list stores an ordered collection of items, so it keeps some order. Dictionaries don’t have any order.
Dictionaries are known to associate each key with a value, while lists just contain values.

Use a dictionary when you have an unordered set of unique keys that map to values.

Note that, because you have keys and values that link to each other, the performance will be better than lists in cases where you’re checking membership of an element.

Question 70

Lists Versus Sets

Answer 70

Just like dictionaries, sets have no order in their collection of items. Not like lists.
Set requires the items contained in it to be hashable, lists store non-hashable items.
Sets require your items to be unique and immutable. Duplicates are not allowed in sets, while lists allow for duplicates and are mutable.

You should make use of sets when you have an unordered set of unique, immutable values that are hashable.

You aren’t sure which values are hashable?

Question 71

What this method does in python? dict.keys()

Answer 71

This method returns a list of all the available keys in the dictionary.

Question 73

How to import help for NLTK tagset explainning part of speech?

Answer 73

nltk.help.upenn_tagset()

Question 74

hypernym and hyponym?

Answer 74

In simpler terms, a hyponym is in a type-of relationship with its hypernym. For example, pigeon, crow, eagle and seagull are all hyponyms of bird (their hypernym); which, in turn, is a hyponym of animal.

Hyponymy shows the relationship between a generic term (hypernym) and a specific instance of it (hyponym). A hyponym is a word or phrase whose semantic field is more specific than its hypernym. The semantic field of a hypernym, also known as a superordinate, is broader than that of a hyponym

Question 75

what is etymology of the word computer?

Answer 75

The Online Etymology Dictionary states that the use of the term to mean “calculating machine” (of any type) is from 1897.”

Question 76

List comprehension using not in

Answer 76

para = “This is my message to you shamsudd I will come to your house tomorrow”

words = word_tokenize(para)
print(words)
useful_words = [word for word in words if word not in stopwords.words(‘english’)]
print(useful_words)