Chapter 1

Question 1

Load all data in NLP ?

Answer 1

>>> from nltk.book import *

Question 2

How do yo find out about ant texts in NLP ?

Answer 2

just type text1

Question 3

What is Concordance :?

Answer 3

An alphabetical list of the words (especially the important ones) present in a text or texts, usually with citations of the passages concerned or with the context displayed on a computer screen. ‘a concordance to the Bible’

Question 4

What is concordance in Python?

Answer 4

.concordance() is a method in the Text class of nltk Basically, if you want to use the .concordance(), you have to instantiate a Text object first, and then call it on that object

Question 5

A concordance permits us ?

Answer 5

A concordance permits us to see words in context. For example, we saw that mon- strous occurred in contexts such as the ___ pictures and the ___ size.

Question 6

What other words appear in a similar range of contexts?

Answer 6

We can find out by appending the term similar to the name of the text in question, then inserting the relevant word in parentheses: >>> text1.similar(“monstrous”)

Question 7

What is Common_context?

Answer 7

The term common_contexts allows us to examine just the contexts that are shared by two or more words, such as monstrous and very. text2.common_contexts([“monstrous”, “very”be_glad am_glad a_pretty is_pretty a_lucky

Question 8

how to find location of word in a text?

Answer 8

Can be done using dispersion plot E.g >>> text4.dispersion_plot([“citizens”, “democracy”, “freedom”, “duties”, “America”])

Question 9

How to Generate Random Text?

Answer 9

The method generate random text from the provided corpus. E.g here text3 . text3.generate()

Question 10

The generate() method is not available in NLTK 3.0? why

Answer 10

but will be reinstated in a subsequent version.

Question 11

is generate truly random?

Answer 11

Although the text is random, it reuses common words and phrases from the source text and gives us a sense of its style and content

Question 12

What is a token?

Answer 12

A token is the technical name for a sequence of characters — such as hairy, his, or :)

Question 13

how to get length of text?

Answer 13

len(text3) : We use the term len to get the length of something,

Question 14

What is vocabulary of a text?

Answer 14

The vocabulary of a text is just the set of tokens that it uses, since in a set, all duplicates are collapsed together.

Question 15

What is python command to find vocabulary of tex?

Answer 15

In Python we can obtain the vocabulary items of text3 with the command: set(text3)

Question 16

what is word type?

Answer 16

A word type is the form or spelling of the word independently of its specific occurrences in a text — that is, the word considered as a unique item of vocabulary. Also called distinct word

Question 17

How to calculate a measure of the lexical richness of the text?

Answer 17

□ example shows us that the number of distinct words is just 6% of the total number of words, or equivalently that each word is used 16 times on average len(set(text3)) / len(text3) 0.06230453042623537

Question 18

Count on particular word?

Answer 18

Count how often a word occur and the percentage of word in text text3.count(“smote”) 5 >>> 100 * text4.count(‘a’) / len(text4) 1.464301643393831

Question 19

How to create function?

Answer 19

def lexical_diversity(text): … return len(set(text)) / len(text)

Question 20

…. in python during function?

Answer 20

The Python interpreter changes the prompt from >>> to … after encountering the colon at the end of the first line. The … prompt indicates that Python expects an indented code block to appear next. It is up to you to do the indentation, by typing four spaces or hitting the tab key. To finish the indented block just enter a blank line.

Question 21

List operation? how to append , concatenate ?

Answer 21

sent1.append(“Some”) + = concatenation

Question 22

Python how to do indexing ? two types what are they ?

Answer 22

1. i. Indexing By position >>> text4[173] By valuetext4.index(‘awaken’)

Question 23

What is Slicing?

Answer 23

Tacking some part of list

Question 24

How to do slicing?

Answer 24

1) text5[16715:16735] 2) Text5[start : last] 3) By convention, m:n means elements m…n-1 4) we can omit the first number if the slice begins at the start of the list , and we can omit the second number if the slice goes to the endE.g sent[:3] [‘word1’, ‘word2’, ‘word2’] 5) text2[141525:] …will report upto the last item in the list

Question 25

How to update a list?

Answer 25

sent[0] = ‘First sent[1:9] = [‘Second’, ‘Third’]

Question 26

In sorting what is diff btw capitalize words and lowercase?

Answer 26

• Remember that capitalized words appear before lowercase words in sorted lists.

Question 27

Can python expression be splitted into several lines?

Answer 27

Python expressions can be split across multiple lines, so long as this happens within any kind of brackets. Python uses the “…” prompt to indicate that more input is expected. It doesn’t matter how much indentation is used in these continuation lines, but some indentation usually makes them easier to read.

Question 28

Strings

Answer 28

same method use with list can be use wth strings. for example : >>> name = ‘Monty’ >>> name[0] ‘M’ >>> name[:4] ‘Mont’

Question 29

How can we do addition and mult with strings?

Answer 29

>>> name * 2 ‘MontyMonty’ >>> name + ‘!’ ‘Monty!’ >>>

Question 30

How can we do addition and mult with strings?

Answer 30

>>> name * 2 ‘MontyMonty’ >>> name + ‘!’ ‘Monty!’ >>>

Question 31

what is join and split function as use in string?

Answer 31

>>> ‘ ‘.join([‘Monty’, ‘Python’]) ‘Monty Python’ >>> ‘Monty Python’.split() [‘Monty’, ‘Python’]

Question 32

Whatis frequency distribution?

Answer 32

§ It is a “distribution” because it tells us how the total number of word tokens in the text are distributed across the vocabulary items Since we often need frequency distributions in language processing, NLTK provides built-in support for them.

Question 33

what is function for finding frequency dist?

Answer 33

fdist1 = FreqDist(text1)

Question 34

what is most_commmon function?

Answer 34

fdist1.most_common(50) return 50 most common §

Question 35

Word that appear only once?

Answer 35

Which function give use word fdist1.hapaxes().. Words that appear only Once.

Question 36

How to produce commulative frequency graph

Answer 36

What proportion of the text is taken up with such words? We can generate a cumulative frequency plot for these words, using fdist1.plot(50, cumulative=True), to produce the graph.

Question 37

What is “the set of all w such that w is an element of V (the vocabulary) and w has property P”.

Answer 37

{w | w ∈ V & P(w)} Set notation [w for w in V if p(w)] in Python , This is a set , so it may contain duplicate

Question 38

Example of using set repr

Answer 38

V = set(text1) long_words = [w for w in V if len(w) > 15]

Question 39

Example of words from the chat corpus that are longer than seven characters, that occur more than seven times:

Answer 39

fdist5 = FreqDist(text5) >>> sorted(w for w in set(text5) if len(w) > 7 and fdist5[w]> 7) Or >>> R = [w for w in set(text5) if len(w)> 7 and fdist5[w] > 7]

Question 40

What is collocation?

Answer 40

A collocation is a sequence of words that occur together unusually often. Thus red wine is a collocation, whereas the wine is not

Question 41

example of using collocation function

Answer 41

text4.collocations()

Question 42

What is the essence of collocation?

Answer 42

§ The collocations that may emerge are very specific to the genre of the texts.

Question 43

Functions by NTLK for frequency distribution

Answer 43

ddddd

Question 44

Word comparison operator

Answer 44

Question 45

[w for w in text if condition]

.

Answer 45

We can also create more complex conditions. If c is a condition, then not c is also a condition. If we have two conditions c1 and c2, then we can combine them to form a new condition using conjunction and disjunction: c1 and c2, c1 or c2

Question 46

How to Operate on Every Element?

Answer 46

[len(w) for w in text1] , [w.upper() for w in text1],

○ These expressions have the form [f(w) for …] or [w.f() for …], where f is a function that operates on a word to compute its length, or to convert it to uppercase.
○ The notation just described is called a “list comprehension

Question 47

What is relation between nexted block and extra blank line?

Answer 47

When we use the Python interpreter we have to add an extra blank line
in order for it to detect that the nested block is complete.

Question 48

Pyhon control; structure end with what? and why?

Answer 48

all Python control structures end with a colon. The colon indicates that the current statement relates to the indented block that follows.

Question 49

Automatic Natural Language Understanding challenges?

Answer 49

○ On a more philosophical level, a long-standing challenge within artificial intelligence has been to build intelligent machines, and a major part of intelligent behaviour is understanding language. For many years this goal has been seen as too difficult. However, as NLP technologies become more mature, and robust methods for analyzing unrestricted text become more widespread, the prospect of natural language understanding has re-emerged as a plausible goal.

Question 51

What is Word Sense Disambiguation?

Answer 51

In word sense disambiguation we want to work out which sense of a word was intended in a given context

Question 52

How to automatically disambiguate words ?

Answer 52

We automatically disambiguate words using context, exploiting the simple fact that nearby words have closely related meaning?

Question 53

Example

Answer 53

xample:

a. The lost children were found by the searchers (agentive)
b. The lost children were found by the mountain (locative)
c. The lost children were found by the afternoon (temporal)

Question 54

What is Pronoun Resolution?

Answer 54

A deeper kind of language understanding is to work out “who did what to whom” — i.e., to detect the subjects and objects of verbs. it is difficult example(one case is ambiguous):

The thieves stole the paintings. They were subsequently sold.

b. The thieves stole the paintings. They were subsequently caught.
c. The thieves stole the paintings. They were subsequently found.

Answering this question involves finding the antecedent of the pronoun they, either thieves or paintings. Computational techniques for tackling this problem include anaphora resolution — identifying what a pronoun or noun phrase refers to — and semantic role labeling — identifying how a noun phrase relates to the verb (as agent, patient, instrument, and so on)

Question 55

Why Machine Translation is difficult?

Answer 55

For a long time now, machine translation (MT) has been the holy grail of language understanding.

Machine translation is difficult because a given word could have several possible translations (depending on its meaning), and because word order must be changed in keeping with the grammatical structure of the target language. Today these difficulties are being faced by collecting massive quantities of parallel texts from news and government websites that publish documents in two or more languages. Given a document in German and English, and possibly a bilingual dictionary, we can automatically pair up the sentences, a process called text alignment. Once we have a million or more sentence pairs, we can detect corresponding words and phrases, and build a model that can be used for translating new text.

Question 56

What is chief measure of intelligence in history of AI?

Answer 56

In the history of artificial intelligence, the chief measure of intelligence has been a linguistic one, namely the Turing Test: can a dialogue system, responding to a user’s text input, perform so naturally that we cannot distinguish it from a human-generated response?

Question 57

Limitations of NLP?

Answer 57

Despite the research-led advances in tasks like RTE, natural language systems that have been deployed for real-world applications still cannot perform common-sense reasoning or draw on world knowledge in a general and robust manner

Question 58

Chapter 1 Summary

Answer 58

• Texts are represented in Python using lists: [‘Monty’, ‘Python’]. We can use indexing, slicing, and the len() function on lists.
• A word “token” is a particular appearance of a given word in a text; a word “type” is the unique form of the word as a particular sequence of letters. We count word tokens using len(text) and word types using len(set(text)).
• We obtain the vocabulary of a text t using sorted(set(t)).
• We operate on each item of a text using [f(x) for x in text].
• To derive the vocabulary, collapsing case distinctions and ignoring punctuation, we can write set(w.lower() for w in text if w.isalpha()).
• We process each word in a text using a for statement, such as for w in t: or for word in text:. This must be followed by the colon character and an indented block of code, to be executed each time through the loop.
• We test a condition using an if statement: if len(word) < 5:. This must be followed by the colon character and an indented block of code, to be executed only if the condition is true.
• A frequency distribution is a collection of items along with their frequency counts (e.g., the words of a text and their frequency of appearance).
• A function is a block of code that has been assigned a name and can be reused. Functions are defined using the def keyword, as indef mult(x, y); x and y are parameters of the function, and act as placeholders for actual data values.
• A function is called by specifying its name followed by zero or more arguments inside parentheses, like this: texts(), mult(3, 4), len(text1).

Question 59

What is collocation?

Answer 59

When you eat at a quick-service restaurant, you are eating fast food. You wouldn’t say you went and got ‘quick food.’ That is because fast food is a collocation, or a pair or set of words that are commonly put together. In a collocation, if you replace one of the words with a synonym, it sounds unnatural to native English speakers