# 3

Question 1

token

Answer 1

every word in a corpus ( punctuation as well)

ex. Are you on campus today? # 5 tokens (6 with ?)

Question 2

type

Answer 2

every distinct word in a corpus

ex. I will eat what you eat, even if I am not hungry.
# 10 ( eat and eat are one type, I’s as well)

Question 3

lemma

Answer 3

words with the same root
ex. I have never eaten that before, but I will eat it.
# 9 (eat and eaten has the same lemma)

Question 4

lemmatization

Answer 4

the process of reducing each word-form in a corpus

is -> be
went -> go
best -> good

Question 5

Stem

Answer 5

the base form of a word that carries out its meaning

ex. talked -> talk

Question 6

what is the stem of reading?

Answer 6

read

Question 7

affix

Answer 7

a morpheme added to the a stem to modify its meaning

work + ed = worked

Question 8

what is the affix of unhappiness

Answer 8

stem: happy
affixes: “un”, “ness”

Question 9

derivational affixes

Answer 9

create new words by changing grammatical category or meaning

happy + un[not] = unhappy

Question 10

Inflectional affixes

Answer 10

encode numbers, tenses, aspects of a word

talk + ed [past] = talked

Question 11

combining stems together to produce new meaning?

Answer 11

compounds

Question 12

example of a compound?

Answer 12

fire -> fighter

Question 13

what entails normalization

Answer 13

discarding variability

Question 14

Disjunction of good and Good

Answer 14

/[gG]ood/