lecture 3

Question 1

corpus

Answer 1

1. large/complete collection of writing
2. linguistics. a body of utterances, as words or sentences, assumed to be representative of and used for lexical, grammatical, or other linguistic analysis

Question 2

properties of corpora

Answer 2

1. naturally occurring corpora serve as realistic samples of language
2. metadata: side information about where the samples come from
3. corpora with linguistic annotations

Question 3

markup (formatting)

Answer 3

common formats for structuring linguistic data

–> e.g., XML, CoNLL, JSON

Question 4

preprocessing

Answer 4

transforming text into a useful format

–> e.g., tokenization

Question 5

splitting sentences into words/tokens

Answer 5

1. using spaces as delimiters
2. tokenization
3. stemming
4. lemmatization

Question 6

tokenization

Answer 6

each word, including punctuation, is considered a separate token and assigned a sequential position

Question 7

stemming + process

Answer 7

consider words within sentences that share a common root, but appear in different forms

reduce word to base/root form
–> this enables us to treat them as a single unit for analysis

process:
1. tokenize sentences
2. use stemming to reduce words into common stems

Question 8

lemmatization

Answer 8

reduce words to common lemma’s

takes context into account to convert words to their base dictionary form

results in more linguistically correct forms than stemming

Question 9

bag of words

Answer 9

text is represented by the frequency of its words, disregarding grammar and word order

considers their occurrences as features for analysis

simplifies text into numerical data

Question 10

BOW process

Answer 10

1. text is converted to list of words
2. words are placed into metaphorical bag
3. each word’s frequency is counted
4. frequency count is represented in table format

Question 11

n-gram

Answer 11

• word sequence of length n
• the more words, the more information is revealed
• compute probabilities of these sequences

Question 12

n-gram probabilities

Answer 12

calculation: occurrence of the sequence / size of the corpus

words in a corpus could appear multiple times. this affects the calculation of probabilities and the overall model we create

it is therefore essential that n-grams model the probability distribution of all possible sequences of all tokens in the corpus

Question 13

what do n-gram models evaluate

Answer 13

the probability of a sequence of n tokens

this is useful for determining the likelihood of specific word sequences and is represented by the joint probability

Question 14

chain rule

Answer 14

product of conditional probabilities

indicates that the probability of the entire sequence is the product of the individual probabilities of each word given all the preceding words

this allows us to systematically compute the probability of a sequence by considering one word at a time, given the context of all previous words in the sequence