Lecture 2

Question 1

Text Foundations:
Symbolic Encoding Systems

Answer 1

From a sociolinguistic perspective: any
symbolically encoded language is considered to be text; symbols come in many types

Question 2

Text Foundations:
Alphabetic

Answer 2

symbols (letters) to represent vowels and
consonants in words
(e.g., Latin)

Question 3

Text Foundations:
Abjads

Answer 3

symbols to represent consonants, with
diacritics (or reused consonants) to represent vowels
(e.g., Arabic, Hebrew)

Question 4

Text Foundations:
Abugida/Syllabary

Answer 4

Symbol systems representing
consonants plus inherent vowels
(e.g., Devanagari)

Question 5

Text Foundations:
Semanto -phonetic

Answer 5

Symbols that carry packets of
meaning and/or sounds
(e.g., Chinese)

Question 6

Tokenization (AKA word segmentation)

Answer 6

Separate the characters into individual words

Question 7

Tokenization:
Recognize and deal with punctuation

Answer 7

*Apostrophes (one word
it s vs. two words it s
*Hyphenation (snow laden)
*Periods (keep with abbrev. or separate as sentence markers)

Question 8

Morphology (To stem or not to stem?)

Answer 8

How words are formed through processes such as affixation (adding prefixes or suffixes)

Question 9

Corpus Linguistics

Answer 9

Following tokenization of a “body ” of text, we can develop a summary analysis the linguistic equivalent of descriptive statistics.

Question 10

Corpus Linguistics: Methods 1

Answer 10

The Corpus is an ensemble of text

Question 11

Corpus Linguistics: Methods 2

Answer 11

Summary and detailed statistical
analysis

Question 12

Corpus Linguistics: Methods 3

Answer 12

Before summarizing: Filter out
junk

Question 13

Corpus Linguistics: Methods 4

Answer 13

Normalization issues -
Ignore capitalization at beginning
of sentence? Is They the same
word as they

Question 14

Corpus Linguistics: Methods 5

Answer 14

Ignore other capitalization? In a
name such as Unilever Corporation is Corporation the same word as corporation

Question 15

Corpus Structures

Answer 15

isolated (scattered)
Categorized (boxes)
overlapping (Venn diagram)
Temporal (over time)

Question 16

Terminology for word occurrences:

Answer 16

Tokens
the total number of words

Distinct Tokens (sometimes called word types) the number of distinct words, not counting repetitions sometimes called
vocabulary, notated as |V|

Question 17

Frequency distribution

Answer 17

a list of all tokens with their frequency, usually sorted in the order of decreasing
frequency

Question 18

Zipf’s Law (1949)

Answer 18

In a natural language corpus, the frequency of any word is inversely proportional
to its rank in a frequency table

eg
* Most frequent word (rank = 1) is twice as frequent as 2nd most frequent
* Most frequent (rank = 1) is 3 times as frequent as 3rd most frequent, etc.

Question 19

• Rank (r)

Answer 19

The numerical position of a word in a list sorted by decreasing frequency
(f ).

Question 20

Stopwords

Answer 20

(commonly occurring words such as
“the” will account for a large fraction of text so eliminating them greatly reduces the number of words in a text

Question 21

Unigrams, Bigrams, Trigrams, Etc.

Answer 21

an nGram is simply a token that spans two or more words

Question 22

frequency of an n gram

Answer 22

the percentage of times the n gram occurs in all the n grams of the corpus and could be useful in corpus statistics

Question 23

Bigram frequency

Answer 23

percentage occurrence of the bigram in
the corpus

Question 24

Mutual information

Answer 24

how the two words/tokens are
associated with each other

Question 25

Statistic: Pointwise Mutual Information (PMI)

Answer 25

• Given a pair of words, compares
  probability that the two occur together
  as a joint event to the probability they
  occur individually & that their co
  occurrences are simply the result of
  chance
• The more strongly connected 2
  items are, the higher will be their
  MI value

Question 26

Regular Expressions

Answer 26

Regular expressions are a miniature programming language for pattern matching. Sometimes people refer to the topic as RegEx

Question 27

Chomsky’s Hierarchy of Languages -1

Answer 27

The greatest complexity is represented by the outer circle no restrictions on how grammar can operate

Question 28

Chomsky’s Hierarchy of Languages -2

Answer 28

Next, a context-sensitive grammar
covers most human languages
grammar rules are complex and varied
parsing is difficult but theoretically
possible

Question 29

Chomsky’s Hierarchy of Languages -3

Answer 29

Context free languages include most
programming languages like C or
Python these have clear rules and
can always be parsed

Question 30

Chomsky’s Hierarchy of Languages -4

Answer 30

Regular grammar is the most
simplistic at all regular expression
languages fit this category