Topic 1: Regular Expression

Question 1

key components of RE

Answer 1

search, string, pattern, corpus

Question 2

what is regular expression

Answer 2

language for specifying text search.

expression used to specify a set of strings required for a particular purpose

Question 3

what is string?

Answer 3

sequence of symbols.

in text based search, string is a sequence of alphanumeric character

Question 4

what is pattern?

Answer 4

a specific sequence of character/symbols. useful in RE for text searching

Question 5

a regular expression search require 2 things. what is it?

Answer 5

pattern to search..corpus (text to search through)

Question 6

what are the application of regular expressions?

Answer 6

1. test for a pattern within a string.
2. use in database for selecting data
3. substitution

Question 7

what are the basic patterns in RE

Answer 7

1. case sensitivie/disjunction..with example
2. negation..with example
3. range
4. RE symbols: ? * +
5. RE: disjunction, precedence

Question 8

types of errors and definition

Answer 8

1. false positive

2. false negative

Question 9

what are the efforts to reduce error rate?

Answer 9

1. increase accy / precision

2. increase coverage / recall

Question 10

what is capture group?

Answer 10

usage of parenthesis storing a pattern in memory.

Question 11

what is a corpus?

Answer 11

a computer-readable collection of text or speech
brown corpus?
brown sentence?

Question 12

what is an utterance?

Answer 12

a unit of speech bounded by silence

Question 13

what are the component in disfluencies?

Answer 13

fragments, filled pauses

Question 14

give example of fragments and filled pauses

Answer 14

1. main mainly

2. uh, uhm

Question 15

definition of word types and tokens

Answer 15

word types are a numer of distinct word in a corpus..tokens are number of running words

Question 16

why code switching is required?

Answer 16

speakers often use multiple languages in single communication act. give example

Question 17

List three task commonly applied as part of any normalization process

Answer 17

1. segmenting words..or tokenizing
2. normalizing word formats
3. segmenting sentences in a running text

Question 18

example of tokenization in UNIX

Answer 18

1. tokenization
2. sorting
3. merging upper & lower case
4. sorting counts

Question 19

what is tokenization, normalization

Answer 19

1. process of segmenting text into words

2. process of making the words into a standard format.

Question 20

what are the issues in tokenization?

Answer 20

the usage of symbols. give example

Question 21

goal of tokenizer

Answer 21

1. expand clitic contractions
2. tokenize multiword expression
3. normalized token

Question 22

case folding

Answer 22

reducing all letters to lower case. however in sentiment analysis, case have sentiments! US vs us is important.

Question 23

word segmentation with max match algorithm

Answer 23

some language don’t use spaces to mark word-boundaries…chinese, thai, japanese.
standard algo is maxmatch algo

Question 24

procedure in max match algorithm

Answer 24

1. start pointer at the beginning of string.
2. find the longest word in dictionary that matches the string starting at pointer.
3. move pointer over the word
4. repeat at step 2 onwards.

Question 25

collapsing words: lemmatization

Answer 25

lemmatization is a task of determinig that two words have the same root despite surface difference.
carried out using morphological parsing.
morphology is a study of the way words are build up from smaller meaning bearing unit called morphemes.

Question 26

what are the 2 broad class of morphemes

Answer 26

stems: central morpheme of the word. supplying main meaning
affixes: additional meanings of various kinds.????

Question 27

what is stemming?

Answer 27

reducing terms to their stems. is a crude chopping of affixes

Question 28

Porter stemmer algorithm

Answer 28

give example also

1. reduce s/ss
2. remove “ing” or “ed”
3. remove ational, izer, ator
4. remove al, able, ate

Question 29

sentence segmentation

Answer 29

period “.” is quite ambiguous. give example..Dr.. .02% 4.3
build binary classifier to look for . to decides end of sentence/not end of sentence
either hand-written rules, regular expression, machine learning (decision tree)

Question 30

what is string distance

Answer 30

measures of how 2 string is alike

Question 31

edit distance

Answer 31

let us quantigy that 2 strings are similar or otherwise

Question 32

minimum edit distance path

Answer 32

example of shortest path between string intention and execution