Stream-based Text Processing

Question 1

What is DFA ?

Answer 1

Deterministic Finite Automaton
Formallydefinedasa5-tuple:(Q,Σ,δ,q0,F) – Qisasetofstates
– Σ is an input alphabet
– δ:Q×Σ→Qisatransitionfunction
– q0 ∈ Q is the start state
– F ⊂ Q is a set of final or accepting states

Question 2

What is NFA ?

Answer 2

Non-deterministic Finite Automaton

Formally:(Q,Σ,δ,q0,F)

Question 3

Reguler expression : Literal ?

Answer 3

/words/

Question 4

REX: Character class ?

Answer 4

/./ (any character)

Question 5

REX: any of the characters ?

Answer 5

/[abc]/ (a or b or c)

Question 6

REX: range of characters ?

Answer 6

/[0-9]/, /[a-z]/, /[A-Za-z0-9_-]/

Question 7

REX: case sensitive ?

Answer 7

/[_-]/

/[A-Z_-]/

Answer 8

start of line

Question 9

$

Answer 9

end of line

Question 10

\s

Answer 10

white space

Question 11

\S

Answer 11

not white space

Question 12

\d

Answer 12

digit

Question 13

\D

Answer 13

not digit

Question 14

\w

Answer 14

word

Question 15

\W

Answer 15

not word

Question 16

(a|b)

Answer 16

a or b

Question 17

[^abc]

Answer 17

not a or b or c

Question 18

*

Answer 18

0 or more

Question 19

+

Answer 19

1 or more

Question 20

?

Answer 20

0 or 1

Question 21

{3}

Answer 21

exactly 3

Question 22

{3,}

Answer 22

3 or more

Question 23

{3,5}

Answer 23

3 or 4 or 5

Question 24

Perl variable names start with $, @, or % ?

Answer 24

$a — a scalar variable
@a — an array variable
%a — an associative array (or hash)

Question 25

Perl read line from input ?

Answer 25

<>

Question 26

What is morphemes ?

Answer 26

smallest meaningful units of a word

Question 27

What is the function of stems and affixes ?

Answer 27

stems provide the “main” meaning,

affixes act as modifiers e.g prefix, suffix

Question 28

What is tokenization ?

Answer 28

break plain text into words or tokens,

includes numbers normalize use lower case

Question 29

What is stemming(词干提取) ?

Answer 29

map words into stems

Question 30

what is Lemmatization(词形还原) ?

Answer 30

convert words in dictionary form
studying to study
studies to study

Question 31

What are the three text/word processing method ?

Answer 31

tokenization
stemming
Lemmatization

Question 32

what is Morphological Processes?

Answer 32

word transformation that happens as a regular language transformation

Question 33

What are three main Morphological Processes ?

Answer 33

1. inflection,
2. derivation,
3. compounding.

Question 34

What is inflection ?

Answer 34

a change in the form of a world,
same lexical class
eg, work, working, worked.

Question 35

what is Derivation ?

Answer 35

transform to different lexical class
eg teach (verb) → teacher (noun).

Question 36

what is Compounding ?

Answer 36

two or more words are combined

lady + bug → ladybug

Question 37

What is Zipf’s law ?

Answer 37

the product of rank and frequency of the words in a text is “quite constant,”

Question 38

What is n-grams ?

Answer 38

contiguous sequence of n items of input text