week5

Question 1

what is a corpus

Answer 1

A collection of pieces of language that are selected and ordered according to explicit linguistic criteria in order to be used as a sample of the language.

Question 2

we can classify corpora by (6)

Answer 2

• mode (written/spoken)
• representativeness (balanced/specialized)
• time (diachronic/synchronic)
• language (monolonual/..)
• sampling (full docs/samples (curated))
• mark-up (raw/POS-tagged)

Question 3

problem with tokenization

& solved by:

Answer 3

there is ambiguity = solved by POS-tagging

Question 4

what is POS tagging?

Answer 4

identification of word class

Question 5

word class determined by:

Answer 5

• semantics
• context of use (nouns after determiners)
• possible affixations

(thus context needed for tagging)

Question 6

zipfs law is F(z) = |C|/z&^a

what are these things?

Answer 6

f is the frequency of a word (token) z divided by the rank of the word. so word of rank 2 should occur 1/2 times the frequency of the word of rank 1.

Question 7

what are tokens

Answer 7

statistical units of study (usually words)

Question 8

what are types

Answer 8

the unique! words

Question 9

give 2 purposes of corpora

Answer 9

• applicative: development of NLP tools

- analytical: empirical basis on the distribution of constructions and language phenomena

Question 10

a balanced corpus needs to be (5)

Answer 10

• big
• mixed (spoken and written words)
• needs to be general (so include many different genres)
• well documented
• cover wide range of text categories (long and short texts)

Question 11

what is the link level in parallel corpora?

Answer 11

at what level are the languages linked. sentence level or word level.