C2

Question 1

markup

Answer 1

meta information in a text file that is clearly distinguishable from the textual content

Question 2

unicode

Answer 2

universal standard for all writing systems, more inclusive than ASCII

for maximum compatibility we encode texts in UTF-8 when reading and writing

Question 3

minimum edit distance between two strings

Answer 3

minimum number of editing operations (insertion, substitution, deletion) needed to transform one string into another

Levenshtein distance: deletion, insertion and substitution all have a cost of 1

Question 4

token count

Answer 4

number of words in a document, including duplicates

Question 5

vocabulary size

Answer 5

number of unique terms, feature size when we use words as features

Question 6

stop words

Answer 6

extremely common words without much content

• remove stop words: keyword extraction
• never remove stop words: sequence labelling tasks or classification tasks with small data

Question 7

basic word forms

Answer 7

reduce number of features and generalizes better

lemma: dictionary form of a word (verbs: infinitive, nouns: singular form)

stem: portion of a word that is common to a set of (inflected) forms when all affixes are removed (not further analyzable into meaningful elements)

Question 8

character encoding

Answer 8

the way that a computer displays text in a way
that humans can understand

Question 9

Levenshtein Distance op (i,j)

Answer 9

min van:
D(i-1, j) + 1
D(i, j-1) + 1
D(i-1, j-1) + 1 als X(i) neq Y(j)
D(i-1, j-1) + 0 als X(i) = Y(j)

Question 10

token

Answer 10

an instance of a word or term occurring in a document

Question 11

term

Answer 11

a token when used as feature (or in an index), generally in normalized form (e.g. lowercased)

Question 12

Optical Character Recognition

Answer 12

a technique for converting the image of a printed text to a digital text