w1 L2 the nlp pipeline and tokenization

Question 1

recall the steps in the nlp pipeline

Answer 1

analyze the task + define framework

preprocess data + get insights

define relevant information + extract from data

select appropriate algorithm + implement

apply it in practice + test and evaluate

Question 2

what is tokenization

Answer 2

the task of seperating out words in raw text, note word definitions are language dependent

Question 3

what are regular expressions

Answer 3

a way of matching patterns,

ie split replace find… based on pattern
these patterns can be complex

Question 4

what are some regular expressions

Answer 4

Special characters: . ^ $ * ? {m}\ [] | ?:
Sequences: \b \B\d \D \s \w \W
Flags: re.I, re.IGNORECASE
Functions: re.compile, re.search, re.match, re.split, re.sub

Question 5

what are the differences between tokenization in LLMs and PLMs

Answer 5

LLM tokenization is purly statistical
larger vocab size

pretrained models look at text in pre trianing paradigmn and try to understand language cues

LLMs are more generalizeable scalable and robust tokenization so they can handle more different applicatiosn

Question 6

what is zipfs law and what does it imply about stop words

Answer 6

a statistical principle that describes the distribution of elements in a dataset

the frequency of any word is inversely proportional to its rank in the frequency table

the zipf curve shows that the most frequent words and vast majority of words are stop words, these words are uninformative so they can be dropped

*might be worth it to rmeber the formula