Levels of Analysis in NLP Flashcards
What are the levels of language?
Words to Grammar to Meanings to Inferences. The NLP levels are basically the same.
In NLP the levels are…?
Words = Lexical Analysis Grammar = Syntactic analysis Meanings = Semantic analysis Inferences = Discourse/entailment analysis
- Lexical analysis (list of all words in the vocabulary). Not trivial in conversational AI.
- Syntactic analysis: Grammar: structure, parse sentences, tag POS. Assumes you’ve normalized the words. Parser based on rules driven from linguistics and statistics.
- Semantic analysis: Meanings: What does this verb mean compared to other verbs or nouns?
- Discourse/entailment analysis: Inferences: Highest level of NLP where implied meanings are not explicitly spelled out.
- 95% of data science projects that need NLP will be lexical, syntactic, or semantic.
- The process is asymmetrical going back and forth between layers.
What is lexical analysis?
-Words: We are dealing with words here. What counts as a word? Is it singular or plural?
-Text and corpus analytics:
Spell correction or OCR (scanning image to text): -Lexical analysis can correct errors. Infer most likely word that was meant. Correct automatically or suggest other options.
Terminology extraction: Extract key terms from documents or collection of documents. Look at word frequency. We don’t have to look at grammar or syntax or semantics. Extract the most prominent terminology from a collection of documents.
Lexical diversity: Total unique vocabulary compared to how long the book is. Automate and scale language education.
Respect the lexical level of analysis! Don’t skip over this!
- Look at Peter Norvig’s classic post
- Multi-Candidate Ranking Algorithm Based Spell Correction
Actual process (2 models)
Offline
Indexing Tokens
Build a language model
Build an error model - how many transformations do you have to make to transform one word to another (addition/deletion/replacement of single characters)
Find 1 and 2 edit words and find out if they are real words
Query Time
Candidate generation
Scoring
Presenting suggestions
What is syntactical analysis?
Grammar: Tags the parts of speech and the structure of the grammar.
- Sentence by sentence
- Dividing documents into sentences (can’t just use a period because of abbreviations).
- If you can’t divide into sentence, you can’t really do grammar parsing.
- POS tagging (noun, verb, proposition, determine)
- Penn Treebank Tagset from UPenn has tags for POS. Different kinds of adjectives, nouns, etc. Dozens of parts of speech.
- Grammar parsing is hierarchical (parse trees)
- What happens if you feed grammatically incorrect sentences into a grammar parser?
- Lemmatization
- Discrete text field analysis (scraping info off web pages). Use syntax to unitize and normalize data. Unitize breaking data points into units that have been jumbled into 1. Normalize consistent nomenclature and units of measures (inches vs. cm). – Smart ETL.
- Unitize happens all the time because marketers want to improve SEO.
What is discourse analysis?
Gets into the nuances of human conversation.
What is a lexicon?
A machine readable dictionary.
What is morphology?
The study of morphemes
What are morphemes?
Units that our words are made of, e.g., Run is a morpheme and ing is a morpheme for the word running.
What is a stemmer?
Stemming is reducing a word to its root. For the “running” example, the “run” portion of the word is the root or stemmer. It seems like the content portion of the word. A stemmer strips off morphemes not the root “ing” or “ed.” A word cloud is a good use case for stemming. Stem first an count the stemmed version so that you don’t have moral, morals, morality.
-We want to make sure that words like toys and toy represented, then we just want to know toy
What is collocations?
Words commonly occurring together.
What is WordNet?
A lexical database of English. Enumeration of word sense for disambiguation. Can help identify that words have multiple meanings.
What is Lemmatization?
- Form that represents a set of related word forms, eg., run, runs, ran, running, the lemma is “run”
- Lemme is symantically the pure form of a word
- Better is an exception where the lemme is good
- Lemme has to do more than a stemmer
- Uses lexical knowledge bases such as WordNet to obtain word base forms. This is the only difference between Lemmatization and Stemming.
What is semantic analysis?
Meanings
The process of understanding natural language – meaning of words and their relations to other words
“You shall know a word by the company it keeps”
-Learning what a document is about
-Examples: Named-entity recognition, relationship extraction, Word Sense Disambiguation (WSD), Classification, Tagging, Topic segmentation, sentiment analysis.
-E.g., spotted is ambiguous, but when in the context of dalmation it becomes more clear
-For ambiguous terms like jaguar, search engines, check if logged in or logged out, it will personalize searches for you (personalization). If not logged in, they will try to find a trend based on location or geography.
-Model success is based on top 3 or top 5
-Search for things not strings!
-Semantic types and semantic roles
What is named entity extraction (NEE) or named entity recognition (NER)?
-Named entity recognition is another term used for the same thing. Named entities are string of words that ar referring to a person, place, or thing. Tells us whether the string of words in a sentence is a named entity or not.
-Walmart example: Query Samsung LEDTV
First match tokens
Then we match entity types (Product, Brand, Display Type, etc) so that we don’t end up with matches like Remote control for TV just because the tokens match.
What is relationship extraction?
Once you have named entities, this will tell you what the relationship is between them. For example, Apple and Tim Cook are entities and CEO is the relationship.. Requires semantics (including ontology) and syntax analysis.
