Week 4 - Text Mining II Flashcards
How many sequences of words are there? And why should there be a sequence of words?
We need a sequence of words because sometimes a sequence of words may contain more information. There are three sections for the sequence of words:
1. compound words
2. phrases with more context
3. sentences for context
By tying a word to its surrounding words, we may retain a greater amount of information and a more precise understanding of context.
What are N-grams?
N-gram is sequence of consecutive n elements extracted from a text. Elements can be words/token, syllables, characters, symbols.
What is the criteria when selecting n?
Generally smaller n (biagram or triagram) works well. Four-gram and five-gram might be useful when you have large data sets.
What is TFIDF with n-grams?
It’s a bag of n grams model (preserving more context) and common to create tfidf with n-grams as features to use in predictive modelling (e.g., machine learning)
What is Key Words in Context (KWiC)?
Keywords-in-context (KWiC) displays the concordances lines with the keyword in the middle along with nearby words.
This helps gain insight into how a word or phrase is used in a corpus, how frequently and in which context
What is Part Of Specch (POS)?
POS is to identify the lexical category of words. This classifies a word with its corresponding part of speech (e.g., nouns, verbs, pronouns, adjectives, adverbs, and many more)
What is POS tagging?
POS tagging is the process of assigning tags to words with its corresponding part of speech.
The context of the word is required to identify its POS.
What is the use/purpose of POS tagging?
POS tagging is used to compare the grammar of different texts, make grammar corrections, auto-complete, help to translate text from one language to another, and create more specific features in a document.
How do you generate POS tags?
POS tags are generate using Python libraries spaCy and NLTK, and Java libraries OpenNLP and CoreNLP.
What is a Sentiment Analysis?
Sentiment analysis (or opinion mining), studies the opinions, attitudes, and emotions of a writer toward a subject matter (e.g., an entity, or event).
What are the potential applications for security and crime science?
Examining discussions, public opinion or reactions related to security and crime: predicting crime patterns, potential signals for online radicalisation, disinformation, hate-speech and harmful behaviour detection, phishing etc.
Main steps in sentiment analysis
- Tokenise the text
- Create a sentiment lexicon (or use an existing sentiment lexicon)
- Judge the sentiment of tokens
- Match tokens with sentiment lexicon
What are the specifics of creating a sentiment lexicon?
A sentiment lexicon is a dictionary of words where each word is assigned a corresponding sentiment score.
Do all the words have a potential sentiment?
Not all words (e.g., neutral words) may have a sentiment, but you may wish to focus on adjectives/adverbs.
Sentiment lexicons exist; lexicons might be designed for a specific intended usage
What’s a limitation of syuzhet?
There’s a limitation when you match tokens with sentiment lexicon II
What is considered as Valence Shifters?
Nearly all sentiment analysis packages in R fail to recognise valence shifters except sentimentr
Negatos, amplifiers, de-amplifiers, adversative conjuctions.
What is the limitation of nuanced approaches?
Sentiment for each sentence:
- needs punctuated data but, data is often not punctuated (e.g., texts on social media) or badly punctuated.
- without punctuation whole text is seen as one sentence
- requires accurate sentence boundary disambiguation (sentence breaking) and data might not be suitable (e.g., Twitter -> too short)
What is the benefit of Sentiment Trajectory Analysis?
It’s a possible to aid understand the style structure of texts, what makes a popular speech & detect misinformation
What is the process for doing a sentiment trajectory?
- Parse the text into tokens
- Match the sentiment lexicon to each token
(match valence shifters to each context; apply valence shifter weights; build a naive context around the sentiment; return a modified sentiment)
What is the ranking of the valence shifters?
There is 4 valence shifters:
1 = negator (not, never, …)
2 = amplifier (very, totally, …)
3 = deamplifier (hardly, barely, …)
4 = adversative conjunction (but, however, …)
What is a sentiment trajectory?
Having a sentiment trajectory to build ‘naive’ context around the sentiment; this means 2 words before the sentiment word and 2 words after.
In some cases of trajectories; the transcript has different lengths. How can you standardise the length?
The solution is length standardisation.
We transform all sentiment values into a standardized vector form. We do this because standard vector length makes comparisons easier and more consistent.
What’s an aspect of trajectory that you need to beware of?
Beware of the “filter” size; increasing the filter parameter will add more granularity. The default filter size is 5, compared to the filter size is 20.
