NLP APPLICATIONS

Question 1

What is Information Extraction

Answer 1

Extract information from text
Identify instances of pre-defined entities (dates, names of people, location) and relations between them

Question 2

How do IE Systems work overall

Answer 2

Find and understand relevant parts of a document
Produce a structured representation of relevant information (semantically more precise form)

Question 3

What are Entities

Answer 3

IE Key step 1
Identification of entities of interest
Entities are named
Find strings in text that denote it
typically designed for each class of interest

Question 4

What are the various methodologies for identifying entities

Answer 4

dictionaries
rule-based: define patterns
machine learning

Question 5

What are Relations

Answer 5

IE Key Step 2
Extract specific facts, relations and events by linking entities
use of templates and regex and grammar
typically designed around important verbs (find the verbs)

Relies on various features
identification of main syntactic units and their relations

Question 6

What is NER

Answer 6

Named Entity Recognition
A systems ability to find an entity and group it in either: name, organisation, date, etc

Question 7

What are the 2 approaches of NER

Answer 7

Knowledge Engineering
-developed by experienced (human) language engineers

Learning Systems
-use statistics or other machine learning

Question 8

What are 4 NER methods

Answer 8

-dictionary based
-rule-based
-machine learning
-hybrid

Question 9

NER Methods: Dictionary look-up

Answer 9

System recognises only entities stored in its lists

Adv
-simple, fast, independent

Disadv
-often impossible to enumerate all names (create exhaustive list)
-ambiguous - name variants

Question 10

NER Methods: Rule-based

Answer 10

Use context clues indicative of specific entity types
but many issues:
- ambiguity in capitalisation, semantic, structure

Question 11

NER Methods: ML sequence model

Answer 11

Consider named entities as sequence of tokens in text
output : sequence of labels that belong to a token (if it is an entity, what entity)

Question 12

Example: HNN NER

Answer 12

Any ML model will work to learn the named entities classification, this is just one way

Each word in the sequence is considered a state
States: Use IO(B) tags
Emission probabilities model the likelihood of observing a word given the state we are in

Issue: cant add a lot of features in HMM to discriminate tags
HMM is a simple model: for a state it does not consider its surrounding context, etc.

Question 13

How does the CRF NER Classifier work

Answer 13

The objective is to learn the weights that maximize the likelihood of the labeled sequences in the training data
taking a global view of the sequence

Question 14

BERT for NER

Answer 14

BERT is considered a local approach, it does not take into account dependencies between tags
Can tune it to identify entities

It is common to add a CRF layer(or any classifier) on top of BERT
local -> global view

Question 15

How do we fine tune pre-trained models

Answer 15

“transfer learning”, contains 2 steps:
- pretrain a large neural network in an unsupervised way
- Fine tune the NN on a specific task of interest

eg use BERT and fine tune it on a IOB token classification problem

Very common approach for NER

Question 16

Traditional ML-based sequence labelling vs Deep learning-based sequence labelling

Answer 16

Traditional
-requires feature engineering
-no need for pretrained embeddings
-models are more interpretable

Deep Learning
-no need for feature engineering
-makes use of pretrained embedding
-Models cannot be interpreted as features are
implicit in hidden layers

Question 17

What is Domain adaptation

Answer 17

There are different entity types in different domains
How do we adapt a model from one domain to another
we have to adjust the output layer, training must be conducted again using both source and target domain, which can be costly
Few shot learning is one of the approaches

Question 18

Performance metrics in NER

Answer 18

Precision = correct answers/answers produced
Recall = correct answers/total possible correct
answers

We may also want to take into account partially correct answers - may recognise half of the entity
so we add +1/2 partially correct to the nominator of precision and recall
and +partial to denominator

Question 19

Adv and disadv of hand-built patterns for relations

Answer 19

Pros
Human patterns tend to be high-precision
Can be tailored to specific domains

Cons
Human patterns are often low-recall
A lot of work to think of all possible patterns
We’d like better accuracy

Question 20

How do we extract relations

Answer 20

Using rules

supervised ML

Question 21

Relation extraction: what are Trigger Words

Answer 21

identify specific trigger words mentioned between entities

Question 22

What is Relation Bootsrapping

Answer 22

A Semi-supervised method for relation extraction
Minimise reliance on large training set
Give a few examples or a few high precision patterns

Gather a set of seed pairs that are linked by the given relation
Iterate:
1. Find sentences with these pairs
2. Look at the context between or around the pair and generalize the context to create patterns
3. Use the patterns to “grep” for more pairs

Question 23

IE: Some challenges

Answer 23

Variability and ambiguity of templates
John Lewis announced - company or person?

Co-references
using different words to refer to same entity
John likes cats, he says the are soft - John/he

Question 24

What is Knowledge Distillation

Answer 24

compression technique in ML
Knowledge from a large, complex model (teacher) is transferred to a smaller, more lightweight model (student)
student model is more computationally efficient with the same knowledge
BERT -> BioBERT
(because large models can be computationally challenging to run)

Question 25

What is a few shot approach

Answer 25

ML method
a model is trained or fine-tuned with only a very small number of examples per class or task
particularly valuable in scenarios where obtaining extensive labeled data is challenging, expensive, or time-consuming