Hoorcollege 10 Natural language inference

Question 1

SICK-NL

Answer 1

dataset for dutch natural language inference

Question 2

RTE (recognizing textual entailments) / NLI

Answer 2

Recognizing textual entailment is an established task in NLP
* RTE covers strictly speaking more than logical entailment

SICK dataset -> sentences involving compositional knowledge
* SNLI is lager, like sick, description of scenes
* MultiNLI, more diverse text ‘genre’ sources

Question 3

Entailment datasets

Answer 3

At the word level entailment is hypernymy (dog -> animal) more general

Categorical vs graded entailment
* Chemistry -> science (10.0)
* Enemy -> crocodile (0.33)
Phrase level
* Parrot -> pet
* Dead parrot -/> pet
Sentence level
what a sentence entails: Entailment, contradiction, neural

Question 4

Dataset formats

Answer 4

• Entailment
• Non entailment: contradiction, UNKNOWN / Neural

(Hella) SWAG
* Different focus: commonsense reasoning
* Different format: multiple choice

Question 5

NLI in language models

Answer 5

• LLMs are fine-tuned on NLI data (don’t transfer well to new inference data)
• Achieve good accuracy
• textual entailment different from logical since it isn’t exact
• issue with variation between logical contradiction and referential contradiction in datasets

a man is smoking vs a man is not smoking

Question 6

Interannotator agreement

Answer 6

expected agreement & Cohen’s kappa (for two annotators) &
Fleiss’s kappa (for more than two annotators) measures agreement between annotators