Performance Evaluation

Question 1

What types of evaluation are there (3)

Answer 1

• performance
• adequacy
• diagnostic

Question 2

What is performance evaluation (3)

Answer 2

based on a benchmark
organised around community/shared task
automated means of scoring

Question 3

key points about gold standard data (3)

Answer 3

• time consuming and costly
• requires annotation guidelines to follow
• annotation done by experts

Question 4

why do we use multiple annotators

Answer 4

to ensure reliability

Question 5

what does the kappa coefficient measure here

Answer 5

inter annotator agreement

Question 6

how do we calculate kappa coefficient

Answer 6

(p(a) - p(e)) / 1 - p(e)

Question 7

p(a) = …

Answer 7

observed agreement

p(a1=y, a2=y) + p(a1=n, a2=n)

Question 8

p(e) = ..

Answer 8

expected agreement

p(a1=y)p(a2=y) + p(a1=n)p(a2=n)

Question 9

how do we interpret the kappa coefficient

Answer 9

slight < 0.2 < fair < 0.4 < moderate < 0.6 < substantial < 0.8 < perfect

Question 10

what can we use in the non binary annotation case

Answer 10

scotts pi, fleiss kappa

Question 11

precision =

Answer 11

TP / TP+FP

Question 12

recall =

Answer 12

TP / TP+FN

Question 13

f1 =

Answer 13

2PR / R+P

Question 14

why is f1 score more informative than mean

Answer 14

it is the harmonic mean- it will show poor performance e.g. if prediction is always no

Question 15

when is accuracy useful

Answer 15

if all classes are equally important

Question 16

what kinds of averages can we use when we have multiple categories

Answer 16

macro average, micro average

Question 17

what is macro average

Answer 17

take the average

Question 18

what is micro average

Answer 18

pool tps, fps and fns. less sensitive to class imbalance

Question 19

what is olympic judging

Answer 19

if there is not enough data for a gold standard, a committee of judges determines whether a proposal is relevant and close to the desired result.

not reproducible

Question 20

what is adequacy evaluation

Answer 20

evaluation as seen by users, not quantifiable and interdependent. Judging the external quality

Question 21

what are some of the factors for adequacy evaluation

Answer 21

adaptability, integrity, efficiency, robustness, correctness, reliability, usability, accuracy

Question 22

what is diagnostic evaluation

Answer 22

concerned with evaluation as seen by developers

Question 23

what are some of the factors for diagnostic evaluation

Answer 23

profitability, reusability, maintainability, testability, understandability, flexibility, readability

Question 24

why cant we use a NLP test suite

Answer 24

the range of phenomena is hard to anticipate