Lecture 9 - Unsupervised Learning II

Question 1

What is anomaly detection?

Answer 1

The detection of rare outliers which differ significantly from the rest of the data, can indicate medical problems and mechanical failures for example

Question 2

What are point anomalies?

Answer 2

Single points that represent small or big values

—–/—-

Question 3

What are contextual anomalies?

Answer 3

Unexpected sequences given history

/\/\/\/\___/\/\

Question 4

What are collective anomalies?

Answer 4

Neither single points nor patterns look strange, but in global sense something looks off

For example all 3 lines drop at the same time

Question 5

What are concept drifts?

Answer 5

A slow and steady drift to some new state

For example slowly climbing upwards

Question 6

What is a change point detection?

Answer 6

a strange step, where a shift occurs, for example first high line then sudden drop

———-_________

Question 7

What is supervised anomaly detection?

Answer 7

Labeled data for normal and abnormal cases, unbalanced data makes this difficult approach

Question 8

What is semi-supervised anomaly detection?

Answer 8

Based on a model of normal behaviour which analyses the probability of a data point being within the normal range

Question 9

What is unsupervised anomaly detection?

Answer 9

Most widely used approach, no labeled data

Question 10

What is deviation analysis?

Answer 10

A form of subgroup discovery. Usually a target property given, for example the goal is to find statisticalaly interesting subgroup from the population

Question 11

What are the ingredients of deviation analysis?

Answer 11

• A target measure and a verification test serving as a filter for patterns
• A quality measure to rank subgroups
• A search method that enumerates candidates subgroups systematically

Question 12

How much must p deviate from p0 to be significant?

Answer 12

• z-score
• tests like chi2
• weighted relative accuracy

Question 13

What is Association rule mining?

Answer 13

Originally designated for market basket analysis, aims at finding patterns in the shopping behaviour of customers.

Find sets of products that are frequently bought together (if buys bread and wine, then probably buys cheese)

Question 14

If user buys A -> B does it mean B -> A?

Answer 14

No. For example sausage -> mustard but not necessarily mustard -> sausage

Question 15

What is support of an item set?

Answer 15

Proportion of transactions that contain the item set

Question 16

What is support of an association rule X -> Y?

Answer 16

How often X and Y are bought together

Question 17

What is confidence of an association rule X -> Y?

Answer 17

The percentage of all transactions satisfying X that also satisfy Y. (estimate that Y is bought given X)

Question 18

How to prune item set tree?

Answer 18

• Structural pruning: make sure that there is only one counter for each possible item set (so no ab and ba)
• Size based pruning: prune the tree if a certain depth is reached
• Support based pruning: no superset of an infrequent item set can be frequent, so if no one buys mustard then no one will buy mustard and milk

Question 19

What is Apriori?

Answer 19

Bread-first search for item set

Question 20

What is Eclat?

Answer 20

Depth-first search for item set

Question 21

What is free item set?

Answer 21

Any frequent item set that has a support higher than the minimal support

Question 22

What is closed item set?

Answer 22

A frequent item set is closed if no superset has the same support.

For example c = 70%, cd exists but only 40% support

Question 23

What is maximal item set?

Answer 23

A frequent item set if no superset is frequent, meaning that you can’t make any more subsets

For example a,d can be made to a,d,e but if after that can’t be made any more sets then a,d,e is maximal item set

Question 24

How to calculate the confidence for association rule?

Answer 24

Example: a,c,e bought together 30% of time. a,e bought together 40% of time.

conf = 30%/40% = 75%. There is a 75% confidence that c will be bought with a,e

Question 25

What should be taken into account before applied in practice?

Answer 25

Being aware that something that is predicted by the model might change people’s behaviour

Question 26

Why should you monitor the system during operation?

Answer 26

To find out a drop in performance, the world might change over time and so might the objectives and assumptions

Question 27

What are the 5 anomalies?

Answer 27

1. Single point anomaly
2. Collective anomaly
3. Contextual anomaly
4. Concept drift
5. Change point detection