General ML, Evaluation & GDPR

Question 1

What does the statement ‘Machine Learning is an ill-posed problem’ mean?

Answer 1

• An ill-posed problem is a problem for which a unique solution cannot be determined using only the information that is available.
• In terms of ML, the training set represents only a small sample of possible sets of instances in the domain
• A consistent model cannot be found based on the sample training dataset alone.
• If a predictive model is to be useful, it must be able to make predictions for queries that are not present in the data.
• A predictive model that makes the correct predictions for these queries captures the underlying relationship between the descriptive features and target features and is said to generalize well.

Question 2

ABT

Answer 2

Analytics Base Table

Question 3

Inductive bias

Answer 3

• necessary for learning to occur
• the set of assumptions that defines the model selection criteria of a machine learning algorithm
• two types (restriction, preference)

Question 4

Two types of inductive bias

Answer 4

1. Restriction bias

2. Preference bias

Question 5

Restriction Bias

Answer 5

Constrains the set of models that the algorithm will consider during the learning process

Question 6

Preference Bias

Answer 6

Guides the learning algorithm to prefer certain models over others

Question 7

No Free Lunch Theorem

Answer 7

There’s no single inductive bias that’s best to use

Question 8

What is Predictive Data Analytics?

Answer 8

The art of building and using models that make predictions based on patterns extracted from historical data

Question 9

Applications of predictive data analytics

Answer 9

• price prediction
• dosage prediction
• risk assessment
• propensity modelling (likelihood of an individual or customer to take different actions)
• diagnosis
• document classification

Question 10

Consistency of a model?

Answer 10

~ memorizing the dataset

• consistency with noise in the data isn’t desirable
• coverage through memorization is never possible in real problems

Question 11

What is the goal of a predictive model?

Answer 11

A model that generalizes well beyond the dataset and that is invariant to the noise in the datast

Question 12

What is under-fitting?

Answer 12

Occurs when the prediction model selected by the algorithm is too simplistic to represent the underlying relationship in the dataset between the descriptive features and the target features.

Question 13

What is over-fitting?

Answer 13

Occurs when the prediction model selected by the algorithm is so complex that the model fits to the dataset too closely and becomes sensitive to noise in the data.

Question 14

Goldilocks model

Answer 14

Strikes a good balance between under-fitting and over–fitting
- found by using ML algorithms with appropriate inductive biases

Question 15

2 defining characteristics of ensembles

Answer 15

1. Build multiple different models from the same dataset by inducing each model using a modified version of the dataset
2. Makes a prediction by aggregating the predictions of the different models in the ensemble

Question 16

What is an ensemble?

Answer 16

A prediction model that is composed of a set of models is called a model ensemble.
- Rather than creating a single model, they generate a set of models and then make predictions by aggregating the output of these models

Question 17

Motivation behind ensembles

Answer 17

The idea that a committee of experts working together on a problem are more likely to solve it successfully than a single expert working alone

Question 18

Bayes Optimal Ensemble

Answer 18

• an ensemble of all the hypotheses in the hypothesis space
• on average, no other ensemble can outperform it
• not possible to practically implement a Bayes Optimal Classifier
• no upper limit (Theory of Large Numbers: As the number of samples gets bigger, your estimate will get better).
• Setting the number of ensembles really really high is going to give you good performance.

Question 19

2 properties of good ensembles

Answer 19

1. Individual models should be strong

2. Correlation between model should be weak

Question 20

What is the bias/variance trade-off?

Answer 20

TLDR: High bias = underfitting, high-variance = overfitting

The bias is an error from erroneous assumptions in the learning algorithm. High bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting).
The variance is an error from sensitivity to small fluctuations in the training set. High variance can cause an algorithm to model the random noise in the training data, rather than the intended outputs (overfitting).

Question 21

Why is it that local minima ain’t so bad after all

Answer 21

In the case of neural nets, local minima are not necessarily that much of a problem.

1. Some of the local minima are due to the fact that you can get a functionally identical model by permuting the hidden layer units, or negating the inputs and output weights of the network etc.
2. Also if the local minima is only slightly non-optimal, then the difference in performance will be minimal and so it won’t really matter.
3. Lastly, and this is an important point, the key problem in fitting a neural network is over-fitting, so aggressively searching for the global minima of the cost function is likely to result in overfitting and a model that performs poorly.

Regularization such as weight decay can help combat overfitting

Question 22

In practice local minima are rarely a problem with large networks. Discuss.

Answer 22

LeCun, Bengio & Hinton (2015) Nature

Regardless of initial conditions, the system nearly always reaches solutions of very similar quality.

Recent theoretical and empirical results suggest -> not a serious issue

Instead, the landscape is packed with a combinatorially large nuber of saddle points where the gradient is zero, and the surface curves up in most dimensions and curves down in the remainder

Analysis seems to show that saddle points with only a few downward curving directions are present in very large numbers, but almost all of them have very similar values of the objective function. So, it doesn’t matter if it’s get stuck at these points

Question 23

Recommendations of GDPR paper (Wachter et al. 2018)

Answer 23

1. Add a right to explanation to legally binding Article 22
2. Clarify ‘significance…envisaged consequences…logic involved’.
3. Clarify ‘solely’ for automated processing
4. Clarify ‘legal’ or ‘significant effect’ of automated processing
5. Clarify ‘necessary for entering or performance of a contract’
6. Clarify if a prohibition is meant by ‘right not to be subject to’
7. Implement external auditing mechanism for automated decision-making (counterweight to trade secret)
8. Support further research to alternative accountability mechanisms

Question 24

GDPR in a nutshell

Answer 24

• 25th May 2018
• replaces 1995 Data Protection directive
• transparency, security, accountability
• standardizing and strengthening the rights of an individual to data privacy

Regulations surrounding profiling, automated decision making

Question 25

Why might be DL become illegal according to GDPR paper?

Answer 25

1. General data protection
2. Prohibition on profiling/automated decision-making
3. Right to explanation

Question 26

GDPR - General Data protection

Answer 26

1. Direct personal data
2. Indirect personal data

Onus on data controllers to be responsible

Data subjects can request to have info erased, object to direct marketing, inaccuracies corrected, restrict automated processing, data portability

Question 27

Article 22 - Prohibition on profiling/automated DM

Answer 27

Allowed under 3 conditions

1. necessary for contract
2. allowed under member state law
3. explicit consent

• right not to be subject to… but what is a ‘legal effect’ or ‘similarly significant effect’?

Question 28

Right to explanation

Answer 28

• system functionality

* specific decision

Question 29

What’s the verdict - does GDPR mandate a right to explanation?

Answer 29

Consensus is no
Article 22 is vague (maybe intentionally)
Recital 71 - some hope - but just guidance and not legally binding

Question 30

Broadly speaking, what is the difference between evaluation ML methods in industry versus academia?

Answer 30

Industry - evaluate a model that we would like to deploy for a specific task

Academia - compare ML methods

Question 31

Reasons for evaluation ML model in industry

Answer 31

1. determine which model is most suitable for a task
2. estimate performance after deployment
3. convince users that model will meet their needs

Question 32

Reasons for evaluation ML model in academia

Answer 32

1. Evaluate the performance of a new method against existing baselines
2. Determine best ML approach for a problem
3. Perform benchmark experiment

All boils down to comparing multiple approaches on multiple datasets

Question 33

Key difference between evaluation in industry versus academia

Answer 33

Significance testing

Question 34

Performances measures for Industry

Answer 34

1. macro-averaging vs micro-averaging
2. hold-out test
3. k-fold CV

Question 35

Performance measures for academia

Answer 35

Two-fold process:

1. Friedman Aligned Rank Test - test if there’s a significant difference between the performances of the algos across the datasets (p < 0.05)
2. Nemenyi Test - If there was a significant difference in part 1, find out where the difference exists between algo-pairings

Nemenyi Test -> Significance matrix, and Critical Differences Plot