Introduction to statistical learning

Question 1

What does learning mean?

Answer 1

using data

to gain models

Question 2

types of statistical learning

Answer 2

• Supervised Learning
  science of inferring a function from a set of labeled data
  given {y1, y2, … , yN} corresponding to {u1, …, uN} find f such that y=f(u)
• unsupervised Learning
  science of inferring a function to desribe hidden structures from unlabeled data
  given {u1, …, uN} find f such that y=f(u)
  y is not an output but a property of the data
• semi-supervised Learning
  learn a function with mixed labeeled and unlabeled data
• reinforcement Learning
  learn a task based on a reward

Question 3

framework of a supervised Learning problem

Answer 3

• input x: vector of attributes, x app X
• output y: y app Y
• target function f: X - > Y
• data: (x1,y1),…,(xN,yN)
• hypothesis g: X - > Y, g app H
• hypothesis set: H

Data are generated from an unknown target function; based on this data set, a Learning algorithm infer a function g starting from the hypothesis set H; H is selected based on the physical knowledge of the problem.

Question 4

When is statistical Learning meaningful?

Answer 4

Statistical Learning is meaningful if:

1) a pattern exists
2) it is not possible to pin it down mathematically
3) data are available

Question 5

In which sense is Learning feasible

Answer 5

From a deterministic point of view, Learning is not feasible: it is not possible to generalize from a set of examples
Learning is possible only in a statistical point of view

Question 6

Hoeffding’s inequality

Answer 6

P[|p^-p| > ε] < = 2e^-2ε^2*N

where
p^ is the observed distribution from N data
p is the real probability distribution
ε is an arbitrary positive number

• the statement p^ = p is probably approximately correct
• to reduce ε, more data are needed

Question 7

Probability of a generalization error

Answer 7

Given a model g in a set H of M models
Ein(g) is the in sample error
Eout(g) is the out of sample error

P[|Ein(g) - Eout(g)| > ε] < = 2Me^-2ε^2N

where M is the number of models in H

Question 8

General objective of learning

Answer 8

Find a model g that

• minimizes the fitting error Ein(g)
• minimizes the generalization error |Ein(g) - Eout(g)|