Lecture 1

Question 1

Pro’s and cons of AML?

Answer 1

Pro:
Emerging research field
Democratization of ML
Less work
Support Data Scientist, not replace

Con:
General AI

Question 2

What is AML?

Answer 2

Algorithm selection (combined with hyperparameter optimization)
Hyperparameter optimization
Workflow Synthesis
Neural Architecture Search
Few shot learning

Question 3

Hyperparameter Optimization

Answer 3

Argmin_(lambda) Loss(Algo_(lambda), TrainSet, ValidSet)

Question 4

Algorithm Selection

Answer 4

Argmin_(algorithm) Loss(Algo, TrainSet, ValidSet)

Question 5

CASH

Answer 5

argmin lambda and algorithm

Question 6

Search Algorithms

Answer 6

Grid
Random
Bayesian Optimization

Question 7

Bayesian Optimization

Answer 7

Find x with low f(x)
Find promising points with acquisition function (expected improvement)
Repeat for next x
(Not parallelizable)

Question 8

Expected improvement

Answer 8

z = ( f s t a r − mu) / s i g m a
r e t u r n ( f s t a r − mu) ∗ norm . c d f ( z ) + s i g m a ∗ norm . p d f ( z )

Question 9

Configuration Spaces

Answer 9

Categorical
Numerical
Conditional Hyperparameters
(Range, Sampling Strat, some shouldnt be optimized(seed))

Question 10

One armed bandit

Answer 10

Every A, hyperp’s is a bandid
Lever pull is a training run cost (budget)

Question 11

Successive Halving

Answer 11

Every part of budget, num of A’s that are trained is halved

Question 12

Budget types

Answer 12

Run time
Observations
Attributes
Suitable hyperparameter value (epoch, ensemble size)

Question 13

Successive halving: pros

Answer 13

Simple
Parallelizable
Converges to optimum
Strong theoretical foundation
Good results

Question 14

Successive halving: cons

Answer 14

Extension to random search
Not data efficient
Human designed bandit strat
Learning curves can cross
Good solutions might be dropped