Hyperparameter Optimization

Question 1

(True or false) hyperparameter optimization (HPO) is a black-box optimization problem.

Answer 1

True

Question 2

R package that provides different tuning approaches

Answer 2

mlr3 tuning

Question 3

The relevant parameter set can be constructed / specified using the ___ command.

Answer 3

ps()

Question 4

All the details for the tuning procedure need to be combined in a ____ .

Answer 4

Tuning instance

Question 5

It specifies the method that is used for optimizing the tuning instance.

Answer 5

Tuner

Question 6

It defines the number of (equidistant) values per configurable parameter that are to be tried during the tuning procedure.

Answer 6

Resolution

Question 7

To avoid overfitting, tuning itself should be performed during the ___.

Answer 7

Training procedure

Question 8

A mixture of learner and tuner

Answer 8

Autotuner

Question 9

(True or false) “irace” returns the elite configurations and can be used for tuning.

Answer 9

True

Question 10

Finding a good hyperparameter configuration for
the problem at hand.

Answer 10

Hyperparameter optimization

Question 11

Which are the two main types of baseline optimizers?

Answer 11

Grid search and random search

Question 12

Baseline in which a number of values is selected for each parameter and evaluate all possible combinations.

Answer 12

Grid search

Question 13

Baseline in which each parameter is sampled uniformly at random.

Answer 13

Random search

Question 14

Evaluating novel configurations (very) different to previous ones.

Answer 14

Exploration

Question 15

Two types of stochastic approaches for hyperparameter optimization.

Answer 15

Simulated Annealing and CMA-ES

Question 16

Two types of model-based approaches for hyperparameter optimization.

Answer 16

iterated F-race (irace) and Hyperband.

Question 17

Trying to improve existing configurations by evaluating similar ones.

Answer 17

Exploitation

Question 18

True or false? In the context of Bayesian optimization, the „expected improvement“ acquisition function trades off exploration and exploitation of the search space. Therefore, it is a suitable method in situations where functions are expensive to evaluate, e.g. hyperparameter configurations in large-scale problems.

Answer 18

True

Question 19

Another name for Bayesian Optimization

Answer 19

Sequential Model-Based Optimization, SMBO

Question 20

A fast-to-evaluate model of performance function, based on
already evaluated configurations, with uncertainty estimate.

Answer 20

Surrogate model

Question 21

Optimization which evaluates more configurations with lower budgets speeds up HPO by terminating bad configurations early, and thus allows better optimization with limited overall budget.

Answer 21

Multifidelity optimization

Question 22

Type of multifidelity optimization that starts all configurations with a certain fraction of the budget. Then, discards half of the configurations with worst performance, doubles the budget and repeats until final budget is reached.

Answer 22

Succesive Halving (SH)

Question 23

The three most important parameters of succesive halving.

Answer 23

Budget factor, final budget, and total used budget.

Question 24

This algorithm tries to alleviate the problem of how
well the initial training phase of SH captures final performance by introducing multiple brackets.

Answer 24

Hyperband

Question 25

(True or false) The smallest allocated budget increases with each consecutive bracket in the hyperband algorithm.

Answer 25

True

Question 26

Another advanced tool for (multifidelity) algorithm configuration which uses statistical tests to identify and stop bad configurations. Its central concept is “racing”.

Answer 26

irace

Question 27

(True or false) In Successive Halving, the budget allocated to the runs is halved after each iteration.

Answer 27

False. After each stage, the budget is multiplied by 𝜂 and the number of active configurations divided by 𝜂.

Question 28

True or false? Hyperparameter optimization can be understood as a special case of the CASH problem.

Answer 28

True