Hyperparameter Optimization Flashcards
(True or false) hyperparameter optimization (HPO) is a black-box optimization problem.
True
R package that provides different tuning approaches
mlr3 tuning
The relevant parameter set can be constructed / specified using the ___ command.
ps()
All the details for the tuning procedure need to be combined in a ____ .
Tuning instance
It specifies the method that is used for optimizing the tuning instance.
Tuner
It defines the number of (equidistant) values per configurable parameter that are to be tried during the tuning procedure.
Resolution
To avoid overfitting, tuning itself should be performed during the ___.
Training procedure
A mixture of learner and tuner
Autotuner
(True or false) “irace” returns the elite configurations and can be used for tuning.
True
Finding a good hyperparameter configuration for
the problem at hand.
Hyperparameter optimization
Which are the two main types of baseline optimizers?
Grid search and random search
Baseline in which a number of values is selected for each parameter and evaluate all possible combinations.
Grid search
Baseline in which each parameter is sampled uniformly at random.
Random search
Evaluating novel configurations (very) different to previous ones.
Exploration
Two types of stochastic approaches for hyperparameter optimization.
Simulated Annealing and CMA-ES
Two types of model-based approaches for hyperparameter optimization.
iterated F-race (irace) and Hyperband.
Trying to improve existing configurations by evaluating similar ones.
Exploitation
True or false? In the context of Bayesian optimization, the „expected improvement“ acquisition function trades off exploration and exploitation of the search space. Therefore, it is a suitable method in situations where functions are expensive to evaluate, e.g. hyperparameter configurations in large-scale problems.
True
Another name for Bayesian Optimization
Sequential Model-Based Optimization, SMBO
A fast-to-evaluate model of performance function, based on
already evaluated configurations, with uncertainty estimate.
Surrogate model
Optimization which evaluates more configurations with lower budgets speeds up HPO by terminating bad configurations early, and thus allows better optimization with limited overall budget.
Multifidelity optimization
Type of multifidelity optimization that starts all configurations with a certain fraction of the budget. Then, discards half of the configurations with worst performance, doubles the budget and repeats until final budget is reached.
Succesive Halving (SH)
The three most important parameters of succesive halving.
Budget factor, final budget, and total used budget.
This algorithm tries to alleviate the problem of how
well the initial training phase of SH captures final performance by introducing multiple brackets.
Hyperband
(True or false) The smallest allocated budget increases with each consecutive bracket in the hyperband algorithm.
True
Another advanced tool for (multifidelity) algorithm configuration which uses statistical tests to identify and stop bad configurations. Its central concept is “racing”.
irace
(True or false) In Successive Halving, the budget allocated to the runs is halved after each iteration.
False. After each stage, the budget is multiplied by 𝜂 and the number of active configurations divided by 𝜂.
True or false? Hyperparameter optimization can be understood as a special case of the CASH problem.
True
