Path4.Mod3.c - Perform Hyperparameter Tuning - Sweep Job Early Termination Flashcards
ei de
Two common parameters for configuring an Early Termination Policy
-
evaluation_interval
- How many trials per evaluation ie 1 means eval on every trial -
delay_evaluation
- How many completed trials to wait before we start evaluating for ET policy (allows training a few Models in order to get some performance numbers)
BP MSP TSP
Three policies for configuring an Early Termination Policy
- Bandit Policy
- Median Stopping Policy
- Truncation Selection Policy
Bandit Policy
- Explain slack_factor
vs slack_amount
- Explain lower vs higher factors
Stop a trial if the target performance metric underperforms the best trial so far by a specified margin, defined by either a slack_factor
or a slack_amount
-
slack_amount
: the next score underperforms by this specific amount -
slack_factor
: the next score underperforms by(highest / (1+slack_factor))
. ex. ifslack_factor = 0.1
and highest = 1.0 then ET happens when the next trial score is less than (1/1.1) = 0.91 - higher factors == more lenient, vs lower factors == steeper thresholds. Ex. 0.12 comes out to 0.89, so there’s some wiggle room for a trial.
Explain Bandit Policy code:
from azure.ai.ml.sweep import BanditPolicy sweepjob.early_termination = BanditPolicy( slack_amount = 0.2, delay_evaluation = 5, evaluation_interval = 1 )
- We delay evaluation for 5 trials, so we let 5 go and we start evaluating at the 6th trial
- We evaluate at each (1) interval, so after the 5th internval we evaluate each one going forward
- We stop the Sweep Job when a new trial’s target metric is 0.2 (or 20%) less than the best performing model thus far. W.r.t. the chart below, since the best was 0.9, the first chart is still good at 0.85, but the second chart shows a value below the slack threshold at 0.65, so we stop the sweep job.
Median Stopping Policy
Stop a trial if the target performance metric is worse than the median or average of all trials
Explain Median Stopping Policy code:
from azure.ai.ml.sweep import MedianStoppingPolicy sweepjob.early_termination = MedianStoppingPolicy( delay_evaluation = 5, evaluation_interval = 1 )
- We delay evaluation for 5 trials, so we let 5 go and we start evaluating at the 6th trial
- We evaluate at each (1) interval, so after the 5th internval we evaluating each one going forward
- We stop the Sweep Job when a new trial’s target metric comes in below the average metric value across all previous trials, denoted by the dashed line.
Truncation Selection Policy
Cancel the lowest performing % of trials at each evaluation interval based on the truncation_percentage parameter
Explain Truncation Selection Policy code:
from azure.ai.ml.sweep import TruncationSelectionPolicy sweepjob.early_termination = TruncationSelectionPolicy( delay_evaluation = 5, truncation_percentage = 20, evaluation_interval = 1 )
- We delay evaluation for 5 trials, so we let 5 go and we start evaluating at the 6th trial
- We evaluate at each (1) interval, so after the 5th internval we evaluating each one going forward
- We stop the Sweep Job when a new trial’s target metric comes in at the lowest 20% of all trials so far. In other words, if the trial isn’t in the lowest percentile of trials, the job continues.