Path4.Mod3.b - Perform Hyperparameter Tuning - Sweep Job Sampling Methods (Sampling)

Question 1

Grid Sampling defined
The prerequisite for Grid Sampling

Answer 1

Creates a grid of all possible combinations of values in order to try each one. Only applicable when ALL hyperparameter values are discrete.

Question 2

Explain Grid Sampling code:

from azure.ai.ml.sweep import Choice

command_job_for_sweep = command_job(
    batch_size=Choice(values=[16, 32, 64]),
    learning_rate=Choice(values=[0.01, 0.1, 1.0]),
)

sweep_job = command_job_for_sweep.sweep(
    sampling_algorithm = "grid",
    ...
)

Answer 2

Creates a command_job instance using a Choice of values for batch_size and learning_rate hyperparameters. Then when creating the sweep, the sample_algorithm is set to “grid”. This tells the job to use all possible combinations of each hyperparameter, one per trial.

Question 3

Grid Sampling - Pros and Cons

Answer 3

Pro: Simplest approach, ensures our trained models gets most of its patterns from the dataset
Con: Suffers when data has high dimensional space (the total number of combinations grows exponentially)

Question 4

Random Sampling defined
- RandomSamplingAlgorithm class, values for rule parameter
- Define Sobol and why it’s useful.

Answer 4

Randomly choose values from the search space. Can be discreet or continuous values.
- RandomSamplingAlgorithm accepts a rule value as a string “sobol” or its default “random”
- Sobol: Add a seed to random sampling to make results reproducible.

Question 5

Explain Random Sampling code

from azure.ai.ml.sweep import Normal, Uniform

command_job_for_sweep = command_job(
    batch_size=Choice(values=[16, 32, 64]),   
    learning_rate=Normal(mu=10, sigma=3),
)

sweep_job = command_job_for_sweep.sweep(
    sampling_algorithm = "random",
    ...
)

Answer 5

Creates a command_job instance using a Choice of values (discrete) for batch_size and a Normal (continuous) distribution for learning_rate hyperparameters. Then when creating the sweep, the sample_algorithm is set to “random”. This tells the job to use a random pairing of batch_size with a value from the Normal Distro , one per trial.

Question 6

Explain Random Sobol Sampling variation code:

from azure.ai.ml.sweep import RandomParameterSampling

command_job_for_sweep = command_job(
    batch_size=Choice(values=[16, 32, 64]),   
    learning_rate=Normal(mu=10, sigma=3),

sweep_job = command_job_for_sweep.sweep(
    sampling_algorithm = RandomParameterSampling(seed=123, rule="sobol"),
    ...
)

Answer 6

Creates a command_job instance using a Choice of values (discrete) for batch_size and a Normal distribution (continuous) for learning_rate hyperparameters. Then when creating the sweep, the sample_algorithm is set to an instance of RandomParameterSampling set to use the “sobol” rule with a seed value. This tells the job to use a random pairing of batch_size + a value starting at the seed value for the Normally Distributed value , one per trial.

Question 7

Random Sampling - Pros and Cons

Answer 7

Pro: Instead of trying all possible combinations from the sample space, it’ll try only a random subset, which allows you to use a mix of Discrete and Continuous values.
Con: Selection is random rather than based on the previous experiments, making it difficult to predict the next set of experiments, and to reproduce results.

Question 8

Bayesian Sampling defined

Answer 8

Choose new values based on previous results. Also allows a mix of Discrete and Continuous values.

Question 9

Explain Bayesian Sampling code:

from azure.ai.ml.sweep import Uniform, Choice

command_job_for_sweep = job(
    batch_size=Choice(values=[16, 32, 64]),    
    learning_rate=Uniform(min_value=0.05, max_value=0.1),
)

sweep_job = command_job_for_sweep.sweep(
    sampling_algorithm = "bayesian",
    ...
)

Answer 9

Creates a command_job instance using a Choice of values (discrete) for batch_size and a Uniform (continuous) distribution for learning_rate hyperparameters. Then when creating the sweep, the sample_algorithm is set to “bayesian”. This tells the job to select parameter combinations that will result in improved performance from the previous selection.

Question 10

Bayesian Sampling - Pros and Cons

Answer 10

Pro: Improved performance based on previous selections
Con:
- Cannot be used in combination with Early Termination Policy.
- Limited parameter expression methods options: Choice, Uniform, and QUniform

Limited to Uniform/QUniform for simplicity and computational efficiency. Bayesian methods can get complex quickly, and using simple distributions like Uniform/QUniform makes it easier for the algorithm to make smart guesses without taking too much time or computational power.