Path4.Mod3.d - Perform Hyperparameter Tuning - Sweep Job Implementation Flashcards
cmd mlf
Two things required by your training code/script before using it in a Sweep Job
- Include a command line argument for each hyperparameter you want to try
- Log the metric with MLFlow; the Sweep Job needs it to evaluate the performance of each trial and identify the best performing model
The code below includes a hyperparameter reg_rate
and logs a metric “Accuracy”:
~~~
// train.py code
// Commandline arg “–regularization” gets value
// put into ‘reg_rate’ input
parser = argparse.ArgumentParser()
parser.add_argument(‘–regularization’, type=float, dest=’reg_rate’, default=0.01)
args = parser.parse_args()
reg = args.reg_rate
// Training code that uses your
// Hyperparameter value ‘reg_rate’
// Log your target metric
acc = np.average(y_hat == y_test)
mlflow.log_metric(“Accuracy”, acc)
~~~
To create a Sweep Job, instantiate a instance of this class
This is not an instance of CommandJob
An instance of command
The code below shows how to create one:
- The command line to execute your script “train.py” script along with its commandline Hyperparameters + input values taken from the input json.
- Note that the inputs
field will create those input parameters dynamically, making them dynamically available to use as input to the command
parameter:
from azure.ai.ml import command configure command job as base cmd_instance = command( code="./src", command="python train.py --regularization ${{inputs.reg_rate}}", inputs={ "reg_rate": 0.01, }, environment="AzureML-sklearn-0.24-ubuntu18.04-py37-cpu@latest", compute="aml-cluster" )
Override a command
instance’s input to use a Search Space
To override the input
with your Search Space, set the hyperparameter input directly through the command instance. Remember that input
creates those key:value pairs dynamically, ergo they are dynamically accessible. You can access them through the command
instance by its reference as a function that will return an instance of a CommandJob
Below, the command’s reg_rate
input is set to a discrete search space via the Choice
function:
from azure.ai.ml.sweep import Choice // access hyperparameter directly through // the command instance command_job_for_sweep = cmd_instance( reg_rate=Choice(values=[0.01, 0.1, 1]), )
Python code to create a Sweep Job from your Job
- Initialize the Sweep Job
- Set the Experiment Name
- Set limits for your Sweep Job
- Submit the job
Use sweep method to get an instance
from azure.ai.ml import MLClient // Get an instance of the actual sweep_job sweep_job = cmd_job.sweep( compute="aml-cluster", sampling_algorithm="grid", primary_metric="Accuracy", goal="Maximize", ) // The sweep function doesn't set this // So set the name of the sweep job experiment sweep_job.experiment_name="sweep-example" // Define the limits for the sweep sweep_job.set_limits( max_total_trials=4, max_concurrent_trials=2, timeout=7200) # Submit the sweep returned_sweep_job = ml_client.create_or_update(sweep_job)
The chart that shows all metrics logged for each hyperdrive child job over the duration of hyperparam tuning, with each line being a child job and each point measuring the primary metric value.
Metrics Chart
The chart that shows the correlation between primary metric performance and individual Hyperparameters.
Parallel Coordinates Chart. The chart itself is Interactive, including an axis that plots the best metric value corresponding to hyperparams set for that job instance.
The chart that shows the correlation between primary metric performance and any TWO Hyperparameters
2-Dimensional Scatter Chart
The chart that shows the correlation between primary metric performance and any THREE Hyperparameters.
3-Dimensional Scatter Chart. You can also reorient along a specified axis.
- Python code to download the best trial output
- CLI code to get ALL outputs
- CLI code to get the best trial output
ml_client.jobs.download(returned_sweep_job.name, output_name="model_name") az ml job download --name <sweep-job-name> --all az ml job download --name <sweep-job-name> --output-name model_name