!pip install deepr[cpu]

Hyper Parameter Tuning

This notebook builds upon the previous pipeline example. The goal is to perform hyperparameter search by varying the learning rate and batch size.

To launch an HP search, the steps are

1. Download / retrieve some config and its macros (usually saved as MLFlow artifacts).

2. Prepare the config by adding new macro parameters for a macro named “params” (for example change "learning_rate": 0.1 to "learning_rate": "$params:learning_rate").

3. Define a sampler.

4. Launch a tuning job using the prepared config, macros and sampler.

First, some imports

import logging
import sys
logging.basicConfig(level=logging.INFO, stream=sys.stdout)
logging.getLogger("tensorflow").setLevel(logging.CRITICAL)
logging.getLogger("cluster_pack").setLevel(logging.CRITICAL)

import numpy as np
import deepr

Prepare config

The easiest way to prepare a config is to

1. Download the config and macros from mlflow with the following command (it will download config and macros artifact to config.json and macros.json paths from some run id):

   deepr download_config_and_macros_from_mlflow --run_id=841d3f9f2ba4b69921b426a81fd --tracking_uri=$MLFLOW_TRACKING_URI
   

   Note that you can also use it in python with

   deepr.cli.download_config_and_macros_from_mlflow(
       run_id=run_id,
       path_config="config.json",
       path_macros="macros.json",
       tracking_uri=TRACKING_URI
   )
   

2. Automatically add new macro references (overwrites the files) with

   deepr add_macro --config=config.json --macros=macros.json --params=learning_rate,batch_size
   

   Note that you can also use it in python with

   deepr.cli.add_macro(
       config="config.json",
       macros="macros.json",
       params=param_grid.keys(),
   )
   

3. Create a short script my_seeded_study_name.py that looks like

import numpy as np
import deepr

# Create sampler
param_grid = {
    "learning_rate": np.logspace(start=-5, stop=-3, num=10),
}
sampler = deepr.jobs.ParamsSampler(param_grid, n_iter=50, seed=42)

# Read job config and macros
job = deepr.io.read_json("config.json")
macros = deepr.io.read_json("macros.json")

# Create tuner and run it
tuner = deepr.jobs.ParamsTuner(job=job, macros=macros, sampler=sampler)
tuner.run()

Our prepared config should look like this

config = {
    "type": "deepr.jobs.yarn_launcher.YarnLauncher",
    "job": {
        "type": "deepr.jobs.Pipeline",
        "eval": None,
        "jobs": [
            {
                "type": "deepr.example.jobs.BuildDataset",
                "path_dataset": "viewfs://root/user/deepr/dev/example/data.tfrecord",
                "num_examples": 1000,
            },
            {
                "type": "deepr.jobs.YarnTrainer",
                "trainer": {
                    "type": "deepr.jobs.Trainer",
                    "eval": "skip",
                    "path_model": "$paths:path_model",  # Uses macro "paths:path_model"
                    "pred_fn": {"type": "deepr.example.layers.Multiply"},
                    "loss_fn": {"type": "deepr.example.layers.SquaredL2"},
                    "optimizer_fn": {
                        "type": "deepr.optimizers.TensorflowOptimizer",
                        "optimizer": "Adam",
                        "learning_rate": "$params:learning_rate",  # Uses macro "params:learning_rate"
                    },
                    "prepro_fn": {
                        "type": "deepr.example.prepros.DefaultPrepro",  # Uses macro "params:batch_size"
                        "batch_size": "$params:batch_size",
                        "repeat_size": 10,
                    },
                    "train_input_fn": {
                        "type": "deepr.readers.TFRecordReader",
                        "path": "viewfs://root/user/deepr/dev/example/data.tfrecord",
                        "num_parallel_reads": 8,
                        "num_parallel_calls": 8,
                        "shuffle": True,
                    },
                    "eval_input_fn": {
                        "type": "deepr.readers.TFRecordReader",
                        "path": "viewfs://root/user/deepr/dev/example/data.tfrecord",
                        "num_parallel_reads": 8,
                        "num_parallel_calls": 8,
                        "shuffle": True,
                    },
                },
                "config": {
                    "type": "deepr.jobs.YarnTrainerConfig"
                },
            },
        ],
    },
    "config": {
        "type": "deepr.jobs.YarnLauncherConfig",
        "path_pex_prefix": "viewfs://root/user/deepr/dev/example/envs"
    },
}

Once the learning rate value is a reference to a macro parameter, if we define the following macros, we can parse the config and fill the parameter values.

We also used a dynamic macro to set the path to the model dynamically (every experiment needs to use a different model path).

import datetime
import time

class Paths(dict):
    """Macro that generates new path_model based on date."""

    def __init__(self, path_model: str = None, path_dataset: str = None):
        now = datetime.datetime.now().strftime("%Y-%m-%d-%H-%M-%S")
        if path_model is None:
            path_model = f"viewfs://root/user/deepr/dev/example/models/{now}"
        if path_dataset is None:
            path_dataset = f"viewfs://root/user/deepr/dev/example/data.tfrecord"
        super().__init__(path_model=path_model, path_dataset=path_dataset)

print(Paths())
time.sleep(2)
print(Paths())  # Creates new paths

{'path_model': 'viewfs://root/user/deepr/dev/example/models/2020-06-03-15-45-11', 'path_dataset': 'viewfs://root/user/deepr/dev/example/data.tfrecord'}
{'path_model': 'viewfs://root/user/deepr/dev/example/models/2020-06-03-15-45-13', 'path_dataset': 'viewfs://root/user/deepr/dev/example/data.tfrecord'}

macros = {
    "paths": {
        "type": "__main__.Paths"
    },
    "params": {
        "learning_rate": 0.01,
        "batch_size": 16
    }
}
parsed = deepr.parse_config(config, macros)
print(parsed["job"]["jobs"][1]["trainer"]["path_model"])
print(parsed["job"]["jobs"][1]["trainer"]["optimizer_fn"]["learning_rate"])
print(parsed["job"]["jobs"][1]["trainer"]["prepro_fn"]["batch_size"])

WARNING:deepr.config.base:- MACRO PARAM NOT USED: macro = 'paths', param = 'path_dataset'
viewfs://root/user/deepr/dev/example/models/2020-06-03-15-45-13
0.01
16

Launch

Now that the config is prepared, we can launch hyper parameter tuning using the ParamsSampler job.

The only thing that it does is sampling some learning rate values and use them as macros.

First, let’s define a sampler for the parameters

param_grid = {
    "learning_rate": np.logspace(start=-3, stop=-1, num=10),
    "batch_size": [8, 16, 32, 64],
}
sampler = deepr.jobs.ParamsSampler(param_grid, n_iter=5, seed=42)

for params in sampler:
    print(params)

INFO:deepr.jobs.params_tuner:Sampling with no replacement (parameter grid only has lists of values)
INFO:deepr.jobs.params_tuner:Sampled 5 parameters from a total of 40
{'batch_size': 64, 'learning_rate': 0.05994842503189409}
{'batch_size': 32, 'learning_rate': 0.05994842503189409}
{'batch_size': 16, 'learning_rate': 0.007742636826811269}
{'batch_size': 8, 'learning_rate': 0.03593813663804626}
{'batch_size': 32, 'learning_rate': 0.001}

and use the sampler with our config and macros to launch hyper params tuning with

HAS_HADOOP = False

if HAS_HADOOP:
    tuner = deepr.jobs.ParamsTuner(job=config, macros=macros, sampler=sampler)
    tuner.run()