Hyperparameter Sweeper

This page explains how to use Doodad's built-in hyperparameter sweeper.

Overview

Doodad's sweeper performs grid-searches over command-line arguments to a python script.

The first step is to create a python script which uses double-dash command-line arguments. This can be easily implemented using Python's built-in argparse module.

The next step is to create a separate python file which will perform the sweep when executed. This file should T create a DoodadSweeper object from the doodad.wrappers.sweeper.launch module. The DoodadSweeper object has a collection of methods which enable grid-searching locally or over GCP.

An Example

Let's use the following script which prints an integer passed as a command-line argument

import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--n', type=int, help='Integer to print')
args = parser.parse_args()
print(args.n)

We can write the following python script which will perform a sweep locally when executed

from doodad.wrappers.sweeper import launcher
sweeper = launcher.DoodadSweeper()
args = {
    'n': [1,3,5,7]
}
output = self.sweeper.run_sweep_local(
    target=<path to script>,
    params=args,
)

Chunking runs

Sometimes it is desirable to aggregate or chunk script executions onto a single machine when using a service such as GCP. For example, you could have 100 experiments but only want to launch 10 machines.

A simple way to handle this is to use the num_chunks argument to the methods of the DoodadSweeper object. The num_chunks argument tells doodad how many machines to use.

The current implementation of chunking has no load-balancing (it randomly allocates jobs to machines), so make sure all jobs take roughly an equal amount of time.