Search Space and Decorator

This tutorial explains the supported search spaces and how to use them, including simple search spaces (Int, Real, Bool and Choice) and nested search spaces (Choice, List, Dict). Each search space describes the set of possible values for a hyperparameter, from which the searcher will try particular values during hyperparameter optimization. AutoTorch also enables search spaces in user-defined objects using the decorator at.obj and user-defined functions using the decorator at.func.

import autotorch as at

Search Space

Simple Search Space

• Integer Space autotorch.space.Int

An integer is chosen between lower and upper value during the searcher sampling.

a = at.Int(lower=0, upper=10)
print(a)

Out:

Int: lower=0, upper=10

Get default value:

print(a.default)

Out:

5

Change default value, which is the first configuration that a random searcher autotorch.searcher.RandomSearcher will try:

a = at.Int(lower=0, upper=10, default=2)
print(a.default)

Out:

2

Pick a random value.

print(a.rand)

Out:

4

• Real Space autotorch.space.Real

A real number is chosen between lower and upper value during the searcher sampling.

b = at.Real(lower=1e-4, upper=1e-2)
print(b)

Out:

Real: lower=0.0001, upper=0.01

Real space in log scale:

c = at.Real(lower=1e-4, upper=1e-2, log=True)
print(c)

Out:

Real: lower=0.0001, upper=0.01

• Choice Space autotorch.space.Choice

Choice Space chooses one value from all the possible values during the searcher sampling.

d = at.Choice('Monday', 'Tuesday', 'Wednesday')
print(d)

Out:

Choice['Monday', 'Tuesday', 'Wednesday']

Nested Search Space

• Choice Space autotorch.space.Choice

Choice Space can also be used as a nested search space. For an example, see NestedExampleObj.

• List Space autotorch.space.List

List Space returns a list of sampled results. In this example, the first element of the list is a Int Space sampled from 0 to 3, and the second element is a Choice Space sampled from the choices of ‘alpha’ and ‘beta’.

f = at.List(
        at.Int(0, 3),
        at.Choice('alpha', 'beta'),
    )
print(f)

Out:

List[Int: lower=0, upper=3, Choice['alpha', 'beta']]

Get one example configuration:

print(f.rand)

Out:

[3, 'beta']

• Dict Space autotorch.space.Dict

Dict Space returns a dict of sampled results. Similar to List Space, the resulting configuraton of Dict is a dict. In this example, the value of ‘key1’ is sampled from a Choice Space with the choices of ‘alpha’ and ‘beta’, and the value of ‘key2’ is sampled from an Int Space between 0 and 3.

g = at.Dict(
        key1=at.Choice('alpha', 'beta'),
        key2=at.Int(0, 3),
        key3='constant'
    )
print(g)

Out:

Dict{'key1': Choice['alpha', 'beta'], 'key2': Int: lower=0, upper=3, 'key3': 'constant'}

Get one example configuration:

print(g.rand)

Out:

{'key1': 'beta', 'key2': 3, 'key3': 'constant'}

Decorators for Searchbale Object and Customized Training Scripts

In this section, we show how to insert search space into customized objects and training functions.

• Searchable Space in Customized Class autotorch.obj()

In AutoTorch searchable object can be returned by a user defined class with a decorator.

@at.obj(
    name=at.Choice('auto', 'gluon'),
    static_value=10,
    rank=at.Int(2, 5),
)
class MyObj:
    def __init__(self, name, rank, static_value):
        self.name = name
        self.rank = rank
        self.static_value = static_value
    def __repr__(self):
        repr = 'MyObj -- name: {}, rank: {}, static_value: {}'.format(
                self.name, self.rank, self.static_value)
        return repr
h = MyObj()
print(h)

Out:

AutoTorchObject -- MyObj

Get one example random object:

print(h.rand)

Out:

MyObj -- name: gluon, rank: 2, static_value: 10

We can also use it within a Nested Space such as autotorch.space.Choice. In this example, the resulting nested space will be sampled from:

nested = at.Choice(
        at.Dict(
                obj1='1',
                obj2=at.Choice('a', 'b'),
            ),
        MyObj(),
    )

print(nested)

Out:

Choice[Dict{'obj1': '1', 'obj2': Choice['a', 'b']}, AutoTorchObject -- MyObj]

Get an example output:

for _ in range(5):
    result = nested.rand
    assert (isinstance(result, dict) and result['obj2'] in ['a', 'b']) or hasattr(result, 'name')
    print(result)

Out:

MyObj -- name: auto, rank: 5, static_value: 10
{'obj1': '1', 'obj2': 'b'}
MyObj -- name: gluon, rank: 3, static_value: 10
MyObj -- name: gluon, rank: 5, static_value: 10
MyObj -- name: gluon, rank: 5, static_value: 10

• Searchable Space in Customized Function autotorch.obj()

We can also insert a searchable space in a customized function:

@at.func(
    framework=at.Choice('mxnet', 'pytorch'),
)
def myfunc(framework):
    return framework
i = myfunc()
print(i)

Out:

AutoTorchObject

We can also put a searchable space inside a nested space:

j = at.Dict(
        a=at.Real(0, 10),
        obj1=MyObj(),
        obj2=myfunc(),
    )
print(j)

Out:

Dict{'a': Real: lower=0, upper=10, 'obj1': AutoTorchObject -- MyObj, 'obj2': AutoTorchObject}

• Customized Train Script Using autotorch.args()

train_func is where to put your model training script, which takes in various keyword args as its hyperparameters and reports the performance of the trained model using the provided reporter. Here, we show a dummy train_func that simply prints these objects.

@at.args(
    a=at.Int(1, 10),
    b=at.Real(1e-3, 1e-2),
    c=at.Real(1e-3, 1e-2, log=True),
    d=at.Choice('a', 'b', 'c', 'd'),
    e=at.Bool(),
    f=at.List(
            at.Int(1, 2),
            at.Choice(4, 5),
        ),
    g=at.Dict(
            a=at.Real(0, 10),
            obj=MyObj(),
        ),
    h=at.Choice('test', MyObj()),
    i = myfunc(),
)
def train_fn(args, reporter):
    print('args: {}'.format(args))

Create Searcher and Run with a Configuration

In this section, we create a Searcher object, which orchestrates a particular hyperparameter-tuning strategy.

• Create a Searcher and Sample Configuration

searcher = at.searcher.RandomSearcher(train_fn.cs)
config = searcher.get_config()
print(config)

Out:

{'a': 6, 'b': 0.0055, 'c': 0.0031622777, 'd.choice': 0, 'e.choice': 0, 'f.0': 2, 'f.1.choice': 0, 'g.a': 5.0, 'g.obj.name.choice': 0, 'g.obj.rank': 4, 'h.1.name.choice': 0, 'h.1.rank': 4, 'h.choice': 0, 'i.framework.choice': 0}

• Run one training job with the sampled configuration:

train_fn(train_fn.args, config)

Out:

args: {'a': 6, 'b': 0.0055, 'c': 0.0031622777, 'd': 'a', 'e': True, 'f': [2, 4], 'g': {'a': 5.0, 'obj': MyObj -- name: auto, rank: 4, static_value: 10}, 'h': 'test', 'i': 'mxnet', '_default_config': {}}