A conditional (hence more interesting) search space¶

Copyright (c) 2021 - for information on the respective copyright owner see the NOTICE file and/or the repository https://github.com/boschresearch/parameterspace

SPDX-License-Identifier: Apache-2.0

In [ ]:

import numpy as np
import matplotlib.pyplot as plt
import parameterspace as ps

import numpy as np import matplotlib.pyplot as plt import parameterspace as ps

As an example, consider optimizing the training parameters for a a neural network, namely which optimizer to use, and the parameters for each of them. The two optimizers are 'Adam' and 'SGD' with a learning rate each and a an additional momentum term for SGD.

In [ ]:

optimizer = ps.CategoricalParameter('optimizer', ['ADAM', 'SGD'])

lr_adam = ps.ContinuousParameter('lr_adam', bounds=[1e-5, 1e-3], 
                                transformation='log')

lr_sgd = ps.ContinuousParameter('lr_sgd', bounds=[1e-3, 1e-0], prior=ps.priors.TruncatedNormal(mean=0.5, std=0.2))

momentum_sgd = ps.ContinuousParameter('momentum', bounds=[0, 0.9])

optimizer = ps.CategoricalParameter('optimizer', ['ADAM', 'SGD']) lr_adam = ps.ContinuousParameter('lr_adam', bounds=[1e-5, 1e-3], transformation='log') lr_sgd = ps.ContinuousParameter('lr_sgd', bounds=[1e-3, 1e-0], prior=ps.priors.TruncatedNormal(mean=0.5, std=0.2)) momentum_sgd = ps.ContinuousParameter('momentum', bounds=[0, 0.9])

Now, we can create the space containing the parameters. The condition argument must be a callable that has its argument names be parameter names and returns True if the parameter should be active and False otherwise.

In [ ]:

s = ps.ParameterSpace()

s.add(optimizer)
s.add(lr_adam,      condition=lambda optimizer: optimizer=='ADAM')
s.add(lr_sgd,       condition=lambda optimizer: optimizer=='SGD')
s.add(momentum_sgd, condition=lambda optimizer: optimizer=='SGD')

print(s)

s = ps.ParameterSpace() s.add(optimizer) s.add(lr_adam, condition=lambda optimizer: optimizer=='ADAM') s.add(lr_sgd, condition=lambda optimizer: optimizer=='SGD') s.add(momentum_sgd, condition=lambda optimizer: optimizer=='SGD') print(s)

Now we can draw samples:

In [ ]:

for i in range(5):
    print(s.sample())

for i in range(5): print(s.sample())

If you are interested in random samples of a subspace, you can also give a partial assignment to the sample method to set those parameters manually. Here are some samples with ADAM as the optimizer:

In [ ]:

for i in range(5):
    print(s.sample({'optimizer': 'ADAM'}))

for i in range(5): print(s.sample({'optimizer': 'ADAM'}))

and here some with SGD as the optimizer:

In [ ]:

for i in range(5):
    print(s.sample({'optimizer': 'SGD'}))

for i in range(5): print(s.sample({'optimizer': 'SGD'}))

When working with these spaces, a purely numerical represetation can be accessed via the to_numerical method. Note that these values are in the transformed space and might contain special values (nan by default) for inactive values.

In [ ]:

sample = s.sample()
print(sample)
s.to_numerical(sample)

sample = s.sample() print(sample) s.to_numerical(sample)