Running an Experiment from Command Line (Python Wrapper)

This is a small toy example to showcase how to start a Experiment with a Wrapper in Python from command line

Given a Python wrapper:

wrapper.py

from __future__ import annotations

from ax import Trial
from stand_alone_model_func import run_branin_from_trial

import boa


class WrapperStandAlone(boa.BaseWrapper):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.data = {}

    def run_model(self, trial: Trial) -> None:
        self.data[trial.index] = run_branin_from_trial(trial)

    def set_trial_status(self, trial: Trial) -> None:
        data_exists = self.data.get(trial.index) is not None
        if data_exists:
            trial.mark_completed()

    def fetch_trial_data(self, trial: Trial, metric_properties: dict, metric_name: str, *args, **kwargs) -> dict:
        # return dict(a=self.data[trial.index])
        return {
            "Mean": {"a": list(trial.arm.parameters.values()), "sem": 4.5},
            "RMSE": {
                "y_true": [1.12, 1.25],
                "y_pred": list(trial.arm.parameters.values()),
            },
        }

and some “model” it is running (in this case it is just calling a synthetic function in python):

stand_alone_model_func.py

from ax import Trial
from ax.utils.measurement.synthetic_functions import branin


def run_branin_from_trial(trial: Trial) -> float:
    x0, x1 = trial.arm.parameters.get("x0"), trial.arm.parameters.get("x1")
    result = branin(x0, x1)
    return result

as well as a configuration file:

config.py

# Single objective optimization config
optimization_options:
    objective_options:
        objectives:
            - name: Cosine8
    trials: 50
    append_timestamp: False
parameters:
    x0:
        type: range
        bounds: [0.0, 1.0]
    x1:
        type: range
        bounds: [0.0, 1.0]
    x2:
        type: range
        bounds: [0.0, 1.0]
    x3:
        type: range
        bounds: [0.0, 1.0]
    x4:
        type: range
        bounds: [0.0, 1.0]
    x5:
        type: range
        bounds: [0.0, 1.0]
    x6:
        type: range
        bounds: [0.0, 1.0]
    x7:
        type: range
        bounds: [0.0, 1.0]
# These are all defaults, so we don't need to specify them in this case
#script_options:
#    wrapper_path: ./wrapper.py
#    wrapper_name: Wrapper
#    working_dir: .
#    experiment_dir: ... # this is where boa will write logs to by default
                         # if not specified it will be working_dir/experiment_name
#    append_timestamp: True
# This last option appends a timestamp to our output experiment directory.
# This is also the default (True)

You can start and run your optimization like this:

$ python -m boa -c config.json
Start time: 20221026T210522
[INFO 10-26 21:05:22] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x0. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 10-26 21:05:22] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x1. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 10-26 21:05:22] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[RangeParameter(name='x0', parameter_type=FLOAT, range=[-5.0, 10.0]), RangeParameter(name='x1', parameter_type=FLOAT, range=[0.0, 15.0])], parameter_constraints=[]).
[INFO 10-26 21:05:22] Scheduler: `Scheduler` requires experiment to have immutable search space and optimization config. Setting property immutable_search_space_and_opt_config to `True` on experiment.
[INFO 10-26 21:05:22] Scheduler: Running trials [0]...
[INFO 10-26 21:05:23] Scheduler: Running trials [1]...
[INFO 10-26 21:05:24] Scheduler: Generated all trials that can be generated currently. Model requires more data to generate more trials.
[INFO 10-26 21:05:24] Scheduler: Retrieved COMPLETED trials: 0 - 1.
[INFO 10-26 21:05:24] Scheduler: Fetching data for trials: 0 - 1.
[INFO 10-26 21:05:24] Scheduler: Running trials [2]...
[INFO 10-26 21:05:25] Scheduler: Running trials [3]...
[INFO 10-26 21:05:25] Scheduler: Running trials [4]...
[INFO 10-26 21:05:26] Scheduler: Retrieved COMPLETED trials: 2 - 4.
[INFO 10-26 21:05:26] Scheduler: Fetching data for trials: 2 - 4.

Trials completed! Total run time: 3
Saved JSON-serialized state of optimization to `/path/to/working/dir/boa_runs_20221026T210522/scheduler.json`.