Running BOA Optimization Directly in Python#
This notebook demonstrates how to:
Write a basic Wrapper in Python and launch a optimization from Python. If you wanted to launch it from command line, you would do a similar thing of defining the Wrapper, and then put in your configuration file the information about where the wrapper is, and use BOA’s CLI tools. See Running an Experiment from Command Line (Python Wrapper) for more information.
1import pathlib
2import shutil
3
4from IPython.display import Code
5from wrapper import Wrapper
6
7import boa
[WARNING 08-11 16:15:48] ax.service.utils.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
Show code cell content
1# Remove old runs to have a clean slate for this example
2old_runs = pathlib.Path().resolve().glob("boa_runs*")
3for path in old_runs:
4 shutil.rmtree(path)
Loading the Config File#
1config_path = pathlib.Path().resolve() / "single_config.yaml"
Here we can see what the configuration file looks like
1Code(config_path)
# Single objective optimization config
optimization_options:
objective_options:
objectives:
- name: Cosine8
trials: 50
append_timestamp: False
parameters:
x0:
type: range
bounds: [0.0, 1.0]
x1:
type: range
bounds: [0.0, 1.0]
x2:
type: range
bounds: [0.0, 1.0]
x3:
type: range
bounds: [0.0, 1.0]
x4:
type: range
bounds: [0.0, 1.0]
x5:
type: range
bounds: [0.0, 1.0]
x6:
type: range
bounds: [0.0, 1.0]
x7:
type: range
bounds: [0.0, 1.0]
# These are all defaults, so we don't need to specify them in this case
#script_options:
# wrapper_path: ./wrapper.py
# wrapper_name: Wrapper
# working_dir: .
# experiment_dir: ... # this is where boa will write logs to by default
# if not specified it will be working_dir/experiment_name
# append_timestamp: True
# This last option appends a timestamp to our output experiment directory.
# This is also the default (True)
we need the config normalized, which modifies the parameter section into a less user friendly form, but what the downstream libraries need
1config = boa.load_jsonlike(config_path)
Define Our Wrapper#
We define our wrapper in wrapper.py and use a synthetic function that stands in for any black box model call
1Code(Wrapper.path())
import numpy as np
from ax.utils.measurement.synthetic_functions import from_botorch
from botorch.test_functions.synthetic import Cosine8
import boa
cosine8 = from_botorch(Cosine8())
def black_box_model(X) -> float:
result = -cosine8(X)
return result
class Wrapper(boa.BaseWrapper):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.data = {}
def run_model(self, trial) -> None:
X = np.array([parameter for parameter in trial.arm.parameters.values()])
# This is a silly toy function, in reality,
# you could instead import your model main() function and use that, and then collect the results.
# You could also call an external script to start a model run from Bash or elsewhere.
self.data[trial.index] = black_box_model(X)
def set_trial_status(self, trial) -> None:
data_exists = self.data.get(trial.index)
if data_exists:
trial.mark_completed()
def fetch_trial_data(self, trial, *args, **kwargs):
return self.data[trial.index]
Initialize our Setup#
1controller = boa.Controller(config_path=config_path, wrapper=Wrapper)
2
3controller.initialize_scheduler()
[INFO 08-11 16:15:49] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x0. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 08-11 16:15:49] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x1. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 08-11 16:15:49] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x2. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 08-11 16:15:49] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x3. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 08-11 16:15:49] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x4. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 08-11 16:15:49] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x5. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 08-11 16:15:49] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x6. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 08-11 16:15:49] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x7. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 08-11 16:15:49] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[RangeParameter(name='x0', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x1', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x2', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x3', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x4', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x5', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x6', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x7', parameter_type=FLOAT, range=[0.0, 1.0])], parameter_constraints=[]).
[INFO 08-11 16:15:49] ax.modelbridge.dispatch_utils: Using Models.GPEI since there are more ordered parameters than there are categories for the unordered categorical parameters.
[INFO 08-11 16:15:49] ax.modelbridge.dispatch_utils: Calculating the number of remaining initialization trials based on num_initialization_trials=None max_initialization_trials=None num_tunable_parameters=8 num_trials=None use_batch_trials=False
[INFO 08-11 16:15:49] ax.modelbridge.dispatch_utils: calculated num_initialization_trials=16
[INFO 08-11 16:15:49] ax.modelbridge.dispatch_utils: num_completed_initialization_trials=0 num_remaining_initialization_trials=16
[INFO 08-11 16:15:49] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 16 trials, GPEI for subsequent trials]). Iterations after 16 will take longer to generate due to model-fitting.
[INFO 08-11 16:15:49] Scheduler: `Scheduler` requires experiment to have immutable search space and optimization config. Setting property immutable_search_space_and_opt_config to `True` on experiment.
(Scheduler(experiment=Experiment(boa_runs), generation_strategy=GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 16 trials, GPEI for subsequent trials]), options=SchedulerOptions(max_pending_trials=10, trial_type=<TrialType.TRIAL: 0>, batch_size=None, total_trials=None, tolerated_trial_failure_rate=0.5, min_failed_trials_for_failure_rate_check=5, log_filepath=None, logging_level=20, ttl_seconds_for_trials=None, init_seconds_between_polls=1, min_seconds_before_poll=1.0, seconds_between_polls_backoff_factor=1.5, timeout_hours=None, run_trials_in_batches=False, debug_log_run_metadata=False, early_stopping_strategy=None, global_stopping_strategy=None, suppress_storage_errors_after_retries=False)),
<wrapper.Wrapper at 0x7f9c3d03c670>)
Run our Experiment#
The Controller will save our scheduler to JSON after it completes the run so we can reload it at a later time for analysis or to resume our experiment
1scheduler = controller.run()
[INFO 2023-08-11 16:15:49,658 MainProcess] boa:
##############################################
BOA Experiment Run
Output Experiment Dir: /home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549
Start Time: 20230811T161549
Version: 0.8.9.dev0+gad156f0.d20230811
##############################################
[INFO 08-11 16:15:49] Scheduler: Running trials [0]...
[INFO 08-11 16:15:50] Scheduler: Running trials [1]...
[INFO 08-11 16:15:51] Scheduler: Running trials [2]...
[INFO 08-11 16:15:52] Scheduler: Running trials [3]...
[INFO 08-11 16:15:52] Scheduler: Running trials [4]...
[INFO 08-11 16:15:53] Scheduler: Running trials [5]...
[INFO 08-11 16:15:54] Scheduler: Running trials [6]...
[INFO 08-11 16:15:55] Scheduler: Running trials [7]...
[INFO 08-11 16:15:56] Scheduler: Running trials [8]...
[INFO 08-11 16:15:57] Scheduler: Running trials [9]...
[INFO 08-11 16:15:58] Scheduler: Retrieved COMPLETED trials: 0 - 9.
[INFO 08-11 16:15:58] Scheduler: Fetching data for trials: 0 - 9.
[ERROR 2023-08-11 16:15:58,885 MainProcess] boa: Object <wrapper.Wrapper object at 0x7f9c3d03c670> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-08-11 16:15:58,903 MainProcess] boa: Saved JSON-serialized state of optimization to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/scheduler.json`.
Boa version: 0.8.9.dev0+gad156f0.d20230811
[INFO 2023-08-11 16:15:58,924 MainProcess] boa: Saved optimization parametrization and objective to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/optimization.csv`.
[INFO 2023-08-11 16:15:58,940 MainProcess] boa: Trials so far: 10
Running trials:
Will Produce next trials from generation step: Sobol
Best trial so far: {2: {'Cosine8': 0.8790163185130037}}
[INFO 08-11 16:15:58] Scheduler: Running trials [10]...
[INFO 08-11 16:15:59] Scheduler: Running trials [11]...
[INFO 08-11 16:15:59] Scheduler: Running trials [12]...
[INFO 08-11 16:16:01] Scheduler: Running trials [13]...
[INFO 08-11 16:16:02] Scheduler: Running trials [14]...
[INFO 08-11 16:16:03] Scheduler: Running trials [15]...
[INFO 08-11 16:16:06] Scheduler: Running trials [16]...
[INFO 08-11 16:16:07] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:10] Scheduler: Running trials [17]...
[INFO 08-11 16:16:11] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:13] Scheduler: Running trials [18]...
[INFO 08-11 16:16:14] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:14] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 08-11 16:16:14] Scheduler: Retrieved COMPLETED trials: 10 - 18.
[INFO 08-11 16:16:14] Scheduler: Fetching data for trials: 10 - 18.
[ERROR 2023-08-11 16:16:14,654 MainProcess] boa: Object <wrapper.Wrapper object at 0x7f9c3d03c670> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-08-11 16:16:14,681 MainProcess] boa: Saved JSON-serialized state of optimization to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/scheduler.json`.
Boa version: 0.8.9.dev0+gad156f0.d20230811
[INFO 2023-08-11 16:16:14,701 MainProcess] boa: Saved optimization parametrization and objective to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/optimization.csv`.
[INFO 2023-08-11 16:16:14,720 MainProcess] boa: Trials so far: 19
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {17: {'Cosine8': 0.6906046755167087}}
[INFO 08-11 16:16:16] Scheduler: Running trials [19]...
[INFO 08-11 16:16:18] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:20] Scheduler: Running trials [20]...
[INFO 08-11 16:16:21] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:24] Scheduler: Running trials [21]...
[INFO 08-11 16:16:25] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:25] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 08-11 16:16:25] Scheduler: Retrieved COMPLETED trials: 19 - 21.
[INFO 08-11 16:16:25] Scheduler: Fetching data for trials: 19 - 21.
[ERROR 2023-08-11 16:16:25,787 MainProcess] boa: Object <wrapper.Wrapper object at 0x7f9c3d03c670> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-08-11 16:16:25,815 MainProcess] boa: Saved JSON-serialized state of optimization to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/scheduler.json`.
Boa version: 0.8.9.dev0+gad156f0.d20230811
[INFO 2023-08-11 16:16:25,835 MainProcess] boa: Saved optimization parametrization and objective to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/optimization.csv`.
[INFO 2023-08-11 16:16:25,856 MainProcess] boa: Trials so far: 22
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {21: {'Cosine8': 0.33052694694903173}}
[INFO 08-11 16:16:27] Scheduler: Running trials [22]...
[INFO 08-11 16:16:28] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:32] Scheduler: Running trials [23]...
[INFO 08-11 16:16:33] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:34] Scheduler: Running trials [24]...
[INFO 08-11 16:16:35] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:35] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 08-11 16:16:35] Scheduler: Retrieved COMPLETED trials: 22 - 24.
[INFO 08-11 16:16:35] Scheduler: Fetching data for trials: 22 - 24.
[ERROR 2023-08-11 16:16:35,999 MainProcess] boa: Object <wrapper.Wrapper object at 0x7f9c3d03c670> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-08-11 16:16:36,030 MainProcess] boa: Saved JSON-serialized state of optimization to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/scheduler.json`.
Boa version: 0.8.9.dev0+gad156f0.d20230811
[INFO 2023-08-11 16:16:36,053 MainProcess] boa: Saved optimization parametrization and objective to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/optimization.csv`.
[INFO 2023-08-11 16:16:36,074 MainProcess] boa: Trials so far: 25
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {22: {'Cosine8': 0.2561592013127164}}
[INFO 08-11 16:16:38] Scheduler: Running trials [25]...
[INFO 08-11 16:16:39] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:41] Scheduler: Running trials [26]...
[INFO 08-11 16:16:42] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:44] Scheduler: Running trials [27]...
[INFO 08-11 16:16:45] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:45] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 08-11 16:16:45] Scheduler: Retrieved COMPLETED trials: 25 - 27.
[INFO 08-11 16:16:45] Scheduler: Fetching data for trials: 25 - 27.
[ERROR 2023-08-11 16:16:45,616 MainProcess] boa: Object <wrapper.Wrapper object at 0x7f9c3d03c670> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-08-11 16:16:45,648 MainProcess] boa: Saved JSON-serialized state of optimization to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/scheduler.json`.
Boa version: 0.8.9.dev0+gad156f0.d20230811
[INFO 2023-08-11 16:16:45,670 MainProcess] boa: Saved optimization parametrization and objective to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/optimization.csv`.
[INFO 2023-08-11 16:16:45,691 MainProcess] boa: Trials so far: 28
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {25: {'Cosine8': 0.17913345640548606}}
[INFO 08-11 16:16:47] Scheduler: Running trials [28]...
[INFO 08-11 16:16:48] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:48] Scheduler: Running trials [29]...
[INFO 08-11 16:16:49] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:50] Scheduler: Running trials [30]...
[INFO 08-11 16:16:51] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:51] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 08-11 16:16:51] Scheduler: Retrieved COMPLETED trials: 28 - 30.
[INFO 08-11 16:16:51] Scheduler: Fetching data for trials: 28 - 30.
[ERROR 2023-08-11 16:16:51,936 MainProcess] boa: Object <wrapper.Wrapper object at 0x7f9c3d03c670> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-08-11 16:16:51,971 MainProcess] boa: Saved JSON-serialized state of optimization to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/scheduler.json`.
Boa version: 0.8.9.dev0+gad156f0.d20230811
[INFO 2023-08-11 16:16:51,993 MainProcess] boa: Saved optimization parametrization and objective to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/optimization.csv`.
[INFO 2023-08-11 16:16:52,015 MainProcess] boa: Trials so far: 31
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {30: {'Cosine8': 0.09079763023254955}}
[INFO 08-11 16:16:54] Scheduler: Running trials [31]...
[INFO 08-11 16:16:55] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:56] Scheduler: Running trials [32]...
[INFO 08-11 16:16:57] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:57] Scheduler: Running trials [33]...
[INFO 08-11 16:16:58] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:58] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 08-11 16:16:58] Scheduler: Retrieved COMPLETED trials: 31 - 33.
[INFO 08-11 16:16:58] Scheduler: Fetching data for trials: 31 - 33.
[ERROR 2023-08-11 16:16:58,766 MainProcess] boa: Object <wrapper.Wrapper object at 0x7f9c3d03c670> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-08-11 16:16:58,805 MainProcess] boa: Saved JSON-serialized state of optimization to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/scheduler.json`.
Boa version: 0.8.9.dev0+gad156f0.d20230811
[INFO 2023-08-11 16:16:58,828 MainProcess] boa: Saved optimization parametrization and objective to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/optimization.csv`.
[INFO 2023-08-11 16:16:59,007 MainProcess] boa: Trials so far: 34
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {31: {'Cosine8': -0.005139013670427406}}
[INFO 08-11 16:16:59] Scheduler: Running trials [34]...
[INFO 08-11 16:16:59] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:16:59] Scheduler: Running trials [35]...
[INFO 08-11 16:17:00] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:17:03] Scheduler: Running trials [36]...
[INFO 08-11 16:17:04] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:17:04] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 08-11 16:17:04] Scheduler: Retrieved COMPLETED trials: 34 - 36.
[INFO 08-11 16:17:04] Scheduler: Fetching data for trials: 34 - 36.
[ERROR 2023-08-11 16:17:04,035 MainProcess] boa: Object <wrapper.Wrapper object at 0x7f9c3d03c670> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-08-11 16:17:04,075 MainProcess] boa: Saved JSON-serialized state of optimization to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/scheduler.json`.
Boa version: 0.8.9.dev0+gad156f0.d20230811
[INFO 2023-08-11 16:17:04,098 MainProcess] boa: Saved optimization parametrization and objective to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/optimization.csv`.
[INFO 2023-08-11 16:17:04,122 MainProcess] boa: Trials so far: 37
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {36: {'Cosine8': -0.0388514385293863}}
/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/conda/0.8.8/lib/python3.10/site-packages/botorch/optim/initializers.py:403: BadInitialCandidatesWarning: Unable to find non-zero acquisition function values - initial conditions are being selected randomly.
warnings.warn(
[INFO 08-11 16:17:04] Scheduler: Running trials [37]...
[INFO 08-11 16:17:05] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:17:06] Scheduler: Running trials [38]...
[INFO 08-11 16:17:07] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:17:07] Scheduler: Running trials [39]...
[INFO 08-11 16:17:08] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:17:08] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 08-11 16:17:08] Scheduler: Retrieved COMPLETED trials: 37 - 39.
[INFO 08-11 16:17:08] Scheduler: Fetching data for trials: 37 - 39.
[ERROR 2023-08-11 16:17:08,668 MainProcess] boa: Object <wrapper.Wrapper object at 0x7f9c3d03c670> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-08-11 16:17:08,712 MainProcess] boa: Saved JSON-serialized state of optimization to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/scheduler.json`.
Boa version: 0.8.9.dev0+gad156f0.d20230811
[INFO 2023-08-11 16:17:08,736 MainProcess] boa: Saved optimization parametrization and objective to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/optimization.csv`.
[INFO 2023-08-11 16:17:08,761 MainProcess] boa: Trials so far: 40
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {36: {'Cosine8': -0.0388514385293863}}
[INFO 08-11 16:17:09] Scheduler: Running trials [40]...
[INFO 08-11 16:17:10] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:17:10] Scheduler: Running trials [41]...
[INFO 08-11 16:17:11] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:17:16] Scheduler: Running trials [42]...
[INFO 08-11 16:17:17] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:17:17] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 08-11 16:17:17] Scheduler: Retrieved COMPLETED trials: 40 - 42.
[INFO 08-11 16:17:17] Scheduler: Fetching data for trials: 40 - 42.
[ERROR 2023-08-11 16:17:17,483 MainProcess] boa: Object <wrapper.Wrapper object at 0x7f9c3d03c670> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-08-11 16:17:17,530 MainProcess] boa: Saved JSON-serialized state of optimization to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/scheduler.json`.
Boa version: 0.8.9.dev0+gad156f0.d20230811
[INFO 2023-08-11 16:17:17,556 MainProcess] boa: Saved optimization parametrization and objective to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/optimization.csv`.
[INFO 2023-08-11 16:17:17,582 MainProcess] boa: Trials so far: 43
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {36: {'Cosine8': -0.0388514385293863}}
[INFO 08-11 16:17:17] Scheduler: Running trials [43]...
[INFO 08-11 16:17:19] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:17:19] Scheduler: Running trials [44]...
[INFO 08-11 16:17:20] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:17:22] Scheduler: Running trials [45]...
[INFO 08-11 16:17:23] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:17:23] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 08-11 16:17:23] Scheduler: Retrieved COMPLETED trials: 43 - 45.
[INFO 08-11 16:17:23] Scheduler: Fetching data for trials: 43 - 45.
[ERROR 2023-08-11 16:17:23,776 MainProcess] boa: Object <wrapper.Wrapper object at 0x7f9c3d03c670> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-08-11 16:17:23,826 MainProcess] boa: Saved JSON-serialized state of optimization to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/scheduler.json`.
Boa version: 0.8.9.dev0+gad156f0.d20230811
[INFO 2023-08-11 16:17:23,853 MainProcess] boa: Saved optimization parametrization and objective to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/optimization.csv`.
[INFO 2023-08-11 16:17:23,880 MainProcess] boa: Trials so far: 46
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {36: {'Cosine8': -0.0388514385293863}}
[INFO 08-11 16:17:25] Scheduler: Running trials [46]...
[INFO 08-11 16:17:26] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:17:26] Scheduler: Running trials [47]...
[INFO 08-11 16:17:28] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:17:32] Scheduler: Running trials [48]...
[INFO 08-11 16:17:33] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 08-11 16:17:33] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 08-11 16:17:33] Scheduler: Retrieved COMPLETED trials: 46 - 48.
[INFO 08-11 16:17:33] Scheduler: Fetching data for trials: 46 - 48.
[ERROR 2023-08-11 16:17:33,995 MainProcess] boa: Object <wrapper.Wrapper object at 0x7f9c3d03c670> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-08-11 16:17:34,048 MainProcess] boa: Saved JSON-serialized state of optimization to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/scheduler.json`.
Boa version: 0.8.9.dev0+gad156f0.d20230811
[INFO 2023-08-11 16:17:34,076 MainProcess] boa: Saved optimization parametrization and objective to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/optimization.csv`.
[INFO 2023-08-11 16:17:34,106 MainProcess] boa: Trials so far: 49
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {36: {'Cosine8': -0.0388514385293863}}
[INFO 08-11 16:17:34] Scheduler: Running trials [49]...
[INFO 08-11 16:17:35] Scheduler: Retrieved COMPLETED trials: [49].
[INFO 08-11 16:17:35] Scheduler: Fetching data for trials: [49].
[ERROR 2023-08-11 16:17:35,924 MainProcess] boa: Object <wrapper.Wrapper object at 0x7f9c3d03c670> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-08-11 16:17:35,978 MainProcess] boa: Saved JSON-serialized state of optimization to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/scheduler.json`.
Boa version: 0.8.9.dev0+gad156f0.d20230811
[INFO 2023-08-11 16:17:36,005 MainProcess] boa: Saved optimization parametrization and objective to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/optimization.csv`.
[INFO 2023-08-11 16:17:36,034 MainProcess] boa: Trials so far: 50
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {36: {'Cosine8': -0.0388514385293863}}
[ERROR 2023-08-11 16:17:36,036 MainProcess] boa: Object <wrapper.Wrapper object at 0x7f9c3d03c670> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-08-11 16:17:36,090 MainProcess] boa: Saved JSON-serialized state of optimization to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/scheduler.json`.
Boa version: 0.8.9.dev0+gad156f0.d20230811
[INFO 2023-08-11 16:17:36,117 MainProcess] boa: Saved optimization parametrization and objective to `/home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549/optimization.csv`.
[INFO 2023-08-11 16:17:36,148 MainProcess] boa: Trials so far: 50
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {36: {'Cosine8': -0.0388514385293863}}
[INFO 2023-08-11 16:17:36,181 MainProcess] boa:
##############################################
Trials Completed!
BOA Experiment Run
Output Experiment Dir: /home/docs/checkouts/readthedocs.org/user_builds/boa-framework/checkouts/0.8.8/docs/examples/boa_runs_20230811T161549
Start Time: 20230811T161549
Version: 0.8.9.dev0+gad156f0.d20230811
End Time: 20230811T161736
Total Run Time: 106.49058032035828
trial_index arm_name trial_status generation_method Cosine8 x0 \
0 0 0_0 COMPLETED Sobol 1.712687 0.390290
1 1 1_0 COMPLETED Sobol 2.113637 0.501035
2 2 2_0 COMPLETED Sobol 0.879016 0.314489
3 3 3_0 COMPLETED Sobol 2.461406 0.009065
4 4 4_0 COMPLETED Sobol 3.874045 0.987132
5 5 5_0 COMPLETED Sobol 1.762933 0.422476
6 6 6_0 COMPLETED Sobol 2.710206 0.768410
7 7 7_0 COMPLETED Sobol 2.154743 0.151266
8 8 8_0 COMPLETED Sobol 3.142967 0.763789
9 9 9_0 COMPLETED Sobol 1.735314 0.087846
10 10 10_0 COMPLETED Sobol 3.017592 0.740672
11 11 11_0 COMPLETED Sobol 2.352555 0.164585
12 12 12_0 COMPLETED Sobol 1.893150 0.528397
13 13 13_0 COMPLETED Sobol 1.986891 0.442737
14 14 14_0 COMPLETED Sobol 3.422707 0.717825
15 15 15_0 COMPLETED Sobol 2.903469 0.193453
16 16 16_0 COMPLETED GPEI 0.865440 0.223493
17 17 17_0 COMPLETED GPEI 0.690605 0.291693
18 18 18_0 COMPLETED GPEI 0.766902 0.405620
19 19 19_0 COMPLETED GPEI 0.760668 0.306687
20 20 20_0 COMPLETED GPEI 0.908515 0.262575
21 21 21_0 COMPLETED GPEI 0.330527 0.327506
22 22 22_0 COMPLETED GPEI 0.256159 0.290656
23 23 23_0 COMPLETED GPEI 0.261850 0.360845
24 24 24_0 COMPLETED GPEI 0.406267 0.337497
25 25 25_0 COMPLETED GPEI 0.179133 0.278791
26 26 26_0 COMPLETED GPEI 0.245168 0.398393
27 27 27_0 COMPLETED GPEI 0.397844 0.205403
28 28 28_0 COMPLETED GPEI 0.143550 0.411683
29 29 29_0 COMPLETED GPEI 0.522042 0.341522
30 30 30_0 COMPLETED GPEI 0.090798 0.368755
31 31 31_0 COMPLETED GPEI -0.005139 0.399175
32 32 32_0 COMPLETED GPEI 0.163411 0.422597
33 33 33_0 COMPLETED GPEI 0.010007 0.359553
34 34 34_0 COMPLETED GPEI 0.016386 0.392692
35 35 35_0 COMPLETED GPEI 0.374686 0.508453
36 36 36_0 COMPLETED GPEI -0.038851 0.374380
37 37 37_0 COMPLETED GPEI 1.809430 0.039100
38 38 38_0 COMPLETED GPEI -0.032717 0.349761
39 39 39_0 COMPLETED GPEI 1.220288 0.470744
40 40 40_0 COMPLETED GPEI -0.023780 0.345149
41 41 41_0 COMPLETED GPEI 0.331096 0.205150
42 42 42_0 COMPLETED GPEI 0.000612 0.359848
43 43 43_0 COMPLETED GPEI 0.402406 0.076097
44 44 44_0 COMPLETED GPEI 0.346127 0.208505
45 45 45_0 COMPLETED GPEI 0.022018 0.026074
46 46 46_0 COMPLETED GPEI 0.180551 0.000000
47 47 47_0 COMPLETED GPEI 0.043655 0.095179
48 48 48_0 COMPLETED GPEI 0.248907 0.017847
49 49 49_0 COMPLETED GPEI 0.167933 0.017785
x1 x2 x3 x4 x5 x6 x7
0 0.128737 0.672232 0.111762 0.213507 0.166561 0.258514 0.862686
1 0.733351 0.138216 0.172541 0.111987 0.917452 0.465467 0.164496
2 0.209327 0.422130 0.037070 0.324276 0.090143 0.281985 0.701065
3 0.971256 0.489229 0.820029 0.335293 0.839587 0.026630 0.248004
4 0.164614 0.780826 0.419884 0.144257 0.975944 0.477668 0.888118
5 0.756627 0.214778 0.043767 0.107372 0.189920 0.606508 0.699061
6 0.111904 0.449029 0.609187 0.791032 0.585851 0.298089 0.783595
7 0.776312 0.251402 0.837112 0.632600 0.397143 0.165105 0.209921
8 0.026861 0.929300 0.341836 0.359111 0.984133 0.676420 0.464356
9 0.234065 0.715403 0.184734 0.610452 0.349489 0.080156 0.718453
10 0.524535 0.662386 0.213563 0.477111 0.972533 0.491618 0.267497
11 0.834657 0.244007 0.881570 0.150837 0.149736 0.766940 0.393923
12 0.664647 0.108174 0.018739 0.911949 0.299132 0.215472 0.403008
13 0.745947 0.469390 0.562906 0.493069 0.667136 0.392450 0.107307
14 0.872024 0.316665 0.300860 0.817357 0.693570 0.427347 0.926557
15 0.220182 0.117399 0.622002 0.573001 0.414348 0.789823 0.991669
16 0.357553 0.397169 0.039202 0.343562 0.123124 0.294615 0.666200
17 0.084200 0.390006 0.008725 0.426691 0.045908 0.220232 0.660771
18 0.268058 0.333751 0.018938 0.226954 0.053348 0.367503 0.699844
19 0.208698 0.258622 0.000000 0.378120 0.000000 0.281497 0.592491
20 0.189549 0.225527 0.000000 0.433665 0.000000 0.275423 0.748866
21 0.131555 0.357889 0.000000 0.362042 0.054215 0.330210 0.481164
22 0.035522 0.369940 0.000000 0.420593 0.095932 0.414831 0.450669
23 0.097055 0.393667 0.000000 0.311247 0.109593 0.297501 0.344570
24 0.154169 0.451355 0.000000 0.441278 0.009648 0.421739 0.441011
25 0.034964 0.399562 0.000000 0.273749 0.078127 0.396755 0.363300
26 0.070583 0.371780 0.000000 0.446300 0.125429 0.361314 0.383940
27 0.082482 0.414084 0.000000 0.441388 0.116547 0.311011 0.323645
28 0.000000 0.365726 0.000000 0.274661 0.070305 0.430986 0.347330
29 0.135406 0.375430 0.000000 0.238825 0.133157 0.473863 0.342245
30 0.000000 0.376500 0.076769 0.322159 0.055658 0.389568 0.331238
31 0.000000 0.400449 0.012998 0.352321 0.039491 0.351413 0.353143
32 0.000000 0.427618 0.060496 0.298382 0.061234 0.367861 0.392970
33 0.000000 0.337570 0.000000 0.362450 0.031914 0.360043 0.297296
34 0.000000 0.397354 0.000000 0.382050 0.008916 0.346636 0.290461
35 0.000000 0.385156 0.000000 0.455517 0.000000 0.315004 0.250628
36 0.000000 0.375906 0.000000 0.370218 0.025660 0.344407 0.345649
37 0.330390 0.738861 0.346845 0.329358 0.230040 0.766101 0.820291
38 0.000000 0.386191 0.000000 0.331044 0.000000 0.348385 0.360020
39 0.267233 0.657531 0.610199 0.029554 0.040163 0.234528 0.085158
40 0.000000 0.395652 0.000000 0.374382 0.026744 0.340384 0.351266
41 0.000000 0.473549 0.102291 0.087878 0.000000 0.184055 0.135849
42 0.000000 0.377112 0.000000 0.325162 0.022951 0.316742 0.343676
43 0.000000 0.550396 0.271479 0.163713 0.000000 0.348159 0.081873
44 0.000000 0.675748 0.000000 0.000000 0.000000 0.312995 0.112425
45 0.000000 0.424845 0.311036 0.029792 0.000000 0.270044 0.273023
46 0.000000 0.517746 0.168016 0.000000 0.000000 0.268264 0.357308
47 0.000000 0.359911 0.222143 0.000000 0.000000 0.334821 0.300010
48 0.000000 0.561075 0.371528 0.000000 0.000000 0.182111 0.320720
49 0.000000 0.341444 0.291967 0.000000 0.121951 0.249804 0.257908
##############################################
Get the Best Trial and Output All Trials#
best_fitted_trials uses the data to do a fitting from all trials and with the noise levels you provided (or if no noise levels was provided, it assumed an unknown level of noise and inferred the noise level from the trial runs)
1trial = scheduler.best_fitted_trials()
2trial
{49: {'params': {'x0': 0.3743796788867365,
'x1': 0.0,
'x2': 0.37590560503136833,
'x3': 0.0,
'x4': 0.37021832484439465,
'x5': 0.025659974222440454,
'x6': 0.34440669394553197,
'x7': 0.34564934741717407},
'means': {'Cosine8': -0.0387303369565013},
'cov_matrix': {'Cosine8': {'Cosine8': 1.1804716789221344e-06}}}}
if you need the exact points of the best trial, maybe because you need the trial number of the best trial to plot results, or for any other reason, best_raw_trails does not do any fitting
1trial = scheduler.best_raw_trials()
2trial
{36: {'params': {'x0': 0.3743796788867365,
'x1': 0.0,
'x2': 0.37590560503136833,
'x3': 0.0,
'x4': 0.37021832484439465,
'x5': 0.025659974222440454,
'x6': 0.34440669394553197,
'x7': 0.34564934741717407},
'means': {'Cosine8': -0.0388514385293863},
'cov_matrix': {'Cosine8': {'Cosine8': 0.0}}}}
Output a DataFrame of All Trials#
1boa.scheduler_to_df(scheduler)
| trial_index | arm_name | trial_status | generation_method | Cosine8 | x0 | x1 | x2 | x3 | x4 | x5 | x6 | x7 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0_0 | COMPLETED | Sobol | 1.712687 | 0.390290 | 0.128737 | 0.672232 | 0.111762 | 0.213507 | 0.166561 | 0.258514 | 0.862686 |
| 1 | 1 | 1_0 | COMPLETED | Sobol | 2.113637 | 0.501035 | 0.733351 | 0.138216 | 0.172541 | 0.111987 | 0.917452 | 0.465467 | 0.164496 |
| 2 | 2 | 2_0 | COMPLETED | Sobol | 0.879016 | 0.314489 | 0.209327 | 0.422130 | 0.037070 | 0.324276 | 0.090143 | 0.281985 | 0.701065 |
| 3 | 3 | 3_0 | COMPLETED | Sobol | 2.461406 | 0.009065 | 0.971256 | 0.489229 | 0.820029 | 0.335293 | 0.839587 | 0.026630 | 0.248004 |
| 4 | 4 | 4_0 | COMPLETED | Sobol | 3.874045 | 0.987132 | 0.164614 | 0.780826 | 0.419884 | 0.144257 | 0.975944 | 0.477668 | 0.888118 |
| 5 | 5 | 5_0 | COMPLETED | Sobol | 1.762933 | 0.422476 | 0.756627 | 0.214778 | 0.043767 | 0.107372 | 0.189920 | 0.606508 | 0.699061 |
| 6 | 6 | 6_0 | COMPLETED | Sobol | 2.710206 | 0.768410 | 0.111904 | 0.449029 | 0.609187 | 0.791032 | 0.585851 | 0.298089 | 0.783595 |
| 7 | 7 | 7_0 | COMPLETED | Sobol | 2.154743 | 0.151266 | 0.776312 | 0.251402 | 0.837112 | 0.632600 | 0.397143 | 0.165105 | 0.209921 |
| 8 | 8 | 8_0 | COMPLETED | Sobol | 3.142967 | 0.763789 | 0.026861 | 0.929300 | 0.341836 | 0.359111 | 0.984133 | 0.676420 | 0.464356 |
| 9 | 9 | 9_0 | COMPLETED | Sobol | 1.735314 | 0.087846 | 0.234065 | 0.715403 | 0.184734 | 0.610452 | 0.349489 | 0.080156 | 0.718453 |
| 10 | 10 | 10_0 | COMPLETED | Sobol | 3.017592 | 0.740672 | 0.524535 | 0.662386 | 0.213563 | 0.477111 | 0.972533 | 0.491618 | 0.267497 |
| 11 | 11 | 11_0 | COMPLETED | Sobol | 2.352555 | 0.164585 | 0.834657 | 0.244007 | 0.881570 | 0.150837 | 0.149736 | 0.766940 | 0.393923 |
| 12 | 12 | 12_0 | COMPLETED | Sobol | 1.893150 | 0.528397 | 0.664647 | 0.108174 | 0.018739 | 0.911949 | 0.299132 | 0.215472 | 0.403008 |
| 13 | 13 | 13_0 | COMPLETED | Sobol | 1.986891 | 0.442737 | 0.745947 | 0.469390 | 0.562906 | 0.493069 | 0.667136 | 0.392450 | 0.107307 |
| 14 | 14 | 14_0 | COMPLETED | Sobol | 3.422707 | 0.717825 | 0.872024 | 0.316665 | 0.300860 | 0.817357 | 0.693570 | 0.427347 | 0.926557 |
| 15 | 15 | 15_0 | COMPLETED | Sobol | 2.903469 | 0.193453 | 0.220182 | 0.117399 | 0.622002 | 0.573001 | 0.414348 | 0.789823 | 0.991669 |
| 16 | 16 | 16_0 | COMPLETED | GPEI | 0.865440 | 0.223493 | 0.357553 | 0.397169 | 0.039202 | 0.343562 | 0.123124 | 0.294615 | 0.666200 |
| 17 | 17 | 17_0 | COMPLETED | GPEI | 0.690605 | 0.291693 | 0.084200 | 0.390006 | 0.008725 | 0.426691 | 0.045908 | 0.220232 | 0.660771 |
| 18 | 18 | 18_0 | COMPLETED | GPEI | 0.766902 | 0.405620 | 0.268058 | 0.333751 | 0.018938 | 0.226954 | 0.053348 | 0.367503 | 0.699844 |
| 19 | 19 | 19_0 | COMPLETED | GPEI | 0.760668 | 0.306687 | 0.208698 | 0.258622 | 0.000000 | 0.378120 | 0.000000 | 0.281497 | 0.592491 |
| 20 | 20 | 20_0 | COMPLETED | GPEI | 0.908515 | 0.262575 | 0.189549 | 0.225527 | 0.000000 | 0.433665 | 0.000000 | 0.275423 | 0.748866 |
| 21 | 21 | 21_0 | COMPLETED | GPEI | 0.330527 | 0.327506 | 0.131555 | 0.357889 | 0.000000 | 0.362042 | 0.054215 | 0.330210 | 0.481164 |
| 22 | 22 | 22_0 | COMPLETED | GPEI | 0.256159 | 0.290656 | 0.035522 | 0.369940 | 0.000000 | 0.420593 | 0.095932 | 0.414831 | 0.450669 |
| 23 | 23 | 23_0 | COMPLETED | GPEI | 0.261850 | 0.360845 | 0.097055 | 0.393667 | 0.000000 | 0.311247 | 0.109593 | 0.297501 | 0.344570 |
| 24 | 24 | 24_0 | COMPLETED | GPEI | 0.406267 | 0.337497 | 0.154169 | 0.451355 | 0.000000 | 0.441278 | 0.009648 | 0.421739 | 0.441011 |
| 25 | 25 | 25_0 | COMPLETED | GPEI | 0.179133 | 0.278791 | 0.034964 | 0.399562 | 0.000000 | 0.273749 | 0.078127 | 0.396755 | 0.363300 |
| 26 | 26 | 26_0 | COMPLETED | GPEI | 0.245168 | 0.398393 | 0.070583 | 0.371780 | 0.000000 | 0.446300 | 0.125429 | 0.361314 | 0.383940 |
| 27 | 27 | 27_0 | COMPLETED | GPEI | 0.397844 | 0.205403 | 0.082482 | 0.414084 | 0.000000 | 0.441388 | 0.116547 | 0.311011 | 0.323645 |
| 28 | 28 | 28_0 | COMPLETED | GPEI | 0.143550 | 0.411683 | 0.000000 | 0.365726 | 0.000000 | 0.274661 | 0.070305 | 0.430986 | 0.347330 |
| 29 | 29 | 29_0 | COMPLETED | GPEI | 0.522042 | 0.341522 | 0.135406 | 0.375430 | 0.000000 | 0.238825 | 0.133157 | 0.473863 | 0.342245 |
| 30 | 30 | 30_0 | COMPLETED | GPEI | 0.090798 | 0.368755 | 0.000000 | 0.376500 | 0.076769 | 0.322159 | 0.055658 | 0.389568 | 0.331238 |
| 31 | 31 | 31_0 | COMPLETED | GPEI | -0.005139 | 0.399175 | 0.000000 | 0.400449 | 0.012998 | 0.352321 | 0.039491 | 0.351413 | 0.353143 |
| 32 | 32 | 32_0 | COMPLETED | GPEI | 0.163411 | 0.422597 | 0.000000 | 0.427618 | 0.060496 | 0.298382 | 0.061234 | 0.367861 | 0.392970 |
| 33 | 33 | 33_0 | COMPLETED | GPEI | 0.010007 | 0.359553 | 0.000000 | 0.337570 | 0.000000 | 0.362450 | 0.031914 | 0.360043 | 0.297296 |
| 34 | 34 | 34_0 | COMPLETED | GPEI | 0.016386 | 0.392692 | 0.000000 | 0.397354 | 0.000000 | 0.382050 | 0.008916 | 0.346636 | 0.290461 |
| 35 | 35 | 35_0 | COMPLETED | GPEI | 0.374686 | 0.508453 | 0.000000 | 0.385156 | 0.000000 | 0.455517 | 0.000000 | 0.315004 | 0.250628 |
| 36 | 36 | 36_0 | COMPLETED | GPEI | -0.038851 | 0.374380 | 0.000000 | 0.375906 | 0.000000 | 0.370218 | 0.025660 | 0.344407 | 0.345649 |
| 37 | 37 | 37_0 | COMPLETED | GPEI | 1.809430 | 0.039100 | 0.330390 | 0.738861 | 0.346845 | 0.329358 | 0.230040 | 0.766101 | 0.820291 |
| 38 | 38 | 38_0 | COMPLETED | GPEI | -0.032717 | 0.349761 | 0.000000 | 0.386191 | 0.000000 | 0.331044 | 0.000000 | 0.348385 | 0.360020 |
| 39 | 39 | 39_0 | COMPLETED | GPEI | 1.220288 | 0.470744 | 0.267233 | 0.657531 | 0.610199 | 0.029554 | 0.040163 | 0.234528 | 0.085158 |
| 40 | 40 | 40_0 | COMPLETED | GPEI | -0.023780 | 0.345149 | 0.000000 | 0.395652 | 0.000000 | 0.374382 | 0.026744 | 0.340384 | 0.351266 |
| 41 | 41 | 41_0 | COMPLETED | GPEI | 0.331096 | 0.205150 | 0.000000 | 0.473549 | 0.102291 | 0.087878 | 0.000000 | 0.184055 | 0.135849 |
| 42 | 42 | 42_0 | COMPLETED | GPEI | 0.000612 | 0.359848 | 0.000000 | 0.377112 | 0.000000 | 0.325162 | 0.022951 | 0.316742 | 0.343676 |
| 43 | 43 | 43_0 | COMPLETED | GPEI | 0.402406 | 0.076097 | 0.000000 | 0.550396 | 0.271479 | 0.163713 | 0.000000 | 0.348159 | 0.081873 |
| 44 | 44 | 44_0 | COMPLETED | GPEI | 0.346127 | 0.208505 | 0.000000 | 0.675748 | 0.000000 | 0.000000 | 0.000000 | 0.312995 | 0.112425 |
| 45 | 45 | 45_0 | COMPLETED | GPEI | 0.022018 | 0.026074 | 0.000000 | 0.424845 | 0.311036 | 0.029792 | 0.000000 | 0.270044 | 0.273023 |
| 46 | 46 | 46_0 | COMPLETED | GPEI | 0.180551 | 0.000000 | 0.000000 | 0.517746 | 0.168016 | 0.000000 | 0.000000 | 0.268264 | 0.357308 |
| 47 | 47 | 47_0 | COMPLETED | GPEI | 0.043655 | 0.095179 | 0.000000 | 0.359911 | 0.222143 | 0.000000 | 0.000000 | 0.334821 | 0.300010 |
| 48 | 48 | 48_0 | COMPLETED | GPEI | 0.248907 | 0.017847 | 0.000000 | 0.561075 | 0.371528 | 0.000000 | 0.000000 | 0.182111 | 0.320720 |
| 49 | 49 | 49_0 | COMPLETED | GPEI | 0.167933 | 0.017785 | 0.000000 | 0.341444 | 0.291967 | 0.000000 | 0.121951 | 0.249804 | 0.257908 |