Loading BOA from JSON and Plotting Results#

This notebook demonstrates how to:

Loading a Scheduler from JSON from a previous run (If you want to see the Experiment that this is from, see Running BOA Optimization Directly in Python. We will look at the output and plot some exploratory data analysis.

 1import pathlib
 2import os
 3
 4import numpy as np
 5from ax.utils.notebook.plotting import init_notebook_plotting
 6from ax.plot.trace import optimization_trace_single_method_plotly
 7from ax.service.utils.report_utils import get_standard_plots, exp_to_df
 8import boa
 9from botorch.test_functions.synthetic import Cosine8
10
11init_notebook_plotting()

[WARNING 07-13 14:31:15] ax.service.utils.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.

[INFO 07-13 14:31:16] ax.utils.notebook.plotting: Injecting Plotly library into cell. Do not overwrite or delete cell.

Loading the Scheduler#

1# setup stuff just because this gets reused from the latest run for the case of the docs
2try:
3    run = list(pathlib.Path().resolve().glob("boa_runs*"))[-1]
4except IndexError:
5    print("No run to load. Make sure you run optimization_run.ipynb first")

1# Filepath to the scheduler.json
2
3scheduler_fp = run / "scheduler.json"

1scheduler = boa.scheduler_from_json_file(scheduler_fp)

1scheduler

Scheduler(experiment=Experiment(boa_runs), generation_strategy=GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 16 trials, GPEI for subsequent trials]), options=SchedulerOptions(max_pending_trials=10, trial_type=<TrialType.TRIAL: 0>, batch_size=None, total_trials=None, tolerated_trial_failure_rate=0.5, min_failed_trials_for_failure_rate_check=5, log_filepath=None, logging_level=20, ttl_seconds_for_trials=None, init_seconds_between_polls=1, min_seconds_before_poll=1.0, seconds_between_polls_backoff_factor=1.5, timeout_hours=None, run_trials_in_batches=False, debug_log_run_metadata=False, early_stopping_strategy=None, global_stopping_strategy=None, suppress_storage_errors_after_retries=False))

Show the Best Fitted Trial and Raw Trial#

best_fitted_trials uses the data to do a fitting from all trials and with the noise levels you provided (or if no noise levels was provided, it assumed an unknown level of noise and inferred the noise level from the trial runs)

1trial = scheduler.best_fitted_trials()
2trial

{18: {'params': {'x0': 0.23794805673981673,
   'x1': 0.43669338710507394,
   'x2': 0.7888537020279267,
   'x3': 0.4327516568525241,
   'x4': 0.22304759837088722,
   'x5': 0.0,
   'x6': 0.7919340883291089,
   'x7': 0.19043021545505892},
  'means': {'Cosine8': 1.5766337853108028},
  'cov_matrix': {'Cosine8': {'Cosine8': 7.17359326106318e-07}}}}

if you need the exact points of the best trial, maybe because you need the trial number of the best trial to plot results, or for any other reason, best_raw_trails does not do any fitting

1trial = scheduler.best_raw_trials()
2trial

{16: {'params': {'x0': 0.23794805673981673,
   'x1': 0.43669338710507394,
   'x2': 0.7888537020279267,
   'x3': 0.4327516568525241,
   'x4': 0.22304759837088722,
   'x5': 0.0,
   'x6': 0.7919340883291089,
   'x7': 0.19043021545505892},
  'means': {'Cosine8': 1.5766326241},
  'cov_matrix': {'Cosine8': {'Cosine8': 0.0}}}}

1boa.scheduler_to_df(scheduler)
trial_index arm_name trial_status generation_method Cosine8 x0 x1 x2 x3 x4 x5 x6 x7
0 0 0_0 COMPLETED Sobol 2.239105 0.005910 0.141367 0.425610 0.810076 0.635368 0.737481 0.202842 0.678401
1 1 1_0 COMPLETED Sobol 4.136699 0.341347 0.802491 0.530528 0.982942 0.558278 0.774273 0.698905 0.884335
2 2 2_0 COMPLETED Sobol 1.838099 0.302048 0.279949 0.768200 0.617394 0.314809 0.049353 0.800600 0.165349
3 3 3_0 COMPLETED Sobol 2.408117 0.378163 0.272300 0.601005 0.585589 0.075222 0.125974 0.767954 0.901783
4 4 4_0 COMPLETED Sobol 2.030690 0.283629 0.755290 0.824802 0.287725 0.288463 0.035348 0.849888 0.135928
5 5 5_0 COMPLETED Sobol 2.714052 0.096529 0.478756 0.827078 0.874856 0.580245 0.656032 0.030549 0.542257
6 6 6_0 COMPLETED Sobol 3.941725 0.487296 0.817129 0.716379 0.836942 0.999895 0.541045 0.672588 0.288226
7 7 7_0 COMPLETED Sobol 3.478950 0.582199 0.793198 0.758041 0.570732 0.532217 0.236029 0.958385 0.249541
8 8 8_0 COMPLETED Sobol 3.067485 0.395262 0.826062 0.535096 0.934967 0.645789 0.598909 0.461659 0.370842
9 9 9_0 COMPLETED Sobol 3.494017 0.182291 0.651113 0.095934 0.183234 0.885109 0.892330 0.478828 0.958136
10 10 10_0 COMPLETED Sobol 3.746949 0.896021 0.811340 0.677272 0.186724 0.665177 0.693099 0.585175 0.535108
11 11 11_0 COMPLETED Sobol 3.460985 0.252204 0.733171 0.100481 0.721839 0.489904 0.966838 0.542897 0.879006
12 12 12_0 COMPLETED Sobol 1.909430 0.322564 0.660139 0.394390 0.604160 0.333827 0.762729 0.174212 0.219013
13 13 13_0 COMPLETED Sobol 3.083720 0.006783 0.942449 0.745871 0.035081 0.765784 0.865521 0.793785 0.291791
14 14 14_0 COMPLETED Sobol 1.728998 0.594221 0.255711 0.124992 0.630445 0.026356 0.738504 0.325613 0.206705
15 15 15_0 COMPLETED Sobol 2.716962 0.497243 0.624900 0.224264 0.841092 0.816033 0.106921 0.200558 0.707149
16 16 16_0 COMPLETED GPEI 1.576633 0.237948 0.436693 0.788854 0.432752 0.223048 0.000000 0.791934 0.190430
17 17 17_0 COMPLETED GPEI 1.779049 0.299262 0.142714 0.705431 0.656360 0.233444 0.058971 0.745862 0.370448
18 18 18_0 COMPLETED GPEI 1.726575 0.243311 0.410822 0.868419 0.523951 0.380031 0.000000 0.850181 0.000000

EDA Plots#

1boa.plot_metrics_trace(scheduler)

1boa.plot_contours(scheduler)

1boa.plot_slice(scheduler)

We can also directly pass in our scheduler file path instead of having to reload it ourselves

1boa.plot_metrics_trace(scheduler_fp)