Loading BOA from JSON and Plotting Results#

This notebook demonstrates how to:

Loading a Scheduler from JSON from a previous run (If you want to see the Experiment that this is from, see Running BOA Optimization Directly in Python. We will look at the output and plot some exploratory data analysis.

import pathlib
import os

import numpy as np
from ax.utils.notebook.plotting import init_notebook_plotting
from ax.plot.trace import optimization_trace_single_method_plotly
from ax.service.utils.report_utils import get_standard_plots, exp_to_df
import boa
from botorch.test_functions.synthetic import Cosine8

init_notebook_plotting()

[INFO 07-10 11:42:13] ax.utils.notebook.plotting: Injecting Plotly library into cell. Do not overwrite or delete cell.

Loading the Scheduler#

# setup stuff just because this gets reused from the latest run for the case of the docs
try:
    run = list(pathlib.Path().resolve().glob("boa_runs*"))[-1]
except IndexError:
    print("No run to load. Make sure you run optimization_run.ipynb first")

# Filepath to the scheduler.json

scheduler_fp = run / "scheduler.json"

scheduler = boa.scheduler_from_json_file(scheduler_fp)

scheduler

Scheduler(experiment=Experiment(boa_runs), generation_strategy=GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 16 trials, GPEI for subsequent trials]), options=SchedulerOptions(max_pending_trials=10, trial_type=<TrialType.TRIAL: 0>, batch_size=None, total_trials=None, tolerated_trial_failure_rate=0.5, min_failed_trials_for_failure_rate_check=5, log_filepath=None, logging_level=20, ttl_seconds_for_trials=None, init_seconds_between_polls=1, min_seconds_before_poll=1.0, seconds_between_polls_backoff_factor=1.5, timeout_hours=None, run_trials_in_batches=False, debug_log_run_metadata=False, early_stopping_strategy=None, global_stopping_strategy=None, suppress_storage_errors_after_retries=False))

Show the Best Fitted Trial and Raw Trial#

best_fitted_trials uses the data to do a fitting from all trials and with the noise levels you provided (or if no noise levels was provided, it assumed an unknown level of noise and inferred the noise level from the trial runs)

trial = scheduler.best_fitted_trials()
trial

/Users/madelinescyphers/miniconda3/envs/boa-dev/lib/python3.10/site-packages/botorch/models/utils/assorted.py:201: InputDataWarning:

Input data is not standardized. Please consider scaling the input to zero mean and unit variance.

/Users/madelinescyphers/miniconda3/envs/boa-dev/lib/python3.10/site-packages/botorch/models/utils/assorted.py:201: InputDataWarning:

Input data is not standardized. Please consider scaling the input to zero mean and unit variance.

{49: {'params': {'x0': 0.35172402518319096,
   'x1': 0.3698520404111157,
   'x2': 0.3666874377748964,
   'x3': 0.3891053518012792,
   'x4': 0.3734271631500215,
   'x5': 0.0,
   'x6': 0.36002879332283894,
   'x7': 0.35626455543974167},
  'means': {'Cosine8': 0.24623248264577358},
  'cov_matrix': {'Cosine8': {'Cosine8': 1.389173793847577e-06}}}}

if you need the exact points of the best trial, maybe because you need the trial number of the best trial to plot results, or for any other reason, best_raw_trails does not do any fitting

trial = scheduler.best_raw_trials()
trial

{46: {'params': {'x0': 0.35172402518319096,
   'x1': 0.3698520404111157,
   'x2': 0.3666874377748964,
   'x3': 0.3891053518012792,
   'x4': 0.3734271631500215,
   'x5': 0.0,
   'x6': 0.36002879332283894,
   'x7': 0.35626455543974167},
  'means': {'Cosine8': 0.24594534280000002},
  'cov_matrix': {'Cosine8': {'Cosine8': 0.0}}}}

boa.scheduler_to_df(scheduler)

	trial_index	arm_name	trial_status	generation_method	Cosine8	x0	x1	x2	x3	x4	x5	x6	x7
0	0	0_0	COMPLETED	Sobol	2.029324	0.990547	0.288610	0.346135	0.301916	0.392528	0.110616	0.251962	0.694829
1	1	1_0	COMPLETED	Sobol	1.646673	0.262968	0.633469	0.616036	0.388401	0.427373	0.075654	0.736025	0.068805
2	2	2_0	COMPLETED	Sobol	2.144039	0.449725	0.480685	0.448159	0.548387	0.476969	0.054146	0.188638	0.978092
3	3	3_0	COMPLETED	Sobol	2.753313	0.431875	0.352470	0.052401	0.387660	0.960082	0.576974	0.864202	0.656100
4	4	4_0	COMPLETED	Sobol	3.049808	0.417872	0.647828	0.534399	0.174281	0.122930	0.771529	0.943236	0.778014
5	5	5_0	COMPLETED	Sobol	3.438582	0.895702	0.123685	0.017906	0.944492	0.357977	0.397257	0.967300	0.830473
6	6	6_0	COMPLETED	Sobol	2.069711	0.730564	0.433245	0.129033	0.007862	0.276506	0.861777	0.722698	0.493578
7	7	7_0	COMPLETED	Sobol	3.978929	0.119162	0.841748	0.990021	0.173180	0.671806	0.980801	0.351636	0.768843
8	8	8_0	COMPLETED	Sobol	2.448196	0.872235	0.815817	0.038919	0.380534	0.389911	0.711213	0.233549	0.776188
9	9	9_0	COMPLETED	Sobol	2.355544	0.286572	0.689979	0.266366	0.717407	0.132874	0.584265	0.825217	0.396867
10	10	10_0	COMPLETED	Sobol	3.009791	0.878739	0.405202	0.912348	0.580230	0.005852	0.909818	0.364704	0.253842
11	11	11_0	COMPLETED	Sobol	4.328626	0.923367	0.330759	0.834218	0.927327	0.997183	0.092835	0.869460	0.131116
12	12	12_0	COMPLETED	Sobol	1.768276	0.243641	0.347330	0.265216	0.186192	0.356840	0.319708	0.540039	0.912528
13	13	13_0	COMPLETED	Sobol	3.453395	0.687077	0.397473	0.776234	0.949205	0.041172	0.667479	0.779359	0.674563
14	14	14_0	COMPLETED	Sobol	3.690702	0.975360	0.064095	0.197106	0.757204	0.497103	0.936394	0.569573	0.615107
15	15	15_0	COMPLETED	Sobol	3.142783	0.429568	0.688117	0.034250	0.896887	0.059263	0.958642	0.881870	0.458131
16	16	16_0	COMPLETED	GPEI	1.479021	0.372412	0.593584	0.499630	0.420313	0.380171	0.086078	0.618598	0.190225
17	17	17_0	COMPLETED	GPEI	1.848654	0.133412	0.684324	0.681570	0.411335	0.436329	0.068753	0.885155	0.000000
18	18	18_0	COMPLETED	GPEI	1.719800	0.325406	0.594220	0.725250	0.252905	0.509522	0.030746	0.666139	0.000000
19	19	19_0	COMPLETED	GPEI	1.046564	0.402912	0.494737	0.378010	0.276616	0.380545	0.153921	0.527507	0.442987
20	20	20_0	COMPLETED	GPEI	1.218009	0.258951	0.508938	0.575936	0.340772	0.332884	0.000000	0.529066	0.350962
21	21	21_0	COMPLETED	GPEI	1.568101	0.481007	0.621265	0.352696	0.290243	0.383432	0.214157	0.624025	0.128180
22	22	22_0	COMPLETED	GPEI	0.694584	0.336467	0.445526	0.348530	0.228832	0.347404	0.025912	0.501000	0.434758
23	23	23_0	COMPLETED	GPEI	0.825017	0.352038	0.451447	0.468967	0.271408	0.324219	0.205404	0.402803	0.416627
24	24	24_0	COMPLETED	GPEI	1.557443	0.395534	0.549492	0.458918	0.248066	0.392106	0.047248	0.617300	0.554689
25	25	25_0	COMPLETED	GPEI	0.480696	0.315712	0.389980	0.319921	0.210277	0.304849	0.065001	0.386094	0.359033
26	26	26_0	COMPLETED	GPEI	0.666511	0.339306	0.340017	0.384153	0.180095	0.305986	0.086064	0.496477	0.330224
27	27	27_0	COMPLETED	GPEI	0.706937	0.291560	0.464385	0.268035	0.237875	0.283826	0.036284	0.310615	0.422044
28	28	28_0	COMPLETED	GPEI	0.462260	0.294458	0.372932	0.331051	0.213052	0.406540	0.045543	0.338189	0.331316
29	29	29_0	COMPLETED	GPEI	0.412818	0.381948	0.360701	0.308135	0.281827	0.271955	0.029180	0.377959	0.373382
30	30	30_0	COMPLETED	GPEI	0.779420	0.201541	0.402730	0.284366	0.215171	0.325871	0.130716	0.428317	0.285005
31	31	31_0	COMPLETED	GPEI	0.360526	0.384370	0.348921	0.333884	0.256066	0.351508	0.000000	0.323477	0.369886
32	32	32_0	COMPLETED	GPEI	0.496104	0.423386	0.378361	0.328701	0.251978	0.341967	0.000000	0.314916	0.250011
33	33	33_0	COMPLETED	GPEI	0.484507	0.316908	0.313341	0.327963	0.270480	0.343131	0.000000	0.330986	0.450200
34	34	34_0	COMPLETED	GPEI	0.511228	0.422185	0.353153	0.259158	0.230322	0.373841	0.044939	0.324527	0.406461
35	35	35_0	COMPLETED	GPEI	0.412813	0.391179	0.366165	0.387284	0.245393	0.318190	0.000000	0.297713	0.397175
36	36	36_0	COMPLETED	GPEI	0.306940	0.363556	0.368205	0.333058	0.286906	0.374388	0.000000	0.351965	0.367086
37	37	37_0	COMPLETED	GPEI	0.277358	0.347890	0.362567	0.370842	0.317156	0.376454	0.000000	0.351853	0.343095
38	38	38_0	COMPLETED	GPEI	1.136160	0.112641	0.189161	0.303689	0.489141	0.372734	0.511617	0.477609	0.238445
39	39	39_0	COMPLETED	GPEI	0.308610	0.341814	0.374499	0.338032	0.340505	0.376037	0.000000	0.299552	0.351262
40	40	40_0	COMPLETED	GPEI	0.301817	0.334515	0.340547	0.343663	0.377408	0.394207	0.000000	0.370891	0.312737
41	41	41_0	COMPLETED	GPEI	0.516885	0.264182	0.317618	0.370233	0.449533	0.350750	0.000000	0.364791	0.229735
42	42	42_0	COMPLETED	GPEI	0.343410	0.354746	0.345775	0.366915	0.346625	0.380737	0.078507	0.338985	0.337162
43	43	43_0	COMPLETED	GPEI	0.290229	0.348906	0.361392	0.379225	0.351250	0.419578	0.000000	0.344883	0.344597
44	44	44_0	COMPLETED	GPEI	0.526956	0.342680	0.344620	0.411267	0.366610	0.502446	0.000000	0.311529	0.329995
45	45	45_0	COMPLETED	GPEI	1.662781	0.399885	0.441332	0.581806	0.777294	0.626267	0.051651	0.416699	0.374300
46	46	46_0	COMPLETED	GPEI	0.245945	0.351724	0.369852	0.366687	0.389105	0.373427	0.000000	0.360029	0.356265
47	47	47_0	COMPLETED	GPEI	0.307105	0.380847	0.364766	0.371232	0.441443	0.358401	0.000000	0.356899	0.365381
48	48	48_0	COMPLETED	GPEI	0.484492	0.357150	0.284339	0.304645	0.428924	0.365574	0.001600	0.306166	0.270031
49	49	49_0	COMPLETED	GPEI	3.524827	0.675317	0.286011	0.860863	0.223397	0.281086	0.897188	0.753581	0.865855

EDA Plots#

boa.plot_metrics_trace(scheduler)

boa.plot_contours(scheduler)

boa.plot_slice(scheduler)

We can also directly pass in our scheduler file path instead of having to reload it ourselves

boa.plot_metrics_trace(scheduler_fp)