Example FETCH3 optimization results#

This notebook demonstrates how to:

  • Get the results of the best optimization trial

  • Visualize optimization results

Hide code cell content 
1import os
2import sys
3
4sys.path.insert(0, os.path.abspath("/Users/jmissik/Desktop/repos/fetch3_nhl/"))

 1from pathlib import Path
 2
 3from ax.utils.notebook.plotting import init_notebook_plotting
 4from ax.service.utils.report_utils import get_standard_plots, exp_to_df
 5from boa import load_yaml, scheduler_from_json_file
 6import plotly.graph_objects as go
 7import yaml
 8
 9from fetch3.optimize.fetch_wrapper import get_model_obs
10
11init_notebook_plotting()

[INFO 05-24 12:41:06] ax.utils.notebook.plotting: Injecting Plotly library into cell. Do not overwrite or delete cell.

Load experiment data#

Load the experiment data. Make sure to use the correct paths for your experiment.

1# Full filepath to the scheduler.json file in the experiment output directory
2scheduler_fp = Path(
3    "/Users/jmissik/Desktop/repos/fetch3_nhl/output/UMBS_test_Ameriflux_M8_20220519T120204/scheduler.json"
4)
5experiment_dir = scheduler_fp.parent

1# CHANGE TO THE CORRECT PATH FOR YOUR CONFIG FILE
2ex_config_file = experiment_dir / "opt_umbs_M8.yml"

Loading the experiment data saved in the scheduler.json file

1scheduler = scheduler_from_json_file(scheduler_fp)
2experiment = scheduler.experiment

View experiment data#

Get the index of the best trial from the experiment

1best_trial, best_params, obj = scheduler.get_best_trial()
2best_trial

/Users/jmissik/anaconda3/envs/fetch3-dev/lib/python3.9/site-packages/gpytorch/lazy/lazy_tensor.py:1741: UserWarning:

torch.triangular_solve is deprecated in favor of torch.linalg.solve_triangularand will be removed in a future PyTorch release.
torch.linalg.solve_triangular has its arguments reversed and does not return a copy of one of the inputs.
X = torch.triangular_solve(B, A).solution
should be replaced with
X = torch.linalg.solve_triangular(A, B). (Triggered internally at  /Users/distiller/project/conda/conda-bld/pytorch_1646756029501/work/aten/src/ATen/native/BatchLinearAlgebra.cpp:1672.)

49

Get a dataframe with the experiment data (objective function value, parameter values, generation strategy).

Note that generation method is in the last column of the dataframe

1exp_to_df(experiment)
root_mean_squared_error trial_index arm_name sat_xylem Vcmax25 alpha_gs wp_s50 c3 Root_depth Soil_depth ... gsmax kr kt Topt kd hx50 nl Emax trial_status generation_method
0 709.977912 0 0_0 0.484067 48.036140 7.519005 -1.592774e+06 18.369667 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED Sobol
3 385.474862 1 1_0 0.404008 83.748104 6.192228 -5.302799e+05 10.317422 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED Sobol
15 454.495921 2 2_0 0.485040 59.255760 7.709103 -7.673659e+05 14.513156 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED Sobol
27 153.036270 3 3_0 0.646628 42.260797 4.249056 -4.639487e+05 16.815446 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED Sobol
39 741.318055 4 4_0 0.647079 50.648946 7.780069 -1.544699e+06 19.820800 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED Sobol
45 152.279980 5 5_0 0.697220 41.402180 8.406831 -4.574240e+05 7.349853 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED Sobol
46 733.060952 6 6_0 0.427319 59.829341 7.816947 -1.699305e+06 2.877928 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED Sobol
47 321.113443 7 7_0 0.535472 29.845795 7.178160 -1.868478e+06 5.427692 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED Sobol
48 342.277369 8 8_0 0.505089 47.162280 7.019449 -7.947982e+05 6.548168 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED Sobol
49 946.311420 9 9_0 0.681587 81.364094 4.779107 -1.269665e+06 11.008100 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED Sobol
1 183.503509 10 10_0 0.660389 39.582186 6.382073 -3.881654e+05 11.232960 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
2 166.737714 11 11_0 0.627494 38.267026 8.684380 -4.536066e+05 4.009241 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
4 308.707790 12 12_0 0.570892 38.305404 4.000000 -2.579448e+05 17.037111 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
5 154.214399 13 13_0 0.683947 34.744913 7.284500 -5.880254e+05 5.593642 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
6 207.760980 14 14_0 0.632824 48.199274 8.135597 -3.159522e+05 6.881961 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
7 149.934242 15 15_0 0.700000 37.254503 5.644381 -5.160795e+05 18.598681 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
8 157.653815 16 16_0 0.700000 35.507540 4.545393 -5.816560e+05 13.598921 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
9 155.134202 17 17_0 0.700000 48.919581 5.234030 -4.342966e+05 18.819887 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
10 151.769351 18 18_0 0.649907 30.179664 8.287210 -5.670498e+05 12.344863 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
11 183.484300 19 19_0 0.673032 42.867508 6.115923 -5.636929e+05 13.801147 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
12 224.985003 20 20_0 0.700000 23.941165 8.738361 -4.822758e+05 6.785448 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
13 152.107130 21 21_0 0.700000 41.064696 4.000000 -4.870069e+05 20.000000 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
14 152.699981 22 22_0 0.700000 41.406706 4.510496 -4.497981e+05 15.729326 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
16 168.235743 23 23_0 0.607748 23.364212 5.587014 -6.334716e+05 14.217228 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
17 170.058351 24 24_0 0.700000 40.368771 9.000000 -4.031079e+05 20.000000 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
18 187.895476 25 25_0 0.597033 28.411033 4.151725 -6.090261e+05 2.342689 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
19 157.974293 26 26_0 0.661747 37.853409 9.000000 -5.623311e+05 8.013860 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
20 169.151487 27 27_0 0.700000 23.475698 7.051710 -6.299935e+05 20.000000 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
21 152.069496 28 28_0 0.672899 30.007216 6.173331 -5.590373e+05 13.808955 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
22 193.009202 29 29_0 0.603175 21.515926 8.133392 -7.542711e+05 3.766795 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
23 239.455562 30 30_0 0.700000 51.876839 4.000000 -4.417862e+05 0.100000 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
24 155.854774 31 31_0 0.517195 26.084640 9.000000 -6.079341e+05 20.000000 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
25 165.325956 32 32_0 0.700000 25.672040 6.071457 -6.899597e+05 9.328652 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
26 166.117078 33 33_0 0.700000 65.200583 9.000000 -2.944046e+05 20.000000 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
28 154.435515 34 34_0 0.700000 54.756052 9.000000 -3.876760e+05 18.327823 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
29 219.614871 35 35_0 0.700000 73.353944 5.110703 -2.277503e+05 20.000000 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
30 157.295638 36 36_0 0.400000 28.641663 9.000000 -6.025696e+05 8.184790 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
31 153.599665 37 37_0 0.521203 30.206635 8.499523 -6.056085e+05 10.857784 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
32 159.021334 38 38_0 0.400000 36.426594 9.000000 -4.988179e+05 20.000000 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
33 180.290613 39 39_0 0.400000 20.000000 9.000000 -7.133994e+05 15.951424 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
34 154.303831 40 40_0 0.400000 29.668028 6.692637 -5.710170e+05 20.000000 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
35 160.102108 41 41_0 0.515107 56.249116 9.000000 -3.633259e+05 20.000000 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
36 153.301994 42 42_0 0.528003 34.472673 9.000000 -5.164456e+05 15.903039 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
37 155.409131 43 43_0 0.616138 45.869999 7.141185 -4.302946e+05 20.000000 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
38 154.322380 44 44_0 0.400000 29.489176 9.000000 -5.837847e+05 17.212053 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
40 415.685479 45 45_0 0.400000 79.405878 9.000000 -1.434335e+05 20.000000 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
41 155.823446 46 46_0 0.700000 59.488081 6.774489 -3.451670e+05 20.000000 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
42 151.020622 47 47_0 0.564941 33.410536 5.737763 -5.339560e+05 20.000000 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
43 163.288165 48 48_0 0.400000 44.742209 9.000000 -4.688367e+05 9.359752 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI
44 149.771795 49 49_0 0.700000 36.660069 7.207708 -5.086869e+05 11.805689 3.2 5 ... 0.01 0.005 0.0016 289.15 0.0011 -1274000 2 1.000000e-09 COMPLETED GPEI

50 rows × 72 columns

Plotting#

Plot optimization results#

1plots = get_standard_plots(experiment, scheduler.generation_strategy.model)

1for plot in plots:
2    plot.show()

Plot model output vs observations for the best trial in the experiment#

1dir_best_trial = experiment_dir / str(best_trial).zfill(6)
2model_file = dir_best_trial / "sapflux.nc"
3
4config_file = dir_best_trial / "config.yml"
5with open(config_file, "r") as yml_config:
6    model_cfg = yaml.safe_load(yml_config)
7
8config = load_yaml(ex_config_file)

1dir_best_trial

PosixPath('/Users/jmissik/Desktop/repos/fetch3_nhl/output/UMBS_test_Ameriflux_M8_20220519T120204/000049')

1ex_settings = config["optimization_options"]
2# obs_file = ex_settings["obsfile"]
3obs_file = "/Users/jmissik/Desktop/repos/fetch3_nhl/data/USA_UMB_CON_sapf_data.csv"
4model_settings = config["model_options"]

1model, obs = get_model_obs(model_file, obs_file, ex_settings, model_settings, model_cfg["parameters"])

/Users/jmissik/Desktop/repos/fetch3_nhl/fetch3/optimize/fetch_wrapper.py:185: FutureWarning:

Indexing a timezone-aware DatetimeIndex with a timezone-naive datetime is deprecated and will raise KeyError in a future version. Use a timezone-aware object instead.

1fig = go.Figure()
2fig.add_trace(go.Scatter(x=obs.index, y=model, name="model"))
3fig.add_trace(go.Scatter(x=obs.index, y=obs, name="observations"))
4fig.update_layout(yaxis_title="sap flux [cm3 hr-1]")
5fig.show()