boa.metrics.metrics.ModularMetric#

class boa.metrics.metrics.ModularMetric(metric_to_eval: Callable, param_names: Optional[list[str]] = None, noise_sd: Optional[float] = 0.0, metric_func_kwargs: Optional[dict] = None, wrapper: Optional[BaseWrapper] = None, properties: Optional[dict[str]] = None, **kwargs)#

Bases: NoisyFunctionMetric

Metric is computed by evaluating a deterministic function, implemented in the f method defined on this class.

f will expect an array x, which is constructed from the arm parameters by extracting the values of the parameter names given in param_names, in that order.

Parameters:

name – Name of the metric
param_names – An ordered list of names of parameters to be passed to the deterministic function.
noise_sd – Scale of normal noise added to the function result. If None, interpret the function as noisy with unknown noise level.
lower_is_better – Flag for metrics which should be minimized.

Methods

`clone`	Create a copy of this Metric.
`deserialize_init_args`	Given a dictionary, deserialize the properties needed to initialize the object.
`f`	The deterministic function that produces the metric outcomes.
`fetch_experiment_data`	Fetch this metric's data for an experiment.
`fetch_experiment_data_multi`	Fetch multiple metrics data for an experiment.
`fetch_trial_data`	Fetch data for one trial.
`fetch_trial_data_multi`	Fetch multiple metrics data for one trial.
`is_available_while_running`	Whether metrics of this class are available while the trial is running.
`lookup_or_fetch_experiment_data_multi`	Fetch or lookup (with fallback to fetching) data for given metrics, depending on whether they are available while running.
`serialize_init_args`	Serialize the properties needed to initialize the object.
`to_dict`	Convert Ax experiment to a dictionary.

Attributes

`db_id`
`fetch_multi_group_by_metric`	Metric class, with which to group this metric in Experiment._metrics_by_class, which is used to combine metrics on experiment into groups and then fetch their data via Metric.fetch_trial_data_multi for each group.
`name`	Get name of metric.

clone() → Metric#: Create a copy of this Metric.

data_constructor#: alias of Data

classmethod deserialize_init_args(args: Dict[str, Any]) → Dict[str, Any]#: Given a dictionary, deserialize the properties needed to initialize the object. Used for storage.

f(*args, **kwargs)#: The deterministic function that produces the metric outcomes.

fetch_experiment_data(experiment: core.experiment.Experiment, **kwargs: Any) → Dict[int, MetricFetchResult]#

Fetch this metric’s data for an experiment.

Returns Dict of trial_index => Result

classmethod fetch_experiment_data_multi(experiment: core.experiment.Experiment, metrics: Iterable[Metric], trials: Optional[Iterable[core.base_trial.BaseTrial]] = None, **kwargs: Any) → Dict[int, Dict[str, MetricFetchResult]]#

Fetch multiple metrics data for an experiment.

Returns Dict of trial_index => (metric_name => Result) Default behavior calls fetch_trial_data_multi for each trial. Subclasses should override to batch data computation across trials + metrics.

property fetch_multi_group_by_metric: Type[Metric]#

Metric class, with which to group this metric in Experiment._metrics_by_class, which is used to combine metrics on experiment into groups and then fetch their data via Metric.fetch_trial_data_multi for each group.

NOTE: By default, this property will just return the class on which it is defined; however, in some cases it is useful to group metrics by their superclass, in which case this property should return that superclass.

fetch_trial_data(trial: BaseTrial, *args, **kwargs)#: Fetch data for one trial.

classmethod fetch_trial_data_multi(trial: core.base_trial.BaseTrial, metrics: Iterable[Metric], **kwargs: Any) → Dict[str, MetricFetchResult]#

Fetch multiple metrics data for one trial.

Returns Dict of metric_name => Result Default behavior calls fetch_trial_data for each metric. Subclasses should override this to trial data computation for multiple metrics.

classmethod is_available_while_running() → bool#

Whether metrics of this class are available while the trial is running. Metrics that are not available while the trial is running are assumed to be available only upon trial completion. For such metrics, data is assumed to never change once the trial is completed.

NOTE: If this method returns False, data-fetching via experiment.fetch_data will return the data cached on the experiment (for the metrics of the given class) whenever its available. Data is cached on experiment when attached via experiment.attach_data.

classmethod lookup_or_fetch_experiment_data_multi(experiment: core.experiment.Experiment, metrics: Iterable[Metric], trials: Optional[Iterable[core.base_trial.BaseTrial]] = None, **kwargs: Any) → Tuple[Dict[int, Dict[str, MetricFetchResult]], bool]#

Fetch or lookup (with fallback to fetching) data for given metrics, depending on whether they are available while running. Return a tuple containing the data, along with a boolean that will be True if new data was fetched, and False if all data was looked up from cache.

If metric is available while running, its data can change (and therefore we should always re-fetch it). If metric is available only upon trial completion, its data does not change, so we can look up that data on the experiment and only fetch the data that is not already attached to the experiment.

NOTE: If fetching data for a metrics class that is only available upon trial completion, data fetched in this function (data that was not yet available on experiment) will be attached to experiment.

property name: str#: Get name of metric.

classmethod serialize_init_args(obj: Any) → Dict[str, Any]#: Serialize the properties needed to initialize the object. Used for storage.

to_dict() → dict#: Convert Ax experiment to a dictionary.