rai_toolbox.mushin.RobustnessCurve

class rai_toolbox.mushin.RobustnessCurve(eval_task_cfg=None, working_dir=None)

Abstract class for workflows that measure performance for different perturbation values.

This workflow requires and uses parameter epsilon as the configuration option for varying the perturbation.

See also

MultiRunMetricsWorkflow

__init__(eval_task_cfg=None, working_dir=None)

Workflows and experiments using Hydra.

Parameters:

eval_task_cfg: Mapping | None (default: None)

The workflow configuration object.

run(*, epsilon, task_fn_wrapper=<function zen>, target_job_dirs=None, version_base='1.1', working_dir=None, sweeper=None, launcher=None, overrides=None, **workflow_overrides)

Run the experiment for varying value epsilon.

Parameters:

epsilon: str | Sequence[float]

The configuration parameter for the perturbation. Unlike Hydra overrides, this parameter can be a list of floats that will be converted into a multirun sequence override for Hydra.

task_fn_wrapper: Callable[[Callable[…, T1]], Callable[[Any], T1]] | None, optional (default=rai_toolbox.mushin.zen)

A wrapper applied to self.task prior to launching the task. The default wrapper is rai_toolbox.mushin.zen. Specify None for no wrapper to be applied.

working_dir: str (default: None, the Hydra default will be used)

The directory to run the experiment in. This value is used for setting hydra.sweep.dir.

sweeper: str | None (default: None)

The configuration name of the Hydra Sweeper to use (i.e., the override for hydra/sweeper=sweeper)

launcher: str | None (default: None)

The configuration name of the Hydra Launcher to use (i.e., the override for hydra/launcher=launcher)

overrides: List[str] | None (default: None)

Parameter overrides not considered part of the workflow parameter set. This is helpful for filtering out parameters stored in self.workflow_overrides.

**workflow_overrides: dict | str | int | float | bool | multirun | hydra_list

These parameters represent the values for configurations to use for the experiment.

These values will be appended to the overrides for the Hydra job.

plot(metric, ax=None, group=None, save_filename=None, non_multirun_params_as_singleton_dims=False, **kwargs)

Plot metrics versus epsilon.

Using the xarray.Dataset from to_xarray, plot the metrics against the workflow perturbation parameters.

Parameters:

metric: str

The metric saved

ax: Axes | None (default: None)

If not None, the matplotlib.Axes to use for plotting.

group: str | None (default: None)

Needed if other parameters besides epsilon were varied.

save_filename: str | None (default: None)

If not None save figure to the filename provided.

non_multirun_params_as_singleton_dimsbool, optional (default=False)

If True then non-multirun entries from workflow_overrides will be included as length-1 dimensions in the xarray. Useful for merging/ concatenation with other Datasets

**kwargs: Any

Additional arguments passed to xarray.plot.

load_from_dir(working_dir, metrics_filename)

Loading workflow job data from a given working directory. The workflow is loaded in-place and “self” is returned by this method.

Parameters:

working_dir: str | Path

The base working directory of the experiment. It is expected that subdirectories within this working directory will contain individual Hydra jobs data (yaml configurations) and saved metrics files.

metrics_filename: str | Sequence[str] | None

The filename(s) or glob-pattern(s) uses to load the metrics. If None, the metrics stored in self.metrics is used.

Returns:

loaded_workflowSelf

Methods

__init__([eval_task_cfg, working_dir])	Workflows and experiments using Hydra.
task(*args, **kwargs)	Abstract staticmethod for users to define the task that is configured and launched by the workflow
jobs_post_process()	Method to extract attributes and metrics relevant to the workflow.
load_from_dir(working_dir, metrics_filename)	Loading workflow job data from a given working directory.
plot(metric[, ax, group, save_filename, ...])	Plot metrics versus epsilon.
run(*, epsilon[, task_fn_wrapper, ...])	Run the experiment for varying value epsilon.
to_xarray([...])	Convert workflow data to xarray Dataset.
validate([include_pre_task])	Validates that the configuration will execute with the user-defined evaluation task

Attributes

cfgs
metrics
workflow_overrides
jobs