rai_toolbox.mushin.BaseWorkflow

class rai_toolbox.mushin.BaseWorkflow(eval_task_cfg=None)

Provides an interface for creating a reusable workflow: encapsulated “boilerplate” for running, aggregating, and analyzing one or more Hydra jobs.

Attributes:

cfgsList[Any]

List of configurations for each Hydra job.

metricsDict[str, List[Any]]

Dictionary of metrics for across all jobs.

workflow_overridesDict[str, Any]

Workflow parameters defined as additional arguments to run.

jobsList[Any]

List of jobs returned for each experiment within the workflow.

working_dir: Optional[pathlib.Path]

The working directory of the experiment defined by Hydra’s sweep directory (hydra.sweep.dir).

__init__(eval_task_cfg=None)

Workflows and experiments using Hydra.

Parameters:

eval_task_cfg: Mapping | None (default: None)

The workflow configuration object.

static task(*args, **kwargs)

User-defined task that is run by the workflow. This should be a static method.

Arguments will be instantiated configuration variables. For example, if the the workflow configuration is structured as:

├── eval_task_cfg
│    ├── trainer
|    ├── module
|    ├── another_config

The inputs to task can be any of the three configurations: trainer, module, or another_config such as:

@staticmethod
def task(trainer: Trainer, module: LightningModule) -> None:
    trainer.fit(module)

Notes

This function is automatically wrapped by zen, which is responsible for parsing the function’s signature and then extracting and instantiating the corresponding fields from a Hydra config object – passing them to the function. This behavior can be modified by self.run(task_fn_wrapper=...)

jobs_post_process()

Method to extract attributes and metrics relevant to the workflow.

run(*, working_dir=None, sweeper=None, launcher=None, overrides=None, task_fn_wrapper=<function zen>, pre_task_fn_wrapper=<function zen>, version_base='1.1', to_dictconfig=False, config_name='rai_workflow', job_name='rai_workflow', with_log_configuration=True, **workflow_overrides)

Run the experiment.

Individual workflows can explicitly define workflow_overrides to improve readability and undstanding of what parameters are expected for a particular workflow.

Parameters:

task_fn_wrapper: Callable[[Callable[…, T1]], Callable[[Any], T1]] | None, optional (default=rai_toolbox.mushin.zen)

A wrapper applied to self.task prior to launching the task. The default wrapper is rai_toolbox.mushin.zen. Specify None for no wrapper to be applied.

working_dir: str (default: None, the Hydra default will be used)

The directory to run the experiment in. This value is used for setting hydra.sweep.dir.

sweeper: str | None (default: None)

The configuration name of the Hydra Sweeper to use (i.e., the override for hydra/sweeper=sweeper)

launcher: str | None (default: None)

The configuration name of the Hydra Launcher to use (i.e., the override for hydra/launcher=launcher)

overrides: List[str] | None (default: None)

Parameter overrides not considered part of the workflow parameter set. This is helpful for filtering out parameters stored in self.workflow_overrides.

version_baseOptional[str], optional (default=1.1)

Available starting with Hydra 1.2.0. - If the version_base parameter is not specified, Hydra 1.x will use defaults compatible with version 1.1. Also in this case, a warning is issued to indicate an explicit version_base is preferred. - If the version_base parameter is None, then the defaults are chosen for the current minor Hydra version. For example for Hydra 1.2, then would imply config_path=None and hydra.job.chdir=False. - If the version_base parameter is an explicit version string like “1.1”, then the defaults appropriate to that version are used.

to_dictconfig: bool (default: False)

If True, convert a dataclasses.dataclass to a omegaconf.DictConfig. Note, this will remove Hydra’s cabability for validation with structured configurations.

config_namestr (default: “rai_workflow”)

Name of the stored configuration in Hydra’s ConfigStore API.

job_namestr (default: “rai_workflow”)

Name of job for logging.

with_log_configurationbool (default: True)

If True, enables the configuration of the logging subsystem from the loaded config.

**workflow_overrides: str | int | float | bool | multirun | hydra_list | dict

These parameters represent the values for configurations to use for the experiment.

Passing param=multirun([1, 2, 3]) will perform a multirun over those three param values, whereas passing param=hydra_list([1, 2, 3]) will pass the entire list as a single input.

These values will be appended to the overrides for the Hydra job.

Methods

__init__([eval_task_cfg])	Workflows and experiments using Hydra.
task(*args, **kwargs)	User-defined task that is run by the workflow.
jobs_post_process()	Method to extract attributes and metrics relevant to the workflow.
plot(**kwargs)	Plot workflow metrics.
run(*[, working_dir, sweeper, launcher, ...])	Run the experiment.
to_xarray()	Convert workflow data to xArray Dataset or DataArray.
validate([include_pre_task])	Validates that the configuration will execute with the user-defined evaluation task

Attributes

cfgs
metrics
workflow_overrides
jobs