hydra_zen.ZenStore#

class hydra_zen.ZenStore(name=None, *, deferred_to_config=True, deferred_hydra_store=True, overwrite_ok=False, warn_node_kwarg=True)[source]#

An abstraction over Hydra’s store, for creating multiple, isolated config stores.

Whereas Hydra exposes a single global config store that provides no warnings when store entries are overwritted, ZenStore instances are isolated, do not populate the global store unless instructed to, and they protect users from unwittingly overwriting store entries.

Attributes:
groups

Returns a sorted list of the groups registered with this store

name

Methods

__call__(target, /, name, NodeName]] = ..., ...)

Store a config or customize the default values of the store.

add_to_hydra_store([overwrite_ok])

Adds all of this store's enqueued entries to Hydra's global config store.

copy([store_name])

Returns a copy of the store with the same overridden defaults.

copy_with_mapped_groups(...[, store_name, ...])

Create a copy of a store whose entries' groups have been updated according to the provided mapping.

enqueue_all()

Add all of the store's entries to the queue to be added to hydra's store.

get_entry(group, name)

Access a store entry, which is a mapping that specifies the entry's name, group, package, provider, and node.

has_enqueued()

True if this store has entries that have not yet been added to Hydra's config store.

merge(_ZenStore__other[, store_name])

Create a new store by merging two stores.

update(_ZenStore__other)

Updates the store inplace with redundant entries being overwritten.

delete_entry

Notes

hydra_zen.store is available as a pre-instantiated globally-available store, which is initialized as:

store = ZenStore(
    name="zen_store",
    deferred_to_config=True,
    deferred_hydra_store=True,
)

Internally, each ZenStore instance holds a mapping of:

tuple[group, name] ->  {node: Dataclass | omegaconf.Container,
                        name: str,
                        group: str,
                        package: Optional[str],
                        provider: Optional[str]}

Auto-config capabilities

ZenStore is also designed to consolidate the config-creation and storage process; it can be used to automatically apply config-creation a function (e.g., builds) to a target in order to auto-generate a config for the target, which is then stored.

from hydra_zen import store
def func(x, y): ...

store(func, x=2, y=3)

from hydra_zen import builds, store
def func(x, y): ...

store(builds(func, x=2, y=3, populate_full_signature=True), name="func")

It can also be used to decorate config-targets and dataclasses, enabling “inline” config creation and storage patterns.

These auto config-creation capabilities are designed to be deferred until a config is actually accessed by users or added to Hydra’s global config store. This enables store-decorator patterns to be used within library code without slowing down import times.

Self-partialing patterns

A ZenStore instance can be called repeatedly - without a config target - with different options to incrementally change the store’s default configurations. E.g. the following are effectively equivalent

from hydra_zen import store

book_store = store(group="books")
romance_store = book_store(provider="genre: romance")
fantasy_store = book_store(provider="genre: fantasy")

romance_store({"title": "heartfelt"})
romance_store({"title": "lustfully longingness"})

fantasy_store({"title": "elvish cookbook"})
fantasy_store({"title": "dwarves can't jump"})

from hydra_zen import store

store(
    {"title": "heartfelt"},
    group="book",
    provider="genre: romance",
)
store(
    {"title": "lustfully longingness"},
    group="book",
    provider="genre: romance",
)

store(
    {"title": "elvish cookbook"},
    group="book",
    provider="genre: fantasy",
)
store(
    {"title": "dwarves can't jump"},
    group="book",
    provider="genre: fantasy",
)

Configuring Hydra itself

Special support is provided for overriding Hydra’s configuration; the name and group of the store entry is inferred to be ‘config’ and ‘hydra’, respectively, when an instance/subclass of HydraConf is being stored. E.g., specifying

from hydra.conf import HydraConf, JobConf
from hydra_zen import store

store(HydraConf(job=JobConf(chdir=True)))

is equivalent to writing the following manually

store(HydraConf(job=JobConf(chdir=True)), name="config", group="hydra", provider="hydra_zen")

Additionally, overwriting the store entry for HydraConf will not raise an error even if ZenStore(overwrite_ok=False) is specified.

Examples

(Some helpful boilerplate code for these examples)

>>> from hydra_zen import to_yaml, store, ZenStore
>>> def pyaml(x):
...     # for pretty printing configs
...     print(to_yaml(x))

Basic usage

Let’s add a config to hydra-zen’s pre-instantiated ZenStore instance. Each store entry must have an associated name. Optionally, a group, package, and/or provider may be specified for the entry as well.

>>> config1 = {'name': 'Roger', 'age': 24}
>>> config2 = {'name': 'Rita', 'age': 27}
>>> _ = store(config1, name="roger", group="profiles")
>>> _ = store(config2, name="rita", group="profiles")
>>> store
zen_store
{'profiles': ['roger', 'rita']}

A store’s entries are keyed by their (group, name) pairs (the default group is None).

>>> store["profiles", "roger"]  # (group, name) -> config node
{'name': 'Roger', age: 24}

By default, the stored config(s) will be “enqueued” for addition to Hydra’s config store. The method add_to_hydra_store() must be called to add the enqueued configs to Hydra’s central store.

>>> store.has_enqueued()
True
>>> store.add_to_hydra_store()  # adds all enqueued entries to Hydra's global store
>>> store.has_enqueued()
False

By default, attempting to overwrite an entry will result in an error.

>>> store({}, name="rita", group="profiles")  # same name and group as above
ValueError: (name=rita group=profiles): Hydra config store entry already exists.
Specify `overwrite_ok=True` to enable replacing config store entries

We can create a distinct store that has an independent internal repository of configs.

>>> new_store = ZenStore("new_store")
>>> _ = new_store([1, 2, 3], name="backbone")

>>> store
zen_store
{'profiles': ['roger', 'rita']}
>>> new_store
new_store
{None: ['backbone']}

Auto-config capabilities

The input to a store is processed by the store’s to_config function prior to creating the stored config node. This defaults to hydra_zen.wrapper.default_to_config, which applies hydra_zen.builds or hydra_zen.just to inputs based on their types.

For instance, consider the following function:

>>> def sum_it(a: int, b: int): return a + b

We can pass sum_it directly to our store to leverage auto-config and auto-naming capabilities. Here, builds(sum_it, a=1, b=2) will be called under the hood by new_store to create the config for sum_it.

>>> store2 = ZenStore()
>>> _ = store2(sum_it, a=1, b=2)  # entry name defaults to `sum_it.__name__`
>>> config = store2[None, "sum_it"]
>>> pyaml(config)
_target_: __main__.sum_it
a: 1
b: 2

Refer to hydra_zen.wrapper.default_to_config for more details about the default auto-config behaviors of ZenStore.

Support for decorator patterns

ZenStore.__call__ is a pass-through and can be used as a decorator. Let’s add two store entries for func by decorating it.

>>> store = ZenStore()

>>> @store(a=1, b=22, name="func1")
... @store(a=-10, name="func2")
... def func(a: int, b: int):
...     return a - b

Each application of @store utilizes the store’s auto-config capability to create and store a config inline. I.e. the above snippet is equivalent to

>>> from hydra_zen import builds
>>>
>>> store(builds(func, a=1, b=22), name="func1")
>>> store(builds(func, a=-10,
...              populate_full_signature=True
...              ),
...       name="func2",
... )

>>> func(10, 3)  # the decorated function is left unchanged
7
>>> pyaml(store[None, "func1"])
_target_: __main__.func
a: 1
b: 22
>>> pyaml(store[None, "func2"])
_target_: __main__.func
b: ???
a: -10

Note that, by default, the application of to_config via the store is deferred until that entry is actually accessed. This offsets the runtime cost of constructing configs for the decorated function so that it need not be paid until the config is actually accessed by the store.

Customizable store defaults via ‘self-partialing’ patterns

The default values for a store’s __call__ parameters – group, to_config, etc. – can easily be customized. Simply call the store with those new values and without specifying an object to be stored. This will return a “mirrored” store instance – with the same internal state as the original store – with updated defaults.

For example, let’s create a store where we want to store multiple configs under a 'math' group and under a 'functools' group, respectively.

>>> import math
>>> import functools

>>> new_store = ZenStore()
>>> math_store = new_store(group="math")  # overwrites group default
>>> tool_store = new_store(group="functools")  # overwrites group default

math_store and tool_store both share the same internal state as new_store, but have overwritten default values for the group.

>>> math_store(math.floor)  # equivalent to: `new_store(math.floor, group="math")`
>>> math_store(math.ceil)
>>> tool_store(functools.lru_cache)
>>> tool_store(functools.wraps)

See that new_store has entries under these corresponding groups:

>>> new_store
custom_store
{'math': ['floor', 'ceil'], 'functools': ['lru_cache', 'wraps']}

These “self-partialing” patterns can be chained indefinitely and can be used to set partial defaults for the to_config function.

>>> profile_store = new_store(group="profile")
>>> schemaless = profile_store(schema="<none>")

>>> from dataclasses import dataclass
>>>
>>> @profile_store(name="admin", has_root=True)
>>> @schemaless(name="test_admin", has_root=True)
>>> @schemaless(name="test_user", has_root=False)
>>> @dataclass
>>> class Profile:
>>>     username: str
>>>     schema: str
>>>     has_root: bool

>>> pyaml(new_store["profile", "admin"])
username: ???
schema: ???
has_root: true
_target_: __main__.Profile

>>> pyaml(new_store["profile", "test_admin"])
username: ???
schema: <none>
has_root: true
_target_: __main__.Profile

Manipulating and updating a store

A store can be copied, updated, and merged. Its entries can have their groups remapped, and individual entries can be deleted. See the docs for the corresponding methods for details and examples.

__init__(name=None, *, deferred_to_config=True, deferred_hydra_store=True, overwrite_ok=False, warn_node_kwarg=True)[source]#
Parameters:
nameOptional[str]

The name for this store.

deferred_to_configbool, default=True

If True (default), this store will a not apply to_config to the target until that specific entry is accessed by the store.

deferred_hydra_storebool, default=True

If True (default), this store will not add entries to Hydra’s global config store until store.add_to_hydra_store is called explicitly.

overwrite_okbool, default=False

If False (default), attempting to overwrite entries in this store and trying to use this store to overwrite entries in Hydra’s global store will raise a ValueError.

warn_node_kwarg: bool, default=True

If True specifying a node kwarg in ZenStore.__call__ will emit a warning.

This helps to protect users from mistakenly self-partializing a store with store(node=Config) instead of actually storing the node with store(Config).

__call__(target : Optional[T] = None, /, name: NodeName | Callable[[Any], NodeName]] = ..., group: GroupName | Callable[[T], GroupName]] = None, package: Optional[str | Callable[[T], str]]] | None], provider: Optional[str], to_config: Callable[[T], Node] = ..., **to_config_kw: Any) → T | ZenStore[source]#

Store a config or customize the default values of the store.

Parameters:
objOptional[T]

The object to be stored. This is a positional-only argument.

If obj is not specified, then the provided arguments are used to create a mirrored store instance with updated default arguments.

nameNodeName | Callable[[T], NodeName]

The entry’s name, or a callable that will be called as (obj) -> entry-name. The default is lambda obj: obj.__name__.

Store entries are keyed off of (group, name).

groupOptional[GroupName | Callable[[T], GroupName]]

The entry’s group’s name, or a callable that will be called as (obj) -> entry-group. The default is None. Subgroups can be specified using / within the group name.

Store entries are keyed off of (group, name).

to_configCallable[[T], Node] = default_to_config

Called on obj to produce the entry’s “node” (the config). Refer to hydra_zen.wrapper.default_to_config for the default behavior. Specify lambda x: x to have obj be stored directly as the entry’s node.

By default the call to to_config is deferred until the entry is actually accessed by the store.

packageOptional[str | Callable[[Any], str]]

The entry’s package. Default is None.

providerOptional[str]

An optional provider name for the entry.

**to_config_kwAny

Additional arguments that will be passed to to_config.

Returns:
T | ZenStore

If obj was specified, it is returned unchanged. Otherwise a new instance of ZenStore is return, which mirrors the internal state of this store and has updated default arguments.

__getitem__(key)[source]#

Access a entry’s config node by specifying (group, name). Or, access a mapping of (group, name) -> node for all nodes in a specified group, including nodes within subgroups.

See also

ZenStore.get_entry

Examples

>>> from hydra_zen import store
>>> store(dict(x=1), name="a", group="fruit")
>>> store(dict(x=2), name="b", group="fruit/apple")
>>> store(dict(x=3), name="c", group="fruit/apple")
>>> store(dict(x=4), name="d", group="fruit/orange")
>>> store(dict(x=5), name="e", group="veggie")

Accessing an individual entry’s config node.

>>> store["fruit/apple", "b"]
{'x': 2}

Accessing all config nodes under the “fruit/apple” group

>>> store["fruit/apple"]
{('fruit/apple', 'b'): {'x': 2}, ('fruit/apple', 'c'): {'x': 3}}

Accessing all config nodes under the “fruit” group

>>> store["fruit"]
{('fruit', 'a'): {'x': 1},
 ('fruit/apple', 'b'): {'x': 2},
 ('fruit/apple', 'c'): {'x': 3},
 ('fruit/orange', 'd'): {'x': 4}}
add_to_hydra_store(overwrite_ok=None)[source]#

Adds all of this store’s enqueued entries to Hydra’s global config store.

This method need not be called for a store initialized as ZenStore(deferred_hydra_store=False).

Parameters:
overwrite_okOptional[bool]

If False, this method raises ValueError if an entry in Hydra’s config store will be overwritten. Defaults to the value of overwrite_ok specified when initializing this store.

Examples

>>> from hydra_zen import ZenStore
>>> store1 = ZenStore()
>>> store2 = ZenStore()

>>> store1({'a': 1}, name="x")
>>> store1.add_to_hydra_store()
>>> store2({'a': 2}, name="x")
>>> store2.add_to_hydra_store()
ValueError: (name=x group=None): Hydra config store entry already exists. Specify `overwrite_ok=True` to enable replacing config store entries

>>> store2.add_to_hydra_store(overwrite_ok=True)  # successfully overwrites entry
copy(store_name=None)[source]#

Returns a copy of the store with the same overridden defaults.

Parameters:
store_namestr | None, optional (default=None)
Returns:
ZenStore

Examples

>>> from hydra_zen import ZenStore
>>> s1 = ZenStore()(group="G")
>>> s1({}, name="a")
>>> s2 = s1.copy()
>>> s2({}, name="b")
>>> s1
s1
{'G': ['a']}
>>> s2
s1_copy
{'G': ['a', 'b']}
copy_with_mapped_groups(old_group_to_new_group, *, store_name=None, overwrite_ok=None)[source]#

Create a copy of a store whose entries’ groups have been updated according to the provided mapping.

Parameters:
old_group_to_new_groupMapping[GroupName, GroupName] | Callable[[GroupName], GroupName]

A mapping or callable that transforms an old group name to a new one. Groups in the store that are not included in the mapping are unaffected.

A GroupName is str | None.

store_nameOptional[None]

If specified, the name of the new store.

overwrite_okOptional[bool]:

If specified, determines if the mapping can overwrite existing store entries. Otherwise, defers to ZenStore(overwrite_ok).

Returns:
new_store

A copy of self with remapped groups.

Examples

>>> from hydra_zen import ZenStore

Creating an initial store

>>> s1 = ZenStore("s1")
>>> s1({}, group=None, name="a")
>>> s1({}, group="A/1", name="b")
>>> s1({}, group="A/2", name="c")
>>> s1
s1
{None: ['a'], 'A/1': ['b'], 'A/2': ['c']}

Replacing group “A/1” with “B”, via a mapping

>>> s2 = s1.copy_with_mapped_groups({"A/1": "B"}, store_name="s2")
>>> s2
s2
{None: ['a'], 'A/2': ['c'], 'B': ['b']}

Placing all entries under group “A/” within a new inner group “p”, via a function

>>> s3 = s1.copy_with_mapped_groups(
...     lambda g: g + "/p" if g and g.startswith("A/") else g, store_name="s3"
... )
>>> s3
s3
{None: ['a'], 'A/1/p': ['b'], 'A/2/p': ['c']}
get_entry(group, name)[source]#

Access a store entry, which is a mapping that specifies the entry’s name, group, package, provider, and node.

Parameters:
groupstr | None
namestr
Returns:
dict

  • name: NodeName

  • group: GroupName

  • package: Optional[str]

  • provider: Optional[str]

  • node: ConfigType

Notes

Mutating the returned mapping will not affect the store’s internal entry. Mutating a node in the returned entry may have unintended consequences and is not advised.

Examples

>>> from hydra_zen import store, ZenStore
>>> store(dict(x=1), name="a", group="fruit")
>>> store.get_entry("fruit", "a")
{'name': 'a',
 'group': 'fruit',
 'package': None,
 'provider': None,
 'node': {'x': 1}}
delete_entry(group, name)[source]#
update(_ZenStore__other)[source]#

Updates the store inplace with redundant entries being overwritten.

Can also be applied via the |= in-place operator.

Examples

>>> from hydra_zen import ZenStore
>>> def f(): ...
>>> def g(): ...
>>> s1 = ZenStore("s1")
>>> s2 = ZenStore("s2")
>>> s1(f)  # store f in s1
>>> s2(g)  # store g in s2

>>> s1.update(s2)
>>> s1  # s1 now has entries for both f and g
s1
{None: ['f', 'g']}

Alternatively, the |= operator can be used to update a store inplace.

>>> s3 = ZenStore("s3")
>>> s3 |= s2
>>> s3
s3
{None: ['g']}
merge(_ZenStore__other, store_name=None)[source]#

Create a new store by merging two stores.

The new store’s default settings will reflect those of self in self.merge(other). This can also be applied via the | operator.

Examples

>>> from hydra_zen import ZenStore
>>> def f(): ...
>>> def g(): ...
>>> s1 = ZenStore("s1")
>>> s2 = ZenStore("s2")
>>> s1(f)  # store f in s1
>>> s2(g)  # store g in s2

>>> s3 = s1.merge(s2)
>>> s3
s1_copy
{None: ['f', 'g']}

Alternatively, the | operator can be used to merge stores.

>>> s4 = s1 | s2
>>> s4
s1_copy
{None: ['f', 'g']}
has_enqueued()[source]#

True if this store has entries that have not yet been added to Hydra’s config store.

Returns:
bool

Examples

>>> from hydra_zen import ZenStore
>>> store = ZenStore(deferred_hydra_store=True)
>>> store.has_enqueued()
False

>>> store({"a": 1}, name)
>>> store.has_enqueued()
True

>>> store.add_to_hydra_store()
>>> store.has_enqueued()
False
enqueue_all()[source]#

Add all of the store’s entries to the queue to be added to hydra’s store.

Examples

>>> from hydra_zen import ZenStore
>>> store = ZenStore(deferred_hydra_store=True)
>>> store({"a": 1}, name)
>>> store.has_enqueued()
True

>>> store.add_to_hydra_store()
>>> store.has_enqueued()
False

>>> store.enqueue_all()
>>> store.has_enqueued()
True
__iter__()[source]#

Yields all entries in this store.

Notes

Mutating the returned mappings will not affect the store’s internal entries. Mutating a node in an entry may have unintended consequences and is not advised.

Examples

>>> from hydra_zen import store
>>> store(dict(x=1), name="a", group="fruit")
>>> store(dict(x=2), name="b", group="fruit/orange")
>>> store(dict(x=3), name="c", group="veggie")

>>> list(store)
[{'name': 'a',
  'group': 'fruit',
  'package': None,
  'provider': None,
  'node': {'x': 1}},
 {'name': 'b',
  'group': 'fruit/orange',
  'package': None,
  'provider': None,
  'node': {'x': 2}},
 {'name': 'c',
  'group': 'veggie',
  'package': None,
  'provider': None,
  'node': {'x': 3}}]
__eq__(_ZenStore__o)[source]#

Returns True if two stores share identical internal repos and queues.

Examples

>>> from hydra_zen import ZenStore
>>> store1 = ZenStore()
>>> store2 = ZenStore()
>>> store1_a = store1(group='a')
>>> _ = store1_a(dict(x=1), name="foo")

>>> store1 == store1_a
True
>>> store1 == store2
False

Attributes

name

groups

Returns a sorted list of the groups registered with this store