hydra_zen.ZenStore#
- class hydra_zen.ZenStore(name=None, *, deferred_to_config=True, deferred_hydra_store=True, overwrite_ok=False, warn_node_kwarg=True)[source]#
An abstraction over Hydra’s store, for creating multiple, isolated config stores.
Whereas Hydra exposes a single global config store that provides no warnings when store entries are overwritted,
ZenStore
instances are isolated, do not populate the global store unless instructed to, and they protect users from unwittingly overwriting store entries.- Attributes:
groups
Returns a sorted list of the groups registered with this store
- name
Methods
__call__
(target, /, name, NodeName]] = ..., ...)Store a config or customize the default values of the store.
add_to_hydra_store
([overwrite_ok])Adds all of this store's enqueued entries to Hydra's global config store.
copy
([store_name])Returns a copy of the store with the same overridden defaults.
copy_with_mapped_groups
(...[, store_name, ...])Create a copy of a store whose entries' groups have been updated according to the provided mapping.
Add all of the store's entries to the queue to be added to hydra's store.
get_entry
(group, name)Access a store entry, which is a mapping that specifies the entry's name, group, package, provider, and node.
True
if this store has entries that have not yet been added to Hydra's config store.merge
(_ZenStore__other[, store_name])Create a new store by merging two stores.
update
(_ZenStore__other)Updates the store inplace with redundant entries being overwritten.
delete_entry
Notes
hydra_zen.store
is available as a pre-instantiated globally-available store, which is initialized as:store = ZenStore( name="zen_store", deferred_to_config=True, deferred_hydra_store=True, )
Internally, each
ZenStore
instance holds a mapping of:tuple[group, name] -> {node: Dataclass | omegaconf.Container, name: str, group: str, package: Optional[str], provider: Optional[str]}
Auto-config capabilities
ZenStore
is also designed to consolidate the config-creation and storage process; it can be used to automatically apply config-creation a function (e.g.,builds
) to a target in order to auto-generate a config for the target, which is then stored.from hydra_zen import store def func(x, y): ... store(func, x=2, y=3)
It can also be used to decorate config-targets and dataclasses, enabling “inline” config creation and storage patterns.
These auto config-creation capabilities are designed to be deferred until a config is actually accessed by users or added to Hydra’s global config store. This enables store-decorator patterns to be used within library code without slowing down import times.
Self-partialing patterns
A
ZenStore
instance can be called repeatedly - without a config target - with different options to incrementally change the store’s default configurations. E.g. the following are effectively equivalentfrom hydra_zen import store book_store = store(group="books") romance_store = book_store(provider="genre: romance") fantasy_store = book_store(provider="genre: fantasy") romance_store({"title": "heartfelt"}) romance_store({"title": "lustfully longingness"}) fantasy_store({"title": "elvish cookbook"}) fantasy_store({"title": "dwarves can't jump"})
from hydra_zen import store store( {"title": "heartfelt"}, group="book", provider="genre: romance", ) store( {"title": "lustfully longingness"}, group="book", provider="genre: romance", ) store( {"title": "elvish cookbook"}, group="book", provider="genre: fantasy", ) store( {"title": "dwarves can't jump"}, group="book", provider="genre: fantasy", )
Configuring Hydra itself
Special support is provided for overriding Hydra’s configuration; the name and group of the store entry is inferred to be ‘config’ and ‘hydra’, respectively, when an instance/subclass of
HydraConf
is being stored. E.g., specifyingfrom hydra.conf import HydraConf, JobConf from hydra_zen import store store(HydraConf(job=JobConf(chdir=True)))
is equivalent to writing the following manually
store(HydraConf(job=JobConf(chdir=True)), name="config", group="hydra", provider="hydra_zen")
Additionally, overwriting the store entry for
HydraConf
will not raise an error even ifZenStore(overwrite_ok=False)
is specified.Examples
(Some helpful boilerplate code for these examples)
Basic usage
Let’s add a config to hydra-zen’s pre-instantiated
ZenStore
instance. Each store entry must have an associated name. Optionally, a group, package, and/or provider may be specified for the entry as well.>>> config1 = {'name': 'Roger', 'age': 24} >>> config2 = {'name': 'Rita', 'age': 27} >>> _ = store(config1, name="roger", group="profiles") >>> _ = store(config2, name="rita", group="profiles") >>> store zen_store {'profiles': ['roger', 'rita']}
A store’s entries are keyed by their
(group, name)
pairs (the default group isNone
).>>> store["profiles", "roger"] # (group, name) -> config node {'name': 'Roger', age: 24}
By default, the stored config(s) will be “enqueued” for addition to Hydra’s config store. The method
add_to_hydra_store()
must be called to add the enqueued configs to Hydra’s central store.>>> store.has_enqueued() True >>> store.add_to_hydra_store() # adds all enqueued entries to Hydra's global store >>> store.has_enqueued() False
By default, attempting to overwrite an entry will result in an error.
>>> store({}, name="rita", group="profiles") # same name and group as above ValueError: (name=rita group=profiles): Hydra config store entry already exists. Specify `overwrite_ok=True` to enable replacing config store entries
We can create a distinct store that has an independent internal repository of configs.
>>> new_store = ZenStore("new_store") >>> _ = new_store([1, 2, 3], name="backbone")
>>> store zen_store {'profiles': ['roger', 'rita']} >>> new_store new_store {None: ['backbone']}
Auto-config capabilities
The input to a store is processed by the store’s
to_config
function prior to creating the stored config node. This defaults tohydra_zen.wrapper.default_to_config
, which applieshydra_zen.builds
orhydra_zen.just
to inputs based on their types.For instance, consider the following function:
We can pass
sum_it
directly to our store to leverage auto-config and auto-naming capabilities. Here,builds(sum_it, a=1, b=2)
will be called under the hood bynew_store
to create the config forsum_it
.>>> store2 = ZenStore() >>> _ = store2(sum_it, a=1, b=2) # entry name defaults to `sum_it.__name__` >>> config = store2[None, "sum_it"] >>> pyaml(config) _target_: __main__.sum_it a: 1 b: 2
Refer to
hydra_zen.wrapper.default_to_config
for more details about the default auto-config behaviors ofZenStore
.Support for decorator patterns
ZenStore.__call__
is a pass-through and can be used as a decorator. Let’s add two store entries forfunc
by decorating it.>>> store = ZenStore()
Each application of
@store
utilizes the store’s auto-config capability to create and store a config inline. I.e. the above snippet is equivalent to>>> func(10, 3) # the decorated function is left unchanged 7 >>> pyaml(store[None, "func1"]) _target_: __main__.func a: 1 b: 22 >>> pyaml(store[None, "func2"]) _target_: __main__.func b: ??? a: -10
Note that, by default, the application of
to_config
via the store is deferred until that entry is actually accessed. This offsets the runtime cost of constructing configs for the decorated function so that it need not be paid until the config is actually accessed by the store.Customizable store defaults via ‘self-partialing’ patterns
The default values for a store’s
__call__
parameters –group
,to_config
, etc. – can easily be customized. Simply call the store with those new values and without specifying an object to be stored. This will return a “mirrored” store instance – with the same internal state as the original store – with updated defaults.For example, let’s create a store where we want to store multiple configs under a
'math'
group and under a'functools'
group, respectively.>>> new_store = ZenStore() >>> math_store = new_store(group="math") # overwrites group default >>> tool_store = new_store(group="functools") # overwrites group default
math_store
andtool_store
both share the same internal state asnew_store
, but have overwritten default values for thegroup
.>>> math_store(math.floor) # equivalent to: `new_store(math.floor, group="math")` >>> math_store(math.ceil) >>> tool_store(functools.lru_cache) >>> tool_store(functools.wraps)
See that
new_store
has entries under these corresponding groups:>>> new_store custom_store {'math': ['floor', 'ceil'], 'functools': ['lru_cache', 'wraps']}
These “self-partialing” patterns can be chained indefinitely and can be used to set partial defaults for the
to_config
function.>>> profile_store = new_store(group="profile") >>> schemaless = profile_store(schema="<none>")
>>> from dataclasses import dataclass >>> >>> @profile_store(name="admin", has_root=True) >>> @schemaless(name="test_admin", has_root=True) >>> @schemaless(name="test_user", has_root=False) >>> @dataclass >>> class Profile: >>> username: str >>> schema: str >>> has_root: bool
>>> pyaml(new_store["profile", "admin"]) username: ??? schema: ??? has_root: true _target_: __main__.Profile
>>> pyaml(new_store["profile", "test_admin"]) username: ??? schema: <none> has_root: true _target_: __main__.Profile
Manipulating and updating a store
A store can be copied, updated, and merged. Its entries can have their groups remapped, and individual entries can be deleted. See the docs for the corresponding methods for details and examples.
- __init__(name=None, *, deferred_to_config=True, deferred_hydra_store=True, overwrite_ok=False, warn_node_kwarg=True)[source]#
- Parameters:
- nameOptional[str]
The name for this store.
- deferred_to_configbool, default=True
If
True
(default), this store will a not applyto_config
to the target until that specific entry is accessed by the store.- deferred_hydra_storebool, default=True
If
True
(default), this store will not add entries to Hydra’s global config store untilstore.add_to_hydra_store
is called explicitly.- overwrite_okbool, default=False
If
False
(default), attempting to overwrite entries in this store and trying to use this store to overwrite entries in Hydra’s global store will raise aValueError
.- warn_node_kwarg: bool, default=True
If
True
specifying anode
kwarg inZenStore.__call__
will emit a warning.This helps to protect users from mistakenly self-partializing a store with
store(node=Config)
instead of actually storing the node withstore(Config)
.
- __call__(target : Optional[T] = None, /, name: NodeName | Callable[[Any], NodeName]] = ..., group: GroupName | Callable[[T], GroupName]] = None, package: Optional[str | Callable[[T], str]]] | None], provider: Optional[str], to_config: Callable[[T], Node] = ..., **to_config_kw: Any) T | ZenStore [source]#
Store a config or customize the default values of the store.
- Parameters:
- objOptional[T]
The object to be stored. This is a positional-only argument.
If
obj
is not specified, then the provided arguments are used to create a mirrored store instance with updated default arguments.- nameNodeName | Callable[[T], NodeName]
The entry’s name, or a callable that will be called as
(obj) -> entry-name
. The default islambda obj: obj.__name__
.Store entries are keyed off of
(group, name)
.- groupOptional[GroupName | Callable[[T], GroupName]]
The entry’s group’s name, or a callable that will be called as
(obj) -> entry-group
. The default isNone
. Subgroups can be specified using / within the group name.Store entries are keyed off of
(group, name)
.- to_configCallable[[T], Node] = default_to_config
Called on
obj
to produce the entry’s “node” (the config). Refer tohydra_zen.wrapper.default_to_config
for the default behavior. Specifylambda x: x
to haveobj
be stored directly as the entry’s node.By default the call to
to_config
is deferred until the entry is actually accessed by the store.- packageOptional[str | Callable[[Any], str]]
The entry’s package. Default is
None
.- providerOptional[str]
An optional provider name for the entry.
- **to_config_kwAny
Additional arguments that will be passed to
to_config
.
- Returns:
- T | ZenStore
If
obj
was specified, it is returned unchanged. Otherwise a new instance ofZenStore
is return, which mirrors the internal state of this store and has updated default arguments.
- __getitem__(key)[source]#
Access a entry’s config node by specifying
(group, name)
. Or, access a mapping of(group, name) -> node
for all nodes in a specified group, including nodes within subgroups.See also
Examples
Accessing an individual entry’s config node.
>>> store["fruit/apple", "b"] {'x': 2}
Accessing all config nodes under the “fruit/apple” group
>>> store["fruit/apple"] {('fruit/apple', 'b'): {'x': 2}, ('fruit/apple', 'c'): {'x': 3}}
Accessing all config nodes under the “fruit” group
>>> store["fruit"] {('fruit', 'a'): {'x': 1}, ('fruit/apple', 'b'): {'x': 2}, ('fruit/apple', 'c'): {'x': 3}, ('fruit/orange', 'd'): {'x': 4}}
- add_to_hydra_store(overwrite_ok=None)[source]#
Adds all of this store’s enqueued entries to Hydra’s global config store.
This method need not be called for a store initialized as
ZenStore(deferred_hydra_store=False)
.- Parameters:
- overwrite_okOptional[bool]
If
False
, this method raisesValueError
if an entry in Hydra’s config store will be overwritten. Defaults to the value ofoverwrite_ok
specified when initializing this store.
Examples
>>> store1({'a': 1}, name="x") >>> store1.add_to_hydra_store() >>> store2({'a': 2}, name="x") >>> store2.add_to_hydra_store() ValueError: (name=x group=None): Hydra config store entry already exists. Specify `overwrite_ok=True` to enable replacing config store entries
>>> store2.add_to_hydra_store(overwrite_ok=True) # successfully overwrites entry
- copy(store_name=None)[source]#
Returns a copy of the store with the same overridden defaults.
- Parameters:
- store_namestr | None, optional (default=None)
- Returns:
- ZenStore
Examples
- copy_with_mapped_groups(old_group_to_new_group, *, store_name=None, overwrite_ok=None)[source]#
Create a copy of a store whose entries’ groups have been updated according to the provided mapping.
- Parameters:
- old_group_to_new_groupMapping[GroupName, GroupName] | Callable[[GroupName], GroupName]
A mapping or callable that transforms an old group name to a new one. Groups in the store that are not included in the mapping are unaffected.
A
GroupName
isstr | None
.- store_nameOptional[None]
If specified, the name of the new store.
- overwrite_okOptional[bool]:
If specified, determines if the mapping can overwrite existing store entries. Otherwise, defers to
ZenStore(overwrite_ok)
.
- Returns:
- new_store
A copy of
self
with remapped groups.
Examples
>>> from hydra_zen import ZenStore
Creating an initial store
>>> s1 = ZenStore("s1") >>> s1({}, group=None, name="a") >>> s1({}, group="A/1", name="b") >>> s1({}, group="A/2", name="c") >>> s1 s1 {None: ['a'], 'A/1': ['b'], 'A/2': ['c']}
Replacing group “A/1” with “B”, via a mapping
>>> s2 = s1.copy_with_mapped_groups({"A/1": "B"}, store_name="s2") >>> s2 s2 {None: ['a'], 'A/2': ['c'], 'B': ['b']}
Placing all entries under group “A/” within a new inner group “p”, via a function
>>> s3 = s1.copy_with_mapped_groups( ... lambda g: g + "/p" if g and g.startswith("A/") else g, store_name="s3" ... ) >>> s3 s3 {None: ['a'], 'A/1/p': ['b'], 'A/2/p': ['c']}
- get_entry(group, name)[source]#
Access a store entry, which is a mapping that specifies the entry’s name, group, package, provider, and node.
- Parameters:
- groupstr | None
- namestr
- Returns:
- dict
name: NodeName
group: GroupName
package: Optional[str]
provider: Optional[str]
node: ConfigType
Notes
Mutating the returned mapping will not affect the store’s internal entry. Mutating a node in the returned entry may have unintended consequences and is not advised.
Examples
- update(_ZenStore__other)[source]#
Updates the store inplace with redundant entries being overwritten.
Can also be applied via the
|=
in-place operator.Examples
>>> s1.update(s2) >>> s1 # s1 now has entries for both f and g s1 {None: ['f', 'g']}
Alternatively, the
|=
operator can be used to update a store inplace.>>> s3 = ZenStore("s3") >>> s3 |= s2 >>> s3 s3 {None: ['g']}
- merge(_ZenStore__other, store_name=None)[source]#
Create a new store by merging two stores.
The new store’s default settings will reflect those of
self
inself.merge(other)
. This can also be applied via the|
operator.Examples
>>> s3 = s1.merge(s2) >>> s3 s1_copy {None: ['f', 'g']}
Alternatively, the
|
operator can be used to merge stores.>>> s4 = s1 | s2 >>> s4 s1_copy {None: ['f', 'g']}
- has_enqueued()[source]#
True
if this store has entries that have not yet been added to Hydra’s config store.- Returns:
- bool
Examples
>>> store({"a": 1}, name) >>> store.has_enqueued() True
>>> store.add_to_hydra_store() >>> store.has_enqueued() False
- enqueue_all()[source]#
Add all of the store’s entries to the queue to be added to hydra’s store.
Examples
>>> store.add_to_hydra_store() >>> store.has_enqueued() False
>>> store.enqueue_all() >>> store.has_enqueued() True
- __iter__()[source]#
Yields all entries in this store.
Notes
Mutating the returned mappings will not affect the store’s internal entries. Mutating a node in an entry may have unintended consequences and is not advised.
Examples
>>> list(store) [{'name': 'a', 'group': 'fruit', 'package': None, 'provider': None, 'node': {'x': 1}}, {'name': 'b', 'group': 'fruit/orange', 'package': None, 'provider': None, 'node': {'x': 2}}, {'name': 'c', 'group': 'veggie', 'package': None, 'provider': None, 'node': {'x': 3}}]
- __eq__(_ZenStore__o)[source]#
Returns
True
if two stores share identical internal repos and queues.Examples
>>> store1 == store1_a True >>> store1 == store2 False
Attributes
name
groups
Returns a sorted list of the groups registered with this store