PyOED Configurations (Configs) System#

Quick Reference

Define configs: subclass PyOEDConfigs as a @dataclass
Bind to object: use the set_configurations() decorator
Access at runtime: obj.configurations, obj.configurations_class
Global defaults: modify SETTINGS attributes
Validate: override validate_configurations()

In order to equip each PyOED object with maximum flexibility and make sure PyOED is highly extensible, we need to assure both accuracy and robustness. Initially PyOED used to configure each object by aggregating keyword arguments with default configurations dictionaries. This proved to be problematic, and we migrated to employing dataclasses. Thus, PyOED is currently adopting configurations data classes with simple, yet powerful, configurations system. This page describes the fundamentals of PyOED configurations as well as the details of the configurations module in PyOED.

Important

The following configurations fundamental rules are mandated by construction in PyOED and are absolutely critical for developers intending to expand PyOED modules/classes or contributing to the package

Tip

To learn about extending PyOED implementations, check the PyOED contribution guide.

PyOED’s Configurations (Configs) Fundamentals#

PyOED configurations system is pyoed.configs.configs.
All PyOED objects inherit the PyOEDObject base class.
All PyOED objects are associated with a configurations object derived from PyOEDConfigs typically named (by convention) as the object with a postfix Configs. For example, a PyOED class ObjectA is typically associated with a ObjectAConfigs configurations class
Each PyOED object is automatically associated with the following attributes:
- _HAS_VALIDATION: a bool class variable describing whether the object has a validate_configurations() function implemented that is different from the one defined by the PyOEDObject base class.
- _CONFIGURATIONS_CLASS: a class variable defining the type of the configurations class. This is typically, the configurations class (derived from PyOEDConfigs) that the developer associates with the PyOED object.
- _CONFIGURATIONS: An instance of the associated configurations class (defined by _CONFIGURATIONS_CLASS) which is created upon instantiating the PyOED object.
The user/developer should never need to access the _HAS_VALIDATION class variable/attribute. Moreover, the two attributes _CONFIGURATIONS_CLASS and _CONFIGURATIONS_CLASS should never be accessed directly. Thus user/developer should always use the corresponding attributes/properties defined by the PyOEDObject base class which are thus associated with all PyOED objects. These properties/attributes are as follows:
- configurations gives access to _CONFIGURATIONS.
- configurations_class gives access to _CONFIGURATIONS_CLASS.
Each PyOED object need to implement methods those override (or call super’s identical one) the following two methods:
1. validate_configurations(): This method checks the types and potentially the values in the configurations object associated with that PyOED object; the configurations object is the one assigned to _CONFIGURATIONS and is accessible by the pyoed.configs.configs.PyOEDObject.validate_configurations.
2. update_configurations(): This method takes a set of key-worded arguments, and associated values and aggregate them with existing (or default) configurations, validate the aggregated configurations, and do further specific updates.
For additional details on how to develop new PyOED objects, see the PyOED contribution guide.
Upon instantiating any PyOED object, the class __mro__ is traversed upward to aggregate all configurations defined by the associated _CONFIGURATIONS_CLASS with its parents. Once the full set of configurations is created, it is validated by calling the method validate_configurations() associated with each object. If no exceptions are thrown, and the configurations are validated (validate_configurations() returns True), the PyOED object is created.

PyOED’s Configurations Module (Classes & Functions)#

Configurations Classes

`PyOEDContainer`()	Base class for creating containers to encompass PyOED's objects configurations and data, etc..
`PyOEDConfigs`(*[, debug, verbose, output_dir])	Base class for PyOED's objects configurations.
`PyOEDData`()	Base class for data/results objects in PyOED.
`PyOEDObject`([configs])	Base class for all PyOED objects.
`PyOEDRulesViolationError`([message])	Exception raised for violating PyOED rules.
`PyOEDConfigsValidationError`([message])	Exception raised for errors in the configurations validators.
`PyOEDABCMeta`(name, bases, namespace, /, **kwargs)	Custom metaclass for PyOED abstract base classes.
`SETTINGS`	Global settings for PyOED.

Configuration Functions

`set_configurations`(configurations_class)	Decorator to bind a configurations class to a `PyOEDObject` subclass.
`remove_unknown_keys`(data, target_dataclass)	Remove keys from data that are not part of target_dataclass.
`extract_unknown_keys`(data, target_dataclass)	Extract the keys from data that are not part of target_dataclass.
`validate_key`(current_configs, new_configs, ...)	Given a configurations dictionary or configurations object (derived from `PyOEDConfigs`), apply the test to the value of the key extracted from configs if it exists.
`aggregate_configurations`(obj, configs, ...)	Given an object obj, configurations object/dictionary configs and a configurations class configs_class, aggregate configs into the current configurations of the object (the attribute _CONFIGURATIONS) if present, otherwise it aggregates configs into the default configurations created by instantiating configs_class.

Configurations Classes#

This module provides access to functionality related to configurations of PyOED objects. For example, any simulation model is instantiated with default configurations some of which can be updated at runtime. This module provides rigorous class and functions to help create, validate and update configurations.

If you are building a new object that inherits from one of the base classes in PyOED, you should familiarize yourself with PyOEDConfigs to understand how to define the parameters of your object. In most instances, you need not to anything more than define a class inheriting PyOEDConfigs and write the fields you need. See the configuration tutorial for more information.

class PyOEDABCMeta(name, bases, namespace, /, **kwargs)[source]#

Bases: ABCMeta

Custom metaclass for PyOED abstract base classes.

Extends ABCMeta with two additional behaviors:

Signature restoration: Fixes the class signature so that inspect.signature(cls) returns the parameters of __init__ (excluding self), which enables accurate Sphinx autodoc output.
Validation detection: On each instantiation, sets the class attribute _HAS_VALIDATION to True if the class defines its own validate_configurations() method, False otherwise.

__init__(clsname, bases, attrs)[source]#

class PyOEDConfigs(*, debug=False, verbose=False, output_dir='./_PYOED_RESULTS_')[source]#

Bases: PyOEDContainer

Base class for PyOED’s objects configurations. All objects (simulation models, optimizers, etc.) have their own configurations. Thus, to unify and simplify checking and configuration definition, we recommend using a class derived from PyOED’s based configuration class PyOEDConfigs.

In the past (up to PyOED 1.0,), configurations were held in dictionaries, which made it possible to pass configurations keys those are ignored by the implementation (e.g., a simulation model). This caused a lot of errors those were hard to track. The intention of this class is to provide access to configurations in a unified form that makes it easy to develop new algorithms while avoiding such complications/errors.

Example — Creating a custom Configs subclass:

from dataclasses import dataclass
from pyoed.configs import PyOEDConfigs

@dataclass(kw_only=True, slots=True)
class MyModelConfigs(PyOEDConfigs):
    num_iterations: int = 100
    tolerance: float = 1e-6

cfg = MyModelConfigs(num_iterations=50)
print(cfg)  # shows all fields with values

Parameters:

verbose (bool) – a boolean flag to control verbosity of the object.
debug (bool) – a boolean flag that enables adding extra functionality in a debug mode
output_dir (str | Path) – the base directory where the output files will be saved.

__init__(*, debug=False, verbose=False, output_dir='./_PYOED_RESULTS_')#

debug: bool#

output_dir: str | Path#

verbose: bool#

class PyOEDContainer[source]#

Bases: object

Base class for creating containers to encompass PyOED’s objects configurations and data, etc..

Note

This is an empty container that provides basic functionality but it does NOT provide any fields/attributes, and is intended to be derived by specific implementations of classes encoding configurations, data, etc.

__init__()#

asdict(deep=False, ignore_error=False)[source]#

Return a dictionary representation of the object attributes. Any object is referenced in the returned configurations.

Parameters:

deep (bool) – a boolean flag to control whether to return a deep copy of the object or not. If True, this mirrors the functionality of asdict from the dataclasses module. If False, only a shallow copy of the values is returned.
ignore_error – if True, errors raised due to using deep are ignored. See the note below about requirement of objects to be picklable/serializable.

Returns:

a dictionary representation of the container with/without deep copy of the objects associated with the container’s attributes.

Return type:

dict

Note

The deep option is tricky as it requires all objets associated with the container attributes to be picklable/serializable. Thus, we added the option ignore_error to ignore such error (if it happens when deep=True) in which case (that is, when both deep and ignore_error are set to True), deep is discarded and a shallow copy is created instead. A proper message is printed to show that.

classmethod data_to_dataclass(data)[source]#

Convert data from a couple different forms into a dataclass instance.

Parameters:

data (dict | dataclass | None) – data dictionary or dataclass instance or None. If None, the default configurations of the dataclass are used. If a dict, the dataclass is populated with the dictionary values. If a dataclass, the dataclass is returned as is.

Raises:

TypeError – if the passed data is not a dictionary, an instance of cls, or None.
PyOEDConfigsValidationError – if instantiating an instance from the dataclass failed.

items()[source]#

Return an iterable of tuples of the object attributes. The tuples are in the form (name, value). This is similar to the items method of a dictionary.

Returns:: an iterable of tuples of the object attributes.
Return type:: Iterable[tuple[str, Any]]

classmethod load_from_file(file)[source]#

A class-level method that returns an instance/object instantiated from saved file. The input file is a binary file created by write_to_file()

Parameters:

cls – the class from which the saved object is expected to be an instance.
file (str | Path) – path to file to load results from

Returns:

an instance of cls created from saved data

Raises:

TypeError – if the file is of the wrong type (string or Path)
TypeError – if the saved data doesn’t match the info needed for instantiating cls (e.g., the unpickled object is not of the type matching cls)
FileNotFoundError – if the results file does not exist
IOError – if failed to read the file

Return type:

object

lookup(name, data)[source]#

A helper function to lookup a name in data if and only if the name is a field of self.

Parameters:

name (str) – the name of the field to lookup.
data (dict | Self) – the dictionary to lookup the name in.

Returns:

a tuple of boolean and value. The boolean indicates whether the name is found in the data or not. The value is the value associated with the name in the data. If the name is not found, the value is None.

Raises:

AttributeError – if the name is not a field in self.

Return type:

tuple[bool, Any]

update(new_configs)[source]#

Update the configurations of this object with the passed configurations. The passed configurations can be a dictionary or an instance of Self.

Parameters:: new_configs (dict | Self) – the configurations to update this object with.

write_to_file(file)[source]#

Dump/save/pickle this object to file so that it can be loaded later and used to instantiate this instance. This contents of this file can be loaded later by invoking the class method load_from_file().

Parameters:: file (str | Path) – path to file to load results from
Raises:: IOError – if failed to write the file

class PyOEDData[source]#

Bases: PyOEDContainer

Base class for data/results objects in PyOED.

Use PyOEDData when you need a container for holding computed results, intermediate data, or output from algorithms (e.g., optimization results, filter/smoother output). Unlike PyOEDConfigs, data containers do not participate in configuration validation and are not bound to PyOEDObject instances.

Supports dictionary-style access via [] subscription in addition to attribute access.

Note

This class is used as data containers, and is used by Results objects in PyOED.

__init__()#

class PyOEDObject(configs=None)[source]#

Bases: object

Base class for all PyOED objects.

This is where the _CONFIGURATIONS attribute is associated with all PyOED objects to hold the configurations/settings of the object. Also, this checks whether a class has a validation method validate_configurations() of its own or not.

Parameters:: configs (dict | PyOEDConfigs | None) – configurations of the object. This can be a full or partial dictionary, or an instance of the registered configurations object.

from pyoed.configs import PyOEDObject

obj = PyOEDObject(configs={"verbose": True})
print(obj.configurations)          # PyOEDConfigs(verbose=True, ...)
print(obj.configurations_class)    # <class 'PyOEDConfigs'>
print(obj.verbose)                 # True

__init__(configs=None)[source]#

aggregate_configurations(configs)[source]#

Aggregate the passed configurations object/dictionary configs into the current configurations (if present, that is after full instantiation of the object/self) or the default configurations if the object has not been full instantiated yet.

Note

This is a wrapper around pyoed.configs.aggregate_configurations

property configurations: PyOEDConfigs#

The configurations dataclass instance associated with this object.

Holds the full set of runtime settings for this object. Individual configuration values are accessible as attributes, e.g. self.configurations.verbose, self.configurations.debug, self.configurations.output_dir.

Note

Prefer the dedicated property shorthands (verbose, debug) over direct attribute access on this object for the most commonly used settings, as those shorthands also trigger validation on assignment.

Return type:: PyOEDConfigs

property configurations_class: Type[PyOEDConfigs]#

The configurations dataclass class (type) associated with this PyOED object.

Set by the @set_configurations decorator and shared across all instances of the same class. Common uses:

Instantiate a fresh default configurations object: self.configurations_class()
Inspect available fields: dataclasses.fields(self.configurations_class)
Convert a plain dictionary to the appropriate dataclass: self.configurations_class.data_to_dataclass(my_dict)

Return type:: Type[PyOEDConfigs]

copy()[source]#

Return a copy of self. The copy is created by serialization.

Return type:: Self

property debug: bool#

Debug mode flag for this object.

When True, the object may perform additional consistency checks, emit extra diagnostic output, or activate development code paths. The exact effect is class-specific; not all classes make use of this flag.

Can be updated at any time via assignment:

obj.debug = True   # enable debug mode
obj.debug = False  # disable debug mode

Return type:: bool

class property default_configurations[source]#

Default configurations for this class (class-level property).

Returns a fresh PyOEDConfigs instance constructed with all default field values, without needing to instantiate the full object. Useful for inspecting available settings or building a partial override:

defaults = MyModel.default_configurations
defaults.verbose = True
model = MyModel(configs=defaults)

None is returned if this class is not associated with any configurations class.

This is a shorthand for get_default_configurations().

Return type:: PyOEDConfigs | None

classmethod get_configurations_class()[source]#

A class-level method that returns the configurations class associated with this class if this class is associated with one. This inspects whether the attribute _CONFIGURATIONS_CLASS is available or not. None is returned if no configurations class is set.

classmethod get_default_configurations()[source]#

A class-level method that returns the default configurations associated with this class if this class is associated with one. This inspects whether the attribute _CONFIGURATIONS_CLASS is available or not. None is returned if no configurations class is set.

Return type:: PyOEDConfigs | None

classmethod load_from_file(file)[source]#

A class-level method that returns an instance/object instantiated from saved file. The input file is a binary file created by write_to_file()

Parameters:

cls – the class from which the saved object is expected to be an instance.
file (str | Path) – path to file to load results from

Returns:

an instance of cls created from saved data

Raises:

TypeError – if the file is of the wrong type (string or Path)
TypeError – if the saved data doesn’t match the info needed for instantiating cls (e.g., the unpickled object is not of the type matching cls)
FileNotFoundError – if the results file does not exist
IOError – if failed to read the file

Return type:

object

update_configurations(**kwargs)[source]#: Take any set of keyword arguments, and lookup each in the configurations object, and update the values accordingly.

Note

This method is elementary and does not do any type of casting. If the user want to do any processing on the data, this method need to be overriden with custom implementation.

Warning

This method can be only called after the PyOEDObject is fully instantiated. Thus, it cannot be called withing __init__ method of the PyOED object.

validate_configurations(configs, raise_for_invalid=True)[source]#

Each simulation model MUST implement it’s own function that validates its own configurations. If the validation is self contained (validates all configuations), then that’s it. However, one can just validate the configurations of of the immediate class and call super to validate configurations associated with the parent class.

If one does not wish to do any validation (we strongly advise against that), simply add the signature of this function to the model class.

Note

The purpose of this method is to make sure that the settings in the configurations object self._CONFIGURATIONS are of the right type/values and are conformable with each other. This function is called upon instantiation of the object, and each time a configuration value is updated. Thus, this function need to be inexpensive and should not do heavy computations.

Parameters:

configs (dict | PyOEDConfigs) – configurations to validate. If a PyOEDConfigs object is passed, validation is performed on the entire set of configurations. However, if a dictionary is passed, validation is performed only on the configurations corresponding to the keys in the dictionary.

Raises:

PyOEDConfigsValidationError – if the configurations are invalid and raise_for_invalid is set to True.
AttributeError – if any (or a group) of the configurations does not exist in the model configurations PyOEDConfigs.

property verbose: bool#

Verbosity flag controlling console/log output for this object.

When True, the object emits informational messages during execution (e.g., progress updates, iteration counts, diagnostic info). When False, the object operates silently.

Can be updated at any time via assignment:

obj.verbose = True   # enable verbosity
obj.verbose = False  # silence output

Return type:: bool

write_to_file(file)[source]#

Dump/save/pickle this object to file so that it can be loaded later and used to instantiate this instance. This contents of this file can be loaded later by invoking the class method load_from_file().

Parameters:: file (str | Path) – path to file to load results from
Raises:: IOError – if failed to write the file

exception PyOEDConfigsValidationError(message=None)[source]#

Exception raised for errors in the configurations validators.

Parameters:: message (str | None) – explanation of the error. Default message is created if None is passed.

exception PyOEDRulesViolationError(message=None)[source]#

Exception raised for violating PyOED rules.

Parameters:: message (str | None) – explanation of the error. Default message is created if None is passed.

Configurations Functions#

set_configurations(configurations_class)[source]#

Decorator to bind a configurations class to a PyOEDObject subclass.

This is syntactic sugar for setting the _CONFIGURATIONS_CLASS attribute of the class. It ensures that the configurations class is a valid subclass of PyOEDConfigs.

Parameters:: configurations_class (Type[PyOEDConfigs]) – the configurations class to set. Must be a subclass of PyOEDConfigs.
Returns:: the decorated class with _CONFIGURATIONS_CLASS set.
Return type:: type

from dataclasses import dataclass
from pyoed.configs import PyOEDConfigs, PyOEDObject, set_configurations

@dataclass(kw_only=True, slots=True)
class SolverConfigs(PyOEDConfigs):
    max_iter: int = 200

@set_configurations(SolverConfigs)
class Solver(PyOEDObject):
    pass

solver = Solver()
print(solver.configurations)  # SolverConfigs(max_iter=200, ...)

class_doc_inheritance(cls, aggr_params=True, aggr_raises=False, aggr_examples=False, aggr_returns=False, ignore_common_params=True)[source]#

A function (also can be used as a class wrapper) that modifies (in-place) class’s docstring __doc__ to include parameters, raises, etc., defined by the child as well as the parent class. This function takes the description from the passed class cls and aggregates parameters, etc., from itself with its parents.

Note

The package docstring_parser gives access to the following elements. If expanded, we can adapt:

depracation

description

params

examples

returns

many_returns

meta

raises

short_description

Parameters:

cls – The class which documentation __doc__ is to be modified
aggr_params – aggregate parameter clauses (:param :) from both cls and its parent class
aggr_raises – aggregate raise clauses from both cls and its parent class
aggr_examples – aggregate example clauses from both cls and its parent class
aggr_returns – aggregate return clauses from both cls and its parent class
ignore_common_params – Only include parameters in child

Returns:

The passed class cls so this function can be used as a wrapper

remove_unknown_keys(data, target_dataclass)[source]#

Remove keys from data that are not part of target_dataclass.

Parameters:

data (dict | PyOEDConfigs) – the dictionary or configs object to remove keys from.
target_dataclass (Type[PyOEDConfigs]) – the target dataclass whose fields define the recognized keys.

Returns:

a dictionary holding keys/values in data that are acceptable/recognized by (i.e., members of) target_dataclass.

Return type:

dict

Raises:

TypeError – if the passed arguments are of wrong type

extract_unknown_keys(data, target_dataclass)[source]#

Extract the keys from data that are not part of target_dataclass. The extracted keys (if any) and the associated values are inserted into a dictionary that is returned.

Parameters:

data (dict | PyOEDConfigs) – the dictionary or configs object to extract unknown keys from.
target_dataclass (Type[PyOEDConfigs]) – the target dataclass whose fields define the recognized keys.

Returns:

a dictionary holding keys in data that are not recognized (i.e., not part of) the target_dataclass.

Return type:

dict

Raises:

TypeError – if the passed arguments are of wrong type

validate_key(current_configs, new_configs, key, test, message=None, raise_for_invalid=True)[source]#

Given a configurations dictionary or configurations object (derived from PyOEDConfigs), apply the test to the value of the key extracted from configs if it exists.

Parameters:

current_configs (PyOEDConfigs) – a configurations object to which the key is expected to fit.
new_configs (dict | PyOEDConfigs) – a configurations dictionary/object holding a subset (or all) of keys or attributes of current_configs to test validity of.
key (str) – name of the new configurations key/attribute to lookup inside new_configs and apply the test to.
message (str | callable | None) – the error message to use if the test fails and raise_for_invalid is True.
test (callable) –
a test that takes as input the value associated with the key attribute/key in new_configs, and returns a bool that is True if the test is satisfied, and False otherwise. For example, the following test is passed if the user wants to make sure new_configs.key > 4.

` test = lambda v: v > 4 `

Note

The logic of this function is as follows:

Assertion error is raised if any of the arguments is invalid

A custom PyOEDConfigsValidationError exception is raised if raise_for_invalid is True and the test is not satisfied or if key is not a member of current_configs.

Return the result (bool value) of test applied to the value associated with new_configs.key returns.

Note

If the message is a callable, it is supposed to take the value as input and return a string that is passed to the exception constructor.

Returns:: a boolean flag based on the output of test
Return type:: bool

aggregate_configurations(obj, configs, configs_class)[source]#

Given an object obj, configurations object/dictionary configs and a configurations class configs_class, aggregate configs into the current configurations of the object (the attribute _CONFIGURATIONS) if present, otherwise it aggregates configs into the default configurations created by instantiating configs_class.

Parameters:

obj (object) – the object whose _CONFIGURATIONS attribute is used as the base for aggregation. If the attribute is absent (pre-instantiation), the default configurations from configs_class are used.
configs (PyOEDConfigs | dict) – a configurations object or dictionary of partial configs to merge into the base.
configs_class (Type[PyOEDConfigs]) – the configurations class defining valid fields.

Returns:

an aggregated configurations object with values from configs overlaid onto the base configurations.

Return type:

PyOEDConfigs