Specialised pipeworks

Pypeworks ships with various pipeworks tailored for specific, commonly recurring tasks.

class pypeworks.pipeworks.Replicator

A Replicator is a procedually generated pipework generated from a single pypeworks.Node, whereby said node is replicated in parallel together with a network of connections that distributes input across these replications. Replicators are ideally suited to scale nodes that have a long execution time.

__init__(node: Node[T], replications: int, processes: int = None, nocopy: bool = False, copying: None | Literal['auto', 'deep', 'shallow'] = 'deep')

Instantiates a new Replicator.

Parameters

node

Node to replicate.

replications

The number of replications to generate.

processes

The number of worker processes to use to operate the underlying pypeworks.Pipework. By default this number is equal to the number of logical CPUs in the system.

nocopy

Deprecated since version v0.3.0: Use copying instead. This argument will be removed by v0.4.0.

copying

Specifies whether and how to assign each replication its own of the node. By default (“deep”) each replication is assigned a deep copy of the given node. Alternatively, one may also opt for a shallow copy (using “shallow”), allowing for easier handling of nodes wrapping unpickable objects. When set to “auto”, the replicator will first try to make a deep copy, opting for a shallow copy if this fails. If no copying is desired, this value may be set to None, allowing for the node to be shared among replications. However, beware thread-safety and side-effects when working with (automatic) shallow or none-copies!