Middleware

Langworks implements various middleware for interfacing with commonly used LLM tooling.

class langworks.middleware.generic.Middleware

Generalised interface for accessing LLMs.

Methods

exec(query: str = None, role: str = None, guidance: str = None, history: Thread = None, context: dict = None, params: SamplingParams = None) → tuple[Thread, dict[str, Any]]

Generate a new message, following up on the message passed using the given guidance and sampling parameters.

Parameters

query

The query to prompt the LLM with, optionally formatted using Langworks’ static DSL.

role

The role of the agent stating this query, usually ‘user’, ‘system’ or ‘assistant’.

guidance

Template for the message to be generated, formatted using Langworks’ dyanmic DSL.

history

Conversational history (thread) to prepend to the prompt.

context

Context to reference when filling in the templated parts of the query, guidance and history. In case the Langwork or the input also define a context, the available contexts are merged. When duplicate attributes are observed, the value is copied from the most specific context, i.e. input context over Query context, and Query context over Langwork context.

params

Sampling parameters, wrapped by a SamplingParams object, specifying how the LLM should select subsequent tokens.

class langworks.middleware.generic.SamplingParams

Generic class used to specify with which sampling parameters a prompt should be processed.

Properties

frequency_penalty: float = 0.0

A number between -2.0 and 2.0 that controls the number of times a tokens appears in the generated text, whereby a positive coefficient puts a constraint on repetition, whereas a negative coefficient encourages repetition.

ignore_eos: bool = False

Flag that controls whether or not generation should continue after the End of Sequence (EOS) token has been generated.

include_stop: bool = True

Flag that controls whether stop sequences are included in the output.

logit_bias: dict[int, int] = None

A dictionary indexed by encoded tokens, each assigned a number between -100 and 100 controlling the likelihood that a specific token is selected or ignored, with a positive number increasing this likelihood, whereas a negative number decreases it.

logprobs: int = None

The number of tokens to generate per position in the generated sequences, ordered by probability.

max_tokens: int = None

The number of tokens that may be generated per generation.

min_p: float = 0.0

A number between 0.0 and 1.0, specifying what a token’s minimal likelihood must be to be considered for selection.

presence_penalty: float = 0.0

A number between -2.0 and 2.0 that controls the likelihood that tokens appear that have not yet appeared in the generated text, whereby a positive coeffient increases these odds, whereas a negative coefficient decreases them.

repetition_penalty: float = 0.0

A number between -2.0 and 2.0 that controls the degree of repetition, taking into account both the generated text and the initial prompt. A positive number encourages usage of new tokens, whereas a negative number favours repetition.

seed: int = None

A number used to initialize any pseudorandom number generations that the model may used. It can be used to enforce a degree of determinism even when using non-nil temperatures.

stop: list[str] = None

List of character sequences, which if generated stop further generation.

stop_tokens: list[int] = None

List of encoded tokens, which if generated stop further generation.

temperature: float = 1.0

A number between 0.0 and 2.0, with higher values increasing randomization, whereas lower values encourage deterministic output.

top_k: int = -1

The number of tokens to consider when deciding on the next token when generating.

top_p: float = 1.0

A number between 0.0 and 1.0, specifying what is considered a top percentile, constraining the selection of tokens to tokens with probabilities considered among these top percentiles. For example, when 0.2 is specified, a selection is made from the tokens with probabilities among the top 20 percentiles.