Llama.cpp

Llama.cpp#

class langworks.middleware.llama_cpp.LlamaCpp#

Llama.cpp is an open source LLM inference engine designed to efficiently run on a wide range of hardware, including CPU-only and CPU+GPU hybrid configurations.

Langworks’ llama.cpp middleware provides a wrapper to access the LLMs served using llama.cpp server API.

Dependencies#

To employ the llama.cpp middleware additional depedencies need to be installed. You may do so using pip:

pip install langworks[llamacpp]

Be sure to employ llama.cpp with LLGuidance enabled to make full use of all features provided by Langworks. It is furthermore recommended to initialize llama.cpp with --parallel set to a value higher than 1, so that llama.cpp may handle multiple requests simulataneously.

Fundamentals#

__init__(url: str | Sequence[str], model: str, authenticator: Authenticator | Sequence[Authenticator] | None = None, timeout: int = 5, retries: int = 2, autoscale_threshold: tuple[float, float] = (0, 0), params: SamplingParams = None, output_cache: ScoredCache = None)#

Initializes the llama.cpp middleware.

Parameters#

Connection#

url: URL or URLs of llama.cpp-instances where the model may be accessed.

Changed in version 0.3.0: A list of URLs may be specified, allowing to simultaneously access multiple inference endpoints.
model: Name of the model as assigned by the instance.
authenticator: Authenticator that may be used to acquire an authentication key when the llama.cpp-instance requests for authentication. Optionally, a list of authenticators may also be provided, allowing to specify an authenticator per llama.cpp -instance as identified by the URLs in url.

Changed in version 0.3.0: A list of authenticators may be specified, making it possible to manage authentication per llama.cpp-instance as specified using url.
timeout: The number of seconds the client awaits the response of the llama.cpp-instance.
retries: The number of times the client tries to submit the same request again after a timeout.

Balancing#

autoscale_threshold: Pair specifying at what number of tasks per instance to scale up (first item) or scale down (second item). By default this is set to (0, 0), setting the middleware up to immediately scale up to use all resources, while never scaling down.

Added in version 0.3.0.

Configuration#

params: Default sampling parameters to use when processing a prompt, specified using an instance of SamplingParams.
output_cache: Cache to use for caching previous prompt outputs. Should be a ScoredCache or subclass.

Methods#

exec(query: str = None, role: str = None, guidance: str = None, history: Thread = None, context: dict = None, params: SamplingParams = None) → tuple[Thread, dict[str, Any]]#

Generate a new message, following up on the message passed using the given guidance and sampling parameters.

Parameters#

query: The query to prompt the LLM with, optionally formatted using Langworks’ static DSL.
role: The role of the agent stating this query, usually ‘user’, ‘system’ or ‘assistant’.
guidance: Template for the message to be generated, formatted using Langworks’ dyanmic DSL.
history: Conversational history (thread) to prepend to the prompt.
context: Context to reference when filling in the templated parts of the query, guidance and history. In case the Langwork or the input also define a context, the available contexts are merged. When duplicate attributes are observed, the value is copied from the most specific context, i.e. input context over Query context, and Query context over Langwork context.
params: Sampling parameters, wrapped by a SamplingParams object, specifying how the LLM should select subsequent tokens.

class langworks.middleware.llama_cpp.SamplingParams#

Specifies additional sampling parameters to be used when passing a prompt to llama.cpp.

Properties#

dry: bool = False#: Enables sampling using the DRY repetition penalty, offering more fine-grained control over repetitions than repetition_penalty.

See also

Weidmann, P.E. (2024). DRY: A modern repetition penalty that reliably prevents looping. oobabooga/text-generation-webui#5677.

dry_base: float = 1.75#: Controls how quickly the DRY repetition penalty grows as repetitions repeat.

dry_last_n: int = -1#: The number of previous tokens to consider when applying DRY sampling.

Note

Within llama.cpp this parameter is referred to as dry_penalty_last_n.

dry_min_length: int = 2#: Minimal required length of repeated tokens subject to the DRY repetition penalty.

Note

Within llama.cpp this parameter is referred to as dry_allowed_length.

dry_multiplier: float = 0.0#: Controls extent to which the DRY repetition penalty is applied.

dry_sequence_breakers: list[str] = None#: List of character sequences, which if generated reset the DRY repetition penalty.

frequency_penalty: float = 0.0#: A number between -2.0 and 2.0 that controls the number of times a tokens appears in the generated text, whereby a positive coefficient puts a constraint on repetition, whereas a negative coefficient encourages repetition.

ignore_eos: bool = False#: Flag that controls whether or not generation should continue after the End of Sequence (EOS) token has been generated.

include_stop: bool = True#: Flag that controls whether stop sequences are included in the output.

logit_bias: dict[int, int] = None#: A dictionary indexed by encoded tokens, each assigned a number between -100 and 100 controlling the likelihood that a specific token is selected or ignored, with a positive number increasing this likelihood, whereas a negative number decreases it.

logprobs: int = None#: The number of tokens to generate per position in the generated sequences, ordered by probability.

max_p: float = 0.1#: Puts an upper bound on the probability of the sampled tokens under consideration, allowing to discard the most likely tokens, increasing originality.

Note

Within llama.cpp this parameter is referred to as xtc_threshold.

See also

Weidmann, P.E. (2024). Exclude Top Choices (XTC): A sampler that boosts creativity, breaks writing clichés, and inhibits non-verbatim repetition. oobabooga/text-generation-webui#6335.

max_p_allowance: float = 0.0#: Controls probability that the least likely sampled token above the threshold specified by max_p is included despite exceeding the threshold.

Note

Within llama.cpp this parameter is referred to as xtc_probability.

max_tokens: int = None#: The number of tokens that may be generated per generation.

min_p: float = 0.0#: A number between 0.0 and 1.0, specifying what a token’s minimal likelihood must be to be considered for selection.

mirostat: Literal[0, 1, 2] = 0#

Enables Mirostat sampling, controlling perplexity during text generation.

A value of 0 disables Mirostat, 1 enables Mirostat, and 2 enables Mirostat 2.0.

Llama.cpp

Contents

Llama.cpp#

Dependencies#

Fundamentals#

Parameters#

Connection#

Balancing#

Configuration#

Methods#

Parameters#

Properties#