Caching#
Langworks implements various utilities allowing for client-side caching of LLM output.
- class langworks.caching.ScoredItem#
Wrapping class to store any cachable item alongside a score. To be used in conjunction with ScoredCache or any of its derivates.
Properties#
- item: Any = None#
Item stored.
- score: float = 0#
Score attributed to the item.
- class langworks.caching.ScoredCache#
A regular cache with a scoring mechanism added to it to control overwrites, only overwriting a cached value if the new value is associated with a higher score as defined by the user.
Note
As ScoredCache is implemented on top of Thomas Kemmer’s cachetools, ScoredCache may also be used under the MIT License under the copyright of Rosaia B.V. (2025).
Fundamentals#
- __init__(maxsize: int, getsizeof: Callable[[Any], int] = None)#
Initializes the cache.
Parameters#
- maxsize
Maximum size up until which the cache may grow, after which the cache will start pruning items to make room for new keys.
- getsizeof
Callable used to determine the size of cached items.
- __setitem__(key: ~typing.Hashable, item: ~langworks.caching.ScoredItem, __cache_getitem__: ~typing.Callable = <function Cache.__getitem__>, __cache_setitem__: ~typing.Callable = <function Cache.__setitem__>)#
Adds an item to the cache.
Unlike regular caches, these items need to be wrapped using a ScoredItem object, holding the actual item, as well as the score attributed to this item.
Parameters#
- key
Key used to index the item.
- item
Item to store, wrapped in a CachedItem, consisting of an item and a score.
- class langworks.caching.ScoredLazyLFUCache#
A scored Least Frequently Used (LFU) cache, implemented in a ‘lazy’ manner, only returning cache hits for a key if said key has been requested more than a set number of times during availability for retrieval. In doing so, this cache effectively samples items before storing them.
Note
As ScoredLazyLFUCache is implemented on top of Thomas Kemmer’s cachetools, ScoredLazyLFUCache may also be used under the MIT License under the copyright of Rosaia B.V. (2025).
Fundamentals#
- __init__(maxsize: int, getsizeof: Callable[[Any], int] = None, threshold: int = 3)#
Initializes the cache.
Parameters#
- maxsize
Maximum size up until which the cache may grow, after which the cache will start pruning items to make room for new keys.
- getsizeof
Callable used to determine the size of cached items.
- threshold
Minimum number of hits required before the cache starts returning for a key.
- class langworks.caching.ScoredLazyLRUCache#
A scored Least Recently Used (LRU) cache, implemented in a ‘lazy’ manner, only returning cache hits for a key if said key has been requested more than a set number of times during availability for retrieval. In doing so, this cache effectively samples items before storing them.
Note
As ScoredLazyLRUCache is implemented on top of Thomas Kemmer’s cachetools, ScoredLazyLRUCache may also be used under the MIT License under the copyright of Rosaia B.V. (2025).
Fundamentals#
- __init__(maxsize: int, getsizeof: Callable[[Any], int] = None, threshold: int = 3)#
Initializes the cache.
Parameters#
- maxsize
Maximum size up until which the cache may grow, after which the cache will start pruning items to make room for new keys.
- getsizeof
Callable used to determine the size of cached items.
- threshold
Minimum number of hits required before the cache starts returning for a key.