mstar.engine.resources.kv.rolling.config#

What a model declares about a rolling KV store.

A request’s keys and values are a stream it appends to step by step. The store keeps the stream’s first sink tokens and its last window (the step’s own included), in stream order and compacted: once a step would overflow, the retained window shifts left behind the sink. A step’s retained tokens and its own are therefore always one contiguous range, which the model hands to whatever attention it runs (a ragged kernel, say). The store runs no attention.

The K/V live in one pool reserved at startup (pool_tokens a layer). Each request takes a contiguous region of exactly its sink + window there, so requests of different sizes share the pool without each holding the largest.

Unlike bounded KV, whose window is a ring that its own kernel reads in pieces, the range here is one slice, so the same kernel over the same keys gives the same bits as a cache the model kept itself.

Classes

RollingKVConfig(num_layers, num_heads, ...)

RollingKVReqConfig(tokens)

The tokens a request keeps a layer, its sink plus window, declared when it opens: it waits in line for that much of the pool before its first step, instead of having that step refused until room frees up.

RollingKVSpec(resource_key, nodes, config)

RollingKVStep([segments, positions, ...])

Each segment's span is the tokens its request writes this step.

RollingPosition(written, sink, window)

Where a request's stream stands when a step starts, and what it keeps.

class mstar.engine.resources.kv.rolling.config.RollingKVConfig(num_layers: int, num_heads: int, head_dim: int, pool_tokens: int, dtype: torch.dtype | None = None)[source]#

Bases: object

Parameters:
  • num_layers (int)

  • num_heads (int)

  • head_dim (int)

  • pool_tokens (int)

  • dtype (dtype | None)

dtype: dtype | None = None#
head_dim: int#
num_heads: int#
num_layers: int#
pool_tokens: int#
token_bytes(tokens, dtype)[source]#

tokens a layer, keys and values of every layer.

Parameters:
Return type:

int

class mstar.engine.resources.kv.rolling.config.RollingKVReqConfig(tokens)[source]#

Bases: ResourceReqConfig

The tokens a request keeps a layer, its sink plus window, declared when it opens: it waits in line for that much of the pool before its first step, instead of having that step refused until room frees up.

Parameters:

tokens (int)

tokens: int#
validate(spec=None)[source]#

Raise ValueError if a client-supplied value is out of range, or one the resource built from spec cannot honour.

The conductor calls it at admission, so a bad request is a 400 before any worker sees it.

Parameters:

spec (NodeResourceSpec | None)

Return type:

None

class mstar.engine.resources.kv.rolling.config.RollingKVSpec(resource_key: str, nodes: set[str], config: mstar.engine.resources.kv.rolling.config.RollingKVConfig)[source]#

Bases: NodeResourceSpec

Parameters:
apply_yaml_overrides(pool_tokens=None)[source]#

The pool’s size is the deployment’s memory trade: what it reserves at startup against how many requests, and how large, it holds at once.

Parameters:

pool_tokens (int | None)

config: RollingKVConfig#
property resource_class: type[Resource]#

What builds this spec. Imported inside the property, so declaring a resource stays free of the manager and its kernels.

The builder, not necessarily the class built: an attention spec names AttentionManager, whose build picks a backend subclass.

class mstar.engine.resources.kv.rolling.config.RollingKVStep(segments=None, positions=<factory>, *, combined_labels=<factory>)[source]#

Bases: ResourceStep

Each segment’s span is the tokens its request writes this step.

The positions come from the model, not the store, so a plan never waits on the previous step’s commit; admit checks them against what was committed.

Parameters:
positions: dict#
class mstar.engine.resources.kv.rolling.config.RollingPosition(written, sink, window)[source]#

Bases: NamedTuple

Where a request’s stream stands when a step starts, and what it keeps.

Parameters:
sink: int#

Alias for field number 1

window: int#

Alias for field number 2

written: int#

Alias for field number 0