mstar.engine.resources.kv.rolling#

class mstar.engine.resources.kv.rolling.RollingKVConfig(num_layers: int, num_heads: int, head_dim: int, pool_tokens: int, dtype: torch.dtype | None = None)[source]#

Bases: object

Parameters:
  • num_layers (int)

  • num_heads (int)

  • head_dim (int)

  • pool_tokens (int)

  • dtype (dtype | None)

dtype: dtype | None = None#
head_dim: int#
num_heads: int#
num_layers: int#
pool_tokens: int#
token_bytes(tokens, dtype)[source]#

tokens a layer, keys and values of every layer.

Parameters:
Return type:

int

class mstar.engine.resources.kv.rolling.RollingKVReqConfig(tokens)[source]#

Bases: ResourceReqConfig

The tokens a request keeps a layer, its sink plus window, declared when it opens: it waits in line for that much of the pool before its first step, instead of having that step refused until room frees up.

Parameters:

tokens (int)

tokens: int#
validate(spec=None)[source]#

Raise ValueError if a client-supplied value is out of range, or one the resource built from spec cannot honour.

The conductor calls it at admission, so a bad request is a 400 before any worker sees it.

Parameters:

spec (NodeResourceSpec | None)

Return type:

None

class mstar.engine.resources.kv.rolling.RollingKVSpec(resource_key: str, nodes: set[str], config: mstar.engine.resources.kv.rolling.config.RollingKVConfig)[source]#

Bases: NodeResourceSpec

Parameters:
apply_yaml_overrides(pool_tokens=None)[source]#

The pool’s size is the deployment’s memory trade: what it reserves at startup against how many requests, and how large, it holds at once.

Parameters:

pool_tokens (int | None)

config: RollingKVConfig#
property resource_class: type[Resource]#

What builds this spec. Imported inside the property, so declaring a resource stays free of the manager and its kernels.

The builder, not necessarily the class built: an attention spec names AttentionManager, whose build picks a backend subclass.

class mstar.engine.resources.kv.rolling.RollingKVStep(segments=None, positions=<factory>, *, combined_labels=<factory>)[source]#

Bases: ResourceStep

Each segment’s span is the tokens its request writes this step.

The positions come from the model, not the store, so a plan never waits on the previous step’s commit; admit checks them against what was committed.

Parameters:
positions: dict#
class mstar.engine.resources.kv.rolling.RollingPosition(written, sink, window)[source]#

Bases: NamedTuple

Where a request’s stream stands when a step starts, and what it keeps.

Parameters:
sink: int#

Alias for field number 1

window: int#

Alias for field number 2

written: int#

Alias for field number 0

Modules

config

What a model declares about a rolling KV store.

manager

A pool of rolling sink + window K/V, one contiguous region per request.