mstar.engine.resources.kv.rolling.config#
What a model declares about a rolling KV store.
A request’s keys and values are a stream it appends to step by step. The store
keeps the stream’s first sink tokens and its last window (the step’s own
included), in stream order and compacted: once a step would overflow, the
retained window shifts left behind the sink. A step’s retained tokens and its
own are therefore always one contiguous range, which the model hands to
whatever attention it runs (a ragged kernel, say). The store runs no attention.
The K/V live in one pool reserved at startup (pool_tokens a layer). Each
request takes a contiguous region of exactly its sink + window there, so
requests of different sizes share the pool without each holding the largest.
Unlike bounded KV, whose window is a ring that its own kernel reads in pieces, the range here is one slice, so the same kernel over the same keys gives the same bits as a cache the model kept itself.
Classes
|
|
|
The tokens a request keeps a layer, its sink plus window, declared when it opens: it waits in line for that much of the pool before its first step, instead of having that step refused until room frees up. |
|
|
|
Each segment's |
|
Where a request's stream stands when a step starts, and what it keeps. |
- class mstar.engine.resources.kv.rolling.config.RollingKVConfig(num_layers: int, num_heads: int, head_dim: int, pool_tokens: int, dtype: torch.dtype | None = None)[source]#
Bases:
object
- class mstar.engine.resources.kv.rolling.config.RollingKVReqConfig(tokens)[source]#
Bases:
ResourceReqConfigThe tokens a request keeps a layer, its sink plus window, declared when it opens: it waits in line for that much of the pool before its first step, instead of having that step refused until room frees up.
- Parameters:
tokens (int)
- validate(spec=None)[source]#
Raise
ValueErrorif a client-supplied value is out of range, or one the resource built fromspeccannot honour.The conductor calls it at admission, so a bad request is a 400 before any worker sees it.
- Parameters:
spec (NodeResourceSpec | None)
- Return type:
None
- class mstar.engine.resources.kv.rolling.config.RollingKVSpec(resource_key: str, nodes: set[str], config: mstar.engine.resources.kv.rolling.config.RollingKVConfig)[source]#
Bases:
NodeResourceSpec- Parameters:
resource_key (str)
config (RollingKVConfig)
- apply_yaml_overrides(pool_tokens=None)[source]#
The pool’s size is the deployment’s memory trade: what it reserves at startup against how many requests, and how large, it holds at once.
- Parameters:
pool_tokens (int | None)
- config: RollingKVConfig#
- class mstar.engine.resources.kv.rolling.config.RollingKVStep(segments=None, positions=<factory>, *, combined_labels=<factory>)[source]#
Bases:
ResourceStepEach segment’s
spanis the tokens its request writes this step.The positions come from the model, not the store, so a plan never waits on the previous step’s commit;
admitchecks them against what was committed.- Parameters:
- class mstar.engine.resources.kv.rolling.config.RollingPosition(written, sink, window)[source]#
Bases:
NamedTupleWhere a request’s stream stands when a step starts, and what it keeps.