mstar.engine.resources.kv.rolling#
- class mstar.engine.resources.kv.rolling.RollingKVConfig(num_layers: int, num_heads: int, head_dim: int, pool_tokens: int, dtype: torch.dtype | None = None)[source]#
Bases:
object
- class mstar.engine.resources.kv.rolling.RollingKVReqConfig(tokens)[source]#
Bases:
ResourceReqConfigThe tokens a request keeps a layer, its sink plus window, declared when it opens: it waits in line for that much of the pool before its first step, instead of having that step refused until room frees up.
- Parameters:
tokens (int)
- validate(spec=None)[source]#
Raise
ValueErrorif a client-supplied value is out of range, or one the resource built fromspeccannot honour.The conductor calls it at admission, so a bad request is a 400 before any worker sees it.
- Parameters:
spec (NodeResourceSpec | None)
- Return type:
None
- class mstar.engine.resources.kv.rolling.RollingKVSpec(resource_key: str, nodes: set[str], config: mstar.engine.resources.kv.rolling.config.RollingKVConfig)[source]#
Bases:
NodeResourceSpec- Parameters:
resource_key (str)
config (RollingKVConfig)
- apply_yaml_overrides(pool_tokens=None)[source]#
The pool’s size is the deployment’s memory trade: what it reserves at startup against how many requests, and how large, it holds at once.
- Parameters:
pool_tokens (int | None)
- config: RollingKVConfig#
- class mstar.engine.resources.kv.rolling.RollingKVStep(segments=None, positions=<factory>, *, combined_labels=<factory>)[source]#
Bases:
ResourceStepEach segment’s
spanis the tokens its request writes this step.The positions come from the model, not the store, so a plan never waits on the previous step’s commit;
admitchecks them against what was committed.- Parameters:
- class mstar.engine.resources.kv.rolling.RollingPosition(written, sink, window)[source]#
Bases:
NamedTupleWhere a request’s stream stands when a step starts, and what it keeps.
Modules