mstar.engine.resources.kv.rolling.manager#
A pool of rolling sink + window K/V, one contiguous region per request.
The pool (pool_tokens a layer, keys and values) is reserved at startup. A
request declares the tokens it keeps, its sink plus window, when it opens
(RollingKVReqConfig), and waits in the engine’s admission queue, in arrival
order, until the pool has room for it (see Resource.ask). The room is set
aside then; the request’s first step takes a region of exactly that size, first
fit, and gives it back when the request is removed. When no free gap is large
enough but the free tokens are, the resident regions slide left to join the gaps.
Functions
|
A step's write range and kept length in K/V that hold |
Classes
|
Move |
|
|
|
|
|
- class mstar.engine.resources.kv.rolling.manager.Compaction(src, dst, length)[source]#
Bases:
NamedTupleMove
lengthtokens fromsrctodstin a request’s K/V (src > dst).
- class mstar.engine.resources.kv.rolling.manager.RollingKVManager(config, device, kv_dtype=None, name='rolling_kv')[source]#
Bases:
Resource- Parameters:
config (RollingKVConfig)
device (torch.device)
kv_dtype (torch.dtype | None)
name (str)
- admit(step, ctx)[source]#
Reserve space for the given step. In the case where requests in a batch must be executed sequentially, this may be called for all requests in a loop before the per-request plan -> forward -> commit cycle.
- Parameters:
step (RollingKVStep)
ctx (StepContext)
- Return type:
- ask(rid, node_name, graph_walk)[source]#
ASKS while
ridwaits for this resource’s capacity to start onnode_name, None when it needs none, or the outcome that fails it when it can never fit.- Parameters:
- Return type:
AdmitOutcome | None
- classmethod build(spec, info)[source]#
- Parameters:
spec (RollingKVSpec)
info (EngineResourceInfo)
- Return type:
- commit(step, ctx)[source]#
record step consumption
- Parameters:
step (RollingKVStep)
ctx (StepContext)
- Return type:
None
- property current: RollingPlan#
- ingest_request(rid, overrides=None)[source]#
Open the request’s state. The worker calls this once per request, on its first NewRequest, though one comes per partition it serves.
- Parameters:
overrides (ResourceReqConfig | None)
- plan(step, ctx)[source]#
Each row’s write range, and the shift that makes room for it.
The shift runs here, on the stream the step then runs on (the store has no pre-plan), so it is ordered after the previous step’s reads of the request’s K/V and before this step’s writes.
- Parameters:
step (RollingKVStep)
ctx (StepContext)
- Return type:
- post_warmup_validate()[source]#
For, e.g., the KV cache to check that num_free_pages is identical across TP ranks after cuda graph capture.
Raises an error (fails loudly) if invalid.
- reserve(rid, node_name, graph_walk, published)[source]#
Take what
ridasked for and return None, or return a mark of what it didn’t fit against for grew_since.
- rollback_admit(step, ctx)[source]#
Free the regions this step’s admit placed, setting their room aside again for the retry; a region a request already held stays with it.
- Parameters:
step (RollingKVStep)
ctx (StepContext)
- Return type:
None
- unreserve(rid)[source]#
Give back what reserve took, because another resource
ridasks didn’t fit.- Return type:
None
- class mstar.engine.resources.kv.rolling.manager.RollingPlan(rows)[source]#
Bases:
NamedTuple- Parameters:
rows (tuple[RollingRow, ...])
- rows: tuple[RollingRow, ...]#
Alias for field number 0
- class mstar.engine.resources.kv.rolling.manager.RollingRow(rid, write_start, write_stop, kept)[source]#
Bases:
NamedTuple
- mstar.engine.resources.kv.rolling.manager.row_layout(pos, span)[source]#
A step’s write range and kept length in K/V that hold
pos.writtentokens of the stream compacted (seeconfig), and the shift that makes room for it. Request 0; the caller sets the request.- Parameters:
pos (RollingPosition)
span (int)
- Return type:
tuple[RollingRow, Compaction | None]