mstar.engine.resources.kv.rolling.manager#

A pool of rolling sink + window K/V, one contiguous region per request.

The pool (pool_tokens a layer, keys and values) is reserved at startup. A request declares the tokens it keeps, its sink plus window, when it opens (RollingKVReqConfig), and waits in the engine’s admission queue, in arrival order, until the pool has room for it (see Resource.ask). The room is set aside then; the request’s first step takes a region of exactly that size, first fit, and gives it back when the request is removed. When no free gap is large enough but the free tokens are, the resident regions slide left to join the gaps.

Functions

row_layout(pos, span)

A step's write range and kept length in K/V that hold pos.written tokens of the stream compacted (see config), and the shift that makes room for it.

Classes

Compaction(src, dst, length)

Move length tokens from src to dst in a request's K/V (src > dst).

RollingKVManager(config, device[, kv_dtype, ...])

RollingPlan(rows)

RollingRow(rid, write_start, write_stop, kept)

class mstar.engine.resources.kv.rolling.manager.Compaction(src, dst, length)[source]#

Bases: NamedTuple

Move length tokens from src to dst in a request’s K/V (src > dst).

Parameters:
dst: int#

Alias for field number 1

length: int#

Alias for field number 2

src: int#

Alias for field number 0

class mstar.engine.resources.kv.rolling.manager.RollingKVManager(config, device, kv_dtype=None, name='rolling_kv')[source]#

Bases: Resource

Parameters:
admit(step, ctx)[source]#

Reserve space for the given step. In the case where requests in a batch must be executed sequentially, this may be called for all requests in a loop before the per-request plan -> forward -> commit cycle.

Parameters:
Return type:

AdmitOutcome

ask(rid, node_name, graph_walk)[source]#

ASKS while rid waits for this resource’s capacity to start on node_name, None when it needs none, or the outcome that fails it when it can never fit.

Parameters:
  • node_name (str)

  • graph_walk (str)

Return type:

AdmitOutcome | None

classmethod build(spec, info)[source]#
Parameters:
Return type:

RollingKVManager

can_capture()[source]#
Return type:

bool

cleanup()[source]#
commit(step, ctx)[source]#

record step consumption

Parameters:
Return type:

None

property current: RollingPlan#
grew_since(mark)[source]#

Whether a request that didn’t fit against mark may fit now.

Parameters:

mark (int)

Return type:

bool

ingest_request(rid, overrides=None)[source]#

Open the request’s state. The worker calls this once per request, on its first NewRequest, though one comes per partition it serves.

Parameters:

overrides (ResourceReqConfig | None)

num_free_tokens()[source]#
Return type:

int

plan(step, ctx)[source]#

Each row’s write range, and the shift that makes room for it.

The shift runs here, on the stream the step then runs on (the store has no pre-plan), so it is ordered after the previous step’s reads of the request’s K/V and before this step’s writes.

Parameters:
Return type:

RollingPlan

post_warmup_validate()[source]#

For, e.g., the KV cache to check that num_free_pages is identical across TP ranks after cuda graph capture.

Raises an error (fails loudly) if invalid.

remove_request(rid)[source]#
reserve(rid, node_name, graph_walk, published)[source]#

Take what rid asked for and return None, or return a mark of what it didn’t fit against for grew_since.

Parameters:
  • node_name (str)

  • graph_walk (str)

  • published (None)

Return type:

int | None

rollback_admit(step, ctx)[source]#

Free the regions this step’s admit placed, setting their room aside again for the retry; a region a request already held stays with it.

Parameters:
Return type:

None

unreserve(rid)[source]#

Give back what reserve took, because another resource rid asks didn’t fit.

Return type:

None

window(layer_idx, row=0)[source]#

Row row’s keys and values for this step, [1, kept, H, D] views in stream order: the sink, the retained window, then the step’s own.

Parameters:
Return type:

tuple[Tensor, Tensor]

write(layer_idx, k, v)[source]#

This step’s k, v [rows, span, H, D] into each row’s K/V. A step may write several times (a denoise loop rewriting its chunk); the last write is what later steps read.

Parameters:
Return type:

None

class mstar.engine.resources.kv.rolling.manager.RollingPlan(rows)[source]#

Bases: NamedTuple

Parameters:

rows (tuple[RollingRow, ...])

rows: tuple[RollingRow, ...]#

Alias for field number 0

class mstar.engine.resources.kv.rolling.manager.RollingRow(rid, write_start, write_stop, kept)[source]#

Bases: NamedTuple

Parameters:
kept: int#

Alias for field number 3

rid: int#

Alias for field number 0

write_start: int#

Alias for field number 1

write_stop: int#

Alias for field number 2

mstar.engine.resources.kv.rolling.manager.row_layout(pos, span)[source]#

A step’s write range and kept length in K/V that hold pos.written tokens of the stream compacted (see config), and the shift that makes room for it. Request 0; the caller sets the request.

Parameters:
Return type:

tuple[RollingRow, Compaction | None]