mstar.worker.rebuild#

What a worker keeps so a request dropped from a node’s KV can be rebuilt there, as the node’s RebuildPolicy declares, and how far each rebuild has run.

Functions

node_writes(labels, nodes, my_walks)

What each of a paged KV cache's nodes that runs here writes of the cache's labels.

policies_run_here(policies, graphs, ...)

The policies whose rebuild this worker runs, each with the walks this worker runs its node in: it runs the node in every walk its loop grows the KV in, and in one of the policy's walks.

Classes

Ingested(rebuildable, unheld)

What Rebuilds.ingest_request decided for a request.

KeptIds(rows[, records, sampled, used, laid_out])

A request's log on one node: the rows of its layout's prefill steps, a record of each step that has run, in order, and each id sampled once the last has.

LayoutRow(walk, length)

One prefill step of a request on a node, as its layout says: the walk it runs and the slots it writes.

NodeWrites(walks, loops)

The walks a node runs on this worker, and the loops that grow its paged KV cache's labels in them.

PlannedRow(walk, total, length, record)

One row of a rebuild: the walk it runs, the slots it writes (total) of an input length long, and the record it reads.

Prefilling(walk, done, total)

A prompt mid-prefill on a node: the walk it prefills in, the tokens its chunks have run, and its length.

QueuedCopy(src, into, after, record)

A device tensor's bytes still to be copied into into once after completes, for record.

RebuildCursor(rows, sampled[, row, done, ran])

A rebuild under way: its rows, the one running (row), the end of that row's chunks that have landed, 0 before the first, and the slots they ran (ran).

Rebuilds(policies, graphs[, pool, ...])

The log of each request on the nodes that declare a RebuildPolicy, checked against the graph when the worker starts, and the rebuilds under way.

RowRecord(inputs, step_metadata[, queued, ...])

One prefill step of a request as it ran, kept so its rebuild can run the step again: host copies of every input the node read in it, as the step read it, in the request's slice, and the step's metadata.

RowRun(row, ran)

A rebuild row that has run, and the slots its chunks ran, which leave out a cached prefix.

class mstar.worker.rebuild.Ingested(rebuildable, unheld)[source]#

Bases: NamedTuple

What Rebuilds.ingest_request decided for a request.

Parameters:
rebuildable: dict[str, bool]#

Alias for field number 0

unheld: frozenset[str]#

Alias for field number 1

class mstar.worker.rebuild.KeptIds(rows, records=<factory>, sampled=<factory>, used=0, laid_out=0)[source]#

Bases: object

A request’s log on one node: the rows of its layout’s prefill steps, a record of each step that has run, in order, and each id sampled once the last has. used counts the bytes of its slice the records take, and laid_out the samples its KV’s layout already holds, which its last rebuild wrote.

Parameters:
laid_out: int#
records: list[RowRecord]#
rows: tuple[LayoutRow, ...]#
sampled: array#
used: int#
class mstar.worker.rebuild.LayoutRow(walk, length)[source]#

Bases: NamedTuple

One prefill step of a request on a node, as its layout says: the walk it runs and the slots it writes.

Parameters:
length: int#

Alias for field number 1

walk: str#

Alias for field number 0

class mstar.worker.rebuild.NodeWrites(walks, loops)[source]#

Bases: NamedTuple

The walks a node runs on this worker, and the loops that grow its paged KV cache’s labels in them.

Parameters:
loops: frozenset[str]#

Alias for field number 1

walks: frozenset[str]#

Alias for field number 0

writes(layout)[source]#
Parameters:

layout (StreamLayout)

Return type:

bool

class mstar.worker.rebuild.PlannedRow(walk, total, length, record)[source]#

Bases: NamedTuple

One row of a rebuild: the walk it runs, the slots it writes (total) of an input length long, and the record it reads. A prompt’s step in progress has no record: it reads the input the engine keeps for the prompt, up to where the prompt stood.

Parameters:
length: int#

Alias for field number 2

record: RowRecord | None#

Alias for field number 3

total: int#

Alias for field number 1

walk: str#

Alias for field number 0

class mstar.worker.rebuild.Prefilling(walk, done, total)[source]#

Bases: NamedTuple

A prompt mid-prefill on a node: the walk it prefills in, the tokens its chunks have run, and its length.

Parameters:
done: int#

Alias for field number 1

total: int#

Alias for field number 2

walk: str#

Alias for field number 0

class mstar.worker.rebuild.QueuedCopy(src, into, after, record)[source]#

Bases: NamedTuple

A device tensor’s bytes still to be copied into into once after completes, for record.

Parameters:
after: Event | None#

Alias for field number 2

into: HostSlice#

Alias for field number 1

record: RowRecord#

Alias for field number 3

src: Tensor#

Alias for field number 0

class mstar.worker.rebuild.RebuildCursor(rows, sampled, row=0, done=0, ran=0)[source]#

Bases: object

A rebuild under way: its rows, the one running (row), the end of that row’s chunks that have landed, 0 before the first, and the slots they ran (ran). The last row read from a record also writes every id sampled since but the newest, which the next decode step reads.

Parameters:
done: int#
property length: int#
ran: int#
property resumes_prefill: bool#

Whether the row running is its prompt’s step in progress, which goes on past the rebuild.

row: int#
rows: tuple[PlannedRow, ...]#
sampled: array#
property total: int#
property walk: str#
class mstar.worker.rebuild.Rebuilds(policies, graphs, pool=None, allowance=None, node_dtypes=MappingProxyType({}), followed=frozenset())[source]#

Bases: object

The log of each request on the nodes that declare a RebuildPolicy, checked against the graph when the worker starts, and the rebuilds under way. Host copies of what a step read besides ids spend the step’s allowance (flush).

Parameters:
can_rebuild(node, layouts)[source]#

Whether a rebuild on node can write a request’s laid-out labels again: the node declares a policy, and its loop grows labels its rebuild rows can lay out again, written in the policy’s walks.

Parameters:
Return type:

bool

property copies_queued: bool#
cursor(rid, node)[source]#
Parameters:
Return type:

RebuildCursor

cursors()[source]#
Return type:

ItemsView[tuple[str, int], RebuildCursor]

fill(node, rids, chunk_ranges, inputs, infos)[source]#

Give each rebuild row of rids its row’s walk and its inputs, and return the events the step waits on before it reads them.

A prompt’s step in progress runs every chunk from the input the engine keeps for the prompt, under the prompt’s own step metadata. Any other row runs under its step’s recorded metadata and reads its record, on the device when it runs its first chunk or runs whole, the last row’s ids with every id sampled since but the newest appended. A later chunk is cut from the input the engine prepared on the first.

Parameters:
Return type:

tuple[Event, …]

flush()[source]#

Start the queued copies to the host, oldest first, up to what the step’s allowance has left, once the step’s outputs are sent or on a pass of the worker’s loop that runs no step (flush_idle). A copy that runs past the allowance is split there, and the rest waits for the next allowance. The device tensor a queued copy reads stays held until its copy starts.

Return type:

None

flush_idle()[source]#

flush on a pass of the worker’s loop that runs no step, with a step’s allowance once the last copy it started has landed. A drop waits for the queued copies, so with every request waiting, no step would ever start them. Waiting for the last copy keeps a burst of such passes from queuing more than one allowance.

Return type:

None

forget(rid, node)[source]#

Give back rid’s log on node, so rid is never dropped there.

Parameters:
Return type:

None

ingest_request(rid, writes, layouts, unpausable)[source]#

For each paged KV cache in writes, whether a drop here can free rid’s pages there: rebuild rows can lay out the request’s layouts in that cache again, and every node that writes those layouts can redo all of its writes. A node that writes nothing for the request isn’t asked. Also the nodes whose log the pool couldn’t hold, which rank 0 journals for its followers to refuse.

A drop takes every cache of its node, so a node is judged by what it writes in all of them. A node keeps a log of rid only when a drop can free rid’s pages in every cache the node writes, and unpausable is False. A node takes the pool’s room for the log now, so a request the pool can’t hold is known before it is admitted and is never dropped on that node. The log’s rows are the layout’s prefill steps (StreamLayout.steps).

Parameters:
Return type:

Ingested

keep_samples(node_batch, cpu_outputs, host_rows, skip_rids=())[source]#

Append each kept row’s sample, from the stop check’s host copy, to its log, once its log holds every prefill step, or forget the log when the sample is one of the policy’s pad_ids and the last step, whose rebuild row carries the samples, read more than the ids. Returns the requests whose logs it forgot. skip_rids are rows whose sample is not real, as for Engine.step_finished.

Parameters:
Return type:

list[int]

kept(rid, node)[source]#
Parameters:
Return type:

KeptIds | None

land_chunk(rid, node, chunk)[source]#

Record that rid’s rebuild ran chunk of its row, or the rest of the row with no chunk, and return the row once it has run. The rebuild then moves to its next row, and ends after its last.

Parameters:
Return type:

RowRun | None

property nodes: frozenset[str]#

The nodes whose requests can be rebuilt here.

property pool: HostPool | None#
rebuilding(rid, node)[source]#
Parameters:
Return type:

bool

record(node_batch, cached_infos=MappingProxyType({}))[source]#

Record each row of the step that runs the next prefill step of a request still prefilling here, on the row’s first chunk, which carries the step’s whole input: its ids into the request’s slice once the step completes, its other inputs queued to follow them (flush), and its step metadata. Once the last step is recorded the request keeps its samples, and the slice’s unused tail goes back to the pool.

A row the prefix cache served whole ran no forward, but a rebuild may find its pages evicted, so it is recorded from its inputs and its info in cached_infos, which the engine takes out of the step.

A step other than the one the request’s layout says comes next, by walk or by length, or one missing an input the node reads, as a row the prefix cache served past its encoder, leaves the request with no log, so it is never dropped.

Parameters:
Return type:

None

refusal(rid, node, walk, loop_iters, prefilling)[source]#

Why rid, in walk, can’t be dropped on node now, or None. Its rebuild reruns each prefill step recorded before the one in progress from its record, which has to be on the host. A request prefilling its prompt reruns the step in progress from the input the engine keeps for the prompt. Any other reruns every step its layout lays out, and its samples cover its KV when the prefill and each iteration of the policy’s loop sampled one id, since the KV holds all but the last.

Parameters:
Return type:

str | None

refuse(rid, node)[source]#

Give back rid’s log on node, which rank 0’s pool couldn’t hold, and return the paged KV caches node writes for rid, where a drop then can’t free rid’s pages, as at rank 0’s ingest_request.

Parameters:
Return type:

list[str]

remove_request(rid)[source]#
Parameters:

rid (int)

Return type:

None

start(rid, node, prefilling)[source]#

Open rid’s rebuild on node over the slots tokens counts, which its drop checked, and return what it writes past its layout. A prompt prefilling its step in progress reruns that step’s chunks last, and changes nothing the layout says, so this returns None for it.

Parameters:
Return type:

Rebuild | None

tokens(rid, node, prefilling)[source]#

How many slots rid’s rebuild on node writes, over every row start opens it with.

Parameters:
Return type:

int

class mstar.worker.rebuild.RowRecord(inputs, step_metadata, queued=0, landed=None)[source]#

Bases: object

One prefill step of a request as it ran, kept so its rebuild can run the step again: host copies of every input the node read in it, as the step read it, in the request’s slice, and the step’s metadata. queued counts the copies that haven’t started, and landed is recorded after the last that has, None where the copies run in place.

Parameters:
inputs: dict[str, list[Tensor]]#
landed: Event | None#
queued: int#
ready()[source]#
Return type:

bool

step_metadata: dict#
class mstar.worker.rebuild.RowRun(row, ran)[source]#

Bases: NamedTuple

A rebuild row that has run, and the slots its chunks ran, which leave out a cached prefix.

Parameters:
ran: int#

Alias for field number 1

row: PlannedRow#

Alias for field number 0

mstar.worker.rebuild.node_writes(labels, nodes, my_walks)[source]#

What each of a paged KV cache’s nodes that runs here writes of the cache’s labels.

Parameters:
  • labels (Mapping[str, KVLabel])

  • nodes (Iterable[str])

  • my_walks (Collection[tuple[str, str]])

Return type:

dict[str, NodeWrites]

mstar.worker.rebuild.policies_run_here(policies, graphs, my_walks, parallel_nodes)[source]#

The policies whose rebuild this worker runs, each with the walks this worker runs its node in: it runs the node in every walk its loop grows the KV in, and in one of the policy’s walks. A disaggregated decode node can’t run the prefill walk its rebuild needs.

Above one rank a policy keeps only the walks with nothing to copy, since rank 0 decides a drop from its own host copies, and a follower’s may not have landed.

Parameters:
Return type:

dict[str, RebuildPolicy]