mstar.worker.scheduled_batch#

A step’s rows as the scheduler hands them on, and the chunk arithmetic that goes with them. Below the scheduler and Lockstep, which both build batches.

Functions

cap_chunk_loops(batch, runtime, ...)

Cap the chunk Loop of each row of batch on its first chunk at its whole input's length, which is what ends the Loop.

chunk_loop_stride(chunk)

How far a row of a chunk-Loop walk moves its Loop: its chunk's stride, or to the end for a row that runs whole.

runs_final_chunk(chunk)

Whether a row with this range ends its prompt.

tp_chunk_ranges(message, rids)

The chunks the leader cut for message's rows, keyed by this rank's handles for them (rids, from tp_rids).

unfinished_chunk_rids(chunk_ranges)

The rows of chunk_ranges whose prompt needs another chunk.

Classes

ContinuingRows(loop_back, mid_prefill)

The rows of an in-flight step that run again in the step speculated after it, split by where their next step's inputs come from.

MidPrefillRow(request_id, worker_graph_id, ...)

A row mid-prefill that nothing popped: a row of the caller's in-flight step whose node needs another chunk, or a rebuild the log holds.

PopReadyResult(wg_ids, edge_specs, ...)

ScheduledBatch(node_name, graph_walk[, ...])

A batch of nodes ready to be executed.

class mstar.worker.scheduled_batch.ContinuingRows(loop_back, mid_prefill)[source]#

Bases: NamedTuple

The rows of an in-flight step that run again in the step speculated after it, split by where their next step’s inputs come from.

Parameters:
loop_back: list[int]#

Alias for field number 0

mid_prefill: list[MidPrefillRow]#

Alias for field number 1

class mstar.worker.scheduled_batch.MidPrefillRow(request_id, worker_graph_id, graph_walk, start, total)[source]#

Bases: NamedTuple

A row mid-prefill that nothing popped: a row of the caller’s in-flight step whose node needs another chunk, or a rebuild the log holds. start is where its next chunk starts, 0 for a rebuild that has run none, and total its prompt’s length.

Parameters:
  • request_id (int)

  • worker_graph_id (int)

  • graph_walk (str)

  • start (int)

  • total (int)

graph_walk: str#

Alias for field number 2

request_id: int#

Alias for field number 0

start: int#

Alias for field number 3

total: int#

Alias for field number 4

worker_graph_id: int#

Alias for field number 1

class mstar.worker.scheduled_batch.PopReadyResult(wg_ids, edge_specs, output_signals, request_walks, walk_output_signals)[source]#

Bases: NamedTuple

Parameters:
edge_specs: ColumnarEdgeSpecs#

Alias for field number 1

output_signals: tuple[str, ...]#

Alias for field number 2

request_walks: dict[int, str]#

Alias for field number 3

walk_output_signals: dict[str, list[str]]#

Alias for field number 4

wg_ids: dict[int, int]#

Alias for field number 0

class mstar.worker.scheduled_batch.ScheduledBatch(node_name, graph_walk, request_to_worker_graph=<factory>, input_edges=<factory>, output_signals=<factory>, tp_seq=-1, request_walks=<factory>, walk_output_signals=<factory>, row_lens=<factory>, read_check_rids=None, turn_order=None, _by_walk=None, chunk_ranges=<factory>, incomplete_node_rids=<factory>, rebuild_rids=frozenset({}), chunk_loop_walks=frozenset({}))[source]#

Bases: object

A batch of nodes ready to be executed.

The term “ScheduledBatch” is a slight misnomer: this might exceed the batch size cap, so it may end up getting split into multiple batches at scheduling time.

Parameters:
chunk_loop_walks: frozenset[str] = frozenset({})#
chunk_ranges: dict[int, ChunkRange]#
discard_rid(rid)[source]#

Drop rid’s row. Every removal goes through here, so the walk grouping never goes stale; input_edges is the caller’s.

Parameters:

rid (int)

Return type:

None

get_continuing_rows(node_name, walk_rids, tearing_down, speculated)[source]#

This step’s rows that continue into a speculated step on node_name: walk_rids (the target walk’s rows) that loop back, and the rows mid-prefill whose (node, walk) speculates its next chunk (speculate_mid_prefill). tearing_down: rows to leave out, or None when none can be.

Parameters:
Return type:

ContinuingRows

graph_rows()[source]#

request_to_worker_graph without the rebuild rows: the rows the graph runtime holds.

Return type:

dict[int, int]

graph_walk: str#
incomplete_node_rids: set[int]#
input_edges: ColumnarEdgeSpecs#
loop_strides(rids)[source]#

RouteInput.loop_strides for rids, in that order: a row of a chunk-Loop walk moves its Loop by its chunk (chunk_loop_stride), and any other row its loop by one. None for a step with no row in a chunk Loop, which is every decode step.

Keyed on the walk having a Loop rather than on the row having a chunk range: a row with no range ran whole, which also ends the Loop – a walk with no token budget is never measured or cut.

Parameters:

rids (Iterable[int])

Return type:

list[int] | None

merge(other)[source]#

Fold other’s requests in, ours first — they have waited longer.

Parameters:

other (ScheduledBatch)

Return type:

None

node_name: str#
output_signals: list[str]#
read_check_rids: tuple[int, ...] | None = None#
rebuild_rids: frozenset[int] = frozenset({})#
relabel_if_uniform()[source]#

Label a combined batch whose rows share one real walk with that walk.

Return type:

None

request_to_worker_graph: dict[int, int]#
request_walks: dict[int, str]#
rids_by_walk()[source]#

The batch’s rids grouped by real walk, in batch order. Shared: callers must not modify it.

Return type:

dict[str, list[int]]

row_lens: dict[int, Any]#
split_off_first(bs, exclude_rids=None)[source]#

Return the first ScheduledBatch, as well as the remainder (None if all requests have been scheduled)

Parameters:
  • bs (int | None)

  • exclude_rids (set[int] | None)

Return type:

tuple[ScheduledBatch | None, ScheduledBatch | None]

split_off_rids(rids)[source]#

rids (all of them in this batch), in that order, and the remainder in batch order (None if nothing is left).

Parameters:

rids (Sequence[int])

Return type:

tuple[ScheduledBatch, ScheduledBatch | None]

tp_seq: int = -1#
turn_order: list[int] | None = None#
walk_of(rid)[source]#
Parameters:

rid (int)

Return type:

str

walk_output_signals: dict[str, list[str]]#
wg_ids()[source]#

set_in_flight’s worker graphs: one id for a step of one walk (a node in a walk lives in one worker graph), else one per rid.

Return type:

list[int]

mstar.worker.scheduled_batch.cap_chunk_loops(batch, runtime, loop_walks_by_node)[source]#

Cap the chunk Loop of each row of batch on its first chunk at its whole input’s length, which is what ends the Loop. A TP follower caps from the leader’s chunks: the cap decides on which step the Loop ends, so every rank of the instance has to set the same one.

Parameters:
Return type:

None

mstar.worker.scheduled_batch.chunk_loop_stride(chunk)[source]#

How far a row of a chunk-Loop walk moves its Loop: its chunk’s stride, or to the end for a row that runs whole.

Parameters:

chunk (ChunkRange | None)

Return type:

int

mstar.worker.scheduled_batch.runs_final_chunk(chunk)[source]#

Whether a row with this range ends its prompt. No range means the row runs whole, which is also its last chunk.

Parameters:

chunk (ChunkRange | None)

Return type:

bool

mstar.worker.scheduled_batch.tp_chunk_ranges(message, rids)[source]#

The chunks the leader cut for message’s rows, keyed by this rank’s handles for them (rids, from tp_rids). Lists that do not match the rows mean the ranks disagree on the step, so they raise.

Parameters:
Return type:

dict[int, ChunkRange]

mstar.worker.scheduled_batch.unfinished_chunk_rids(chunk_ranges)[source]#

The rows of chunk_ranges whose prompt needs another chunk.

Parameters:

chunk_ranges (Mapping[int, ChunkRange])

Return type:

set[int]