mstar.worker.scheduled_batch#
A step’s rows as the scheduler hands them on, and the chunk arithmetic that
goes with them. Below the scheduler and Lockstep, which both build batches.
Functions
|
Cap the chunk Loop of each row of |
|
How far a row of a chunk-Loop walk moves its Loop: its chunk's stride, or to the end for a row that runs whole. |
|
Whether a row with this range ends its prompt. |
|
The chunks the leader cut for |
|
The rows of |
Classes
|
The rows of an in-flight step that run again in the step speculated after it, split by where their next step's inputs come from. |
|
A row mid-prefill that nothing popped: a row of the caller's in-flight step whose node needs another chunk, or a rebuild the log holds. |
|
|
|
A batch of nodes ready to be executed. |
- class mstar.worker.scheduled_batch.ContinuingRows(loop_back, mid_prefill)[source]#
Bases:
NamedTupleThe rows of an in-flight step that run again in the step speculated after it, split by where their next step’s inputs come from.
- Parameters:
mid_prefill (list[MidPrefillRow])
- mid_prefill: list[MidPrefillRow]#
Alias for field number 1
- class mstar.worker.scheduled_batch.MidPrefillRow(request_id, worker_graph_id, graph_walk, start, total)[source]#
Bases:
NamedTupleA row mid-prefill that nothing popped: a row of the caller’s in-flight step whose node needs another chunk, or a rebuild the log holds.
startis where its next chunk starts, 0 for a rebuild that has run none, andtotalits prompt’s length.
- class mstar.worker.scheduled_batch.PopReadyResult(wg_ids, edge_specs, output_signals, request_walks, walk_output_signals)[source]#
Bases:
NamedTuple- Parameters:
- edge_specs: ColumnarEdgeSpecs#
Alias for field number 1
- class mstar.worker.scheduled_batch.ScheduledBatch(node_name, graph_walk, request_to_worker_graph=<factory>, input_edges=<factory>, output_signals=<factory>, tp_seq=-1, request_walks=<factory>, walk_output_signals=<factory>, row_lens=<factory>, read_check_rids=None, turn_order=None, _by_walk=None, chunk_ranges=<factory>, incomplete_node_rids=<factory>, rebuild_rids=frozenset({}), chunk_loop_walks=frozenset({}))[source]#
Bases:
objectA batch of nodes ready to be executed.
The term “ScheduledBatch” is a slight misnomer: this might exceed the batch size cap, so it may end up getting split into multiple batches at scheduling time.
- Parameters:
- chunk_ranges: dict[int, ChunkRange]#
- discard_rid(rid)[source]#
Drop
rid’s row. Every removal goes through here, so the walk grouping never goes stale;input_edgesis the caller’s.- Parameters:
rid (int)
- Return type:
None
- get_continuing_rows(node_name, walk_rids, tearing_down, speculated)[source]#
This step’s rows that continue into a speculated step on
node_name:walk_rids(the target walk’s rows) that loop back, and the rows mid-prefill whose (node, walk) speculates its next chunk (speculate_mid_prefill).tearing_down: rows to leave out, or None when none can be.
- graph_rows()[source]#
request_to_worker_graphwithout the rebuild rows: the rows the graph runtime holds.
- input_edges: ColumnarEdgeSpecs#
- loop_strides(rids)[source]#
RouteInput.loop_stridesforrids, in that order: a row of a chunk-Loop walk moves its Loop by its chunk (chunk_loop_stride), and any other row its loop by one. None for a step with no row in a chunk Loop, which is every decode step.Keyed on the walk having a Loop rather than on the row having a chunk range: a row with no range ran whole, which also ends the Loop – a walk with no token budget is never measured or cut.
- merge(other)[source]#
Fold
other’s requests in, ours first — they have waited longer.- Parameters:
other (ScheduledBatch)
- Return type:
None
- relabel_if_uniform()[source]#
Label a combined batch whose rows share one real walk with that walk.
- Return type:
None
- rids_by_walk()[source]#
The batch’s rids grouped by real walk, in batch order. Shared: callers must not modify it.
- split_off_first(bs, exclude_rids=None)[source]#
Return the first ScheduledBatch, as well as the remainder (None if all requests have been scheduled)
- Parameters:
- Return type:
tuple[ScheduledBatch | None, ScheduledBatch | None]
- split_off_rids(rids)[source]#
rids(all of them in this batch), in that order, and the remainder in batch order (None if nothing is left).- Parameters:
- Return type:
tuple[ScheduledBatch, ScheduledBatch | None]
- mstar.worker.scheduled_batch.cap_chunk_loops(batch, runtime, loop_walks_by_node)[source]#
Cap the chunk Loop of each row of
batchon its first chunk at its whole input’s length, which is what ends the Loop. A TP follower caps from the leader’s chunks: the cap decides on which step the Loop ends, so every rank of the instance has to set the same one.- Parameters:
batch (ScheduledBatch)
runtime (GraphRuntime)
- Return type:
None
- mstar.worker.scheduled_batch.chunk_loop_stride(chunk)[source]#
How far a row of a chunk-Loop walk moves its Loop: its chunk’s stride, or to the end for a row that runs whole.
- Parameters:
chunk (ChunkRange | None)
- Return type:
- mstar.worker.scheduled_batch.runs_final_chunk(chunk)[source]#
Whether a row with this range ends its prompt. No range means the row runs whole, which is also its last chunk.
- Parameters:
chunk (ChunkRange | None)
- Return type:
- mstar.worker.scheduled_batch.tp_chunk_ranges(message, rids)[source]#
The chunks the leader cut for
message’s rows, keyed by this rank’s handles for them (rids, fromtp_rids). Lists that do not match the rows mean the ranks disagree on the step, so they raise.- Parameters:
message (ScheduleTPNode)
- Return type: