mstar.engine.chunking#
Chunked prefill’s state on the engine.
Per (node, walk): the model’s chunking policy and the token budget a step runs under, after the serving config’s overrides. Per row mid-prefill: its whole prepared input, which every chunk is cut from. And the count of steps that ran over their budget, for the warning.
Classes
|
The tokens |
|
- class mstar.engine.chunking.ChunkRange(start, end, total, stride)[source]#
Bases:
NamedTupleThe tokens
[start, end)a chunked row runs this step, of thetotalits whole input was measured at.strideis how far the chunk moves the row’s chunk Loop, which counts the tokens run up tototal(GraphRuntime.set_loop_caps):endfor a prompt’s first chunk, whose Loop starts at 0 even when a cached prefix movesstarton, andend - startfor each later one.
- class mstar.engine.chunking.ChunkingManagement(submodules)[source]#
Bases:
object- check_budget(batch, node_inputs)[source]#
Warn when a step runs over the budget it was scheduled under: that of the combined walk it ran in, if any. A row too long to cut runs whole, and a speculation merge sizes its continuing rows by estimate, before their inputs exist. Logged on the 1st, 2nd, 4th, … overrun per walk.
- Parameters:
batch (ExecutingBatch)
node_inputs (list[NodeInputs])
- Return type:
None
- chunkable(node_name, graph_walk)[source]#
The submodule’s say, unless the serving config turned the walk’s chunking off (a null budget) or the worker chunks nothing on the node.
- cut_budgets()[source]#
(node, walk) -> budget, for each budget that cuts a prompt: a chunkable walk’s own, and that of the combined walk it runs in.
- max_batch_tokens(node_name, graph_walk)[source]#
The walk’s budget, unless the serving config overrides it or the worker chunks nothing on the node.
- max_step_tokens(node_name, graph_walk, max_rows)[source]#
The most tokens the scheduler gives a step of
graph_walkof at mostmax_rowsrows, None where it runs under no budget.That is the budget it was scheduled under: that of the combined walk it runs in, if any (a combined batch whose rows share one real walk runs relabelled to it, ScheduledBatch.relabel_if_uniform, and still ran under the combined walk’s budget). For a combined walk’s step, or more, by what a speculation chain’s rows, one-token decode rows, take of
min_chunk_tokens. A real walk’s step carries no chain rows of its own node: decode rows beside a prompt make the step the combined walk’s (MicroScheduler.label_for), and a same-node chain on a prompt walk continues no row, its rows mid-prefill being the step’s own.None too where the step may hold a prompt of a walk whose chunking the serving config turned off: it runs whole, however long.
- min_chunk_tokens(budget, align=1)[source]#
The fewest tokens the scheduler cuts a row to under
budget, or the whole budget when that is smaller. Without a floor, decode rows that fill the budget leave a prompt a token or two a step. A cut is a multiple ofalign(Engine.chunk_alignment), so the floor rounds up to one, within the budget: a floor between two multiples would leave the oldest prompt’s share of a step too small to cut.
- policy(node_name, graph_walk)[source]#
The model’s, or
NO_CHUNKINGwhere it declares none.Declared on the model rather than the submodule because the graph build needs it too: a chunkable walk runs the node inside a Loop, and
get_graph_walk_graphsis read long before any submodule loads.- Parameters:
- Return type:
- restore(rid, node_name)[source]#
Put a prompt’s whole input back, for its rebuild and then its own chunks to cut from.
- set_aside(rid, node_name)[source]#
Move a prompt’s whole input out of prepared, where its rebuild’s rows prepare their own.
- set_budgets(overrides, walks, combined_walk_of=None, override_mixed=(), no_chunking=(), min_chunk_tokens=DEFAULT_MIN_CHUNK_TOKENS)[source]#
Apply a serving config’s
max_batch_tokensandmin_chunk_tokensto this engine’s nodes.Overrides the budgets
set_policiesdeclared, so it runs after it.Per node, one budget for every walk the model budgets, or a mapping of walk -> budget; null turns chunking off. Only walks the model budgets can be overridden: elsewhere the setting would do nothing.
A walk inside a combined walk runs in the combined walk’s steps, under the combined walk’s budget, so its own budget applies there only on a node in
override_mixed(can_override_mixed_max_tokens), where it caps that walk’s rows within the step. Elsewhere a per-walk override of such a walk would do nothing, so it is refused; the combined walk itself can be named instead.no_chunkingnames nodes this worker chunks nowhere (SP, which chunked prefill does not support yet). A budget set on one is still validated but has no effect, since the deployment, not the config, is what turned chunking off there.
- set_policies(policies)[source]#
The model’s
chunking_policy_of(), bound at load.Separate from
set_budgets: the policy is the model’s and fixed, the budget is the deployment’s and layered on top.- Parameters:
policies (Mapping[tuple[str, str], ChunkingPolicy])
- Return type:
None