mstar.model.rebuild#

How a node recomputes a dropped request’s KV, as the model declares it.

Classes

RebuildPolicy(walks, ids_input, ...[, ...])

A request that prefilled in walks is rebuilt by one row per prefill step, each run in its step's walk.

class mstar.model.rebuild.RebuildPolicy(walks, ids_input, sample_output, loop, copy_width=<factory>, pad_ids=frozenset({}))[source]#

Bases: object

A request that prefilled in walks is rebuilt by one row per prefill step, each run in its step’s walk. The rows recompute the request’s prompt and every id it has sampled since, except the last, which the next decode step reads.

ids_input is the input a row’s ids go under, and the ids sampled since the prompt follow it in the last row. It needn’t carry all the prompt’s ids. Whatever else the node reads in a walk, such as the encoder’s output for a prompt with images, or the text before the audio in a Higgs Audio prompt, the worker copies to the host when the walk prefills a request, and the rebuild row reads it again. copy_width gives, for each such walk, how many values its copies hold per KV slot, each in the dtype the node runs in. sample_output is the output that carries the one id a row samples, and loop the loop each of whose iterations samples one.

pad_ids are the ids a step replaces with the encoders’ outputs, one row each in order. A request whose last prefill step read more than its ids and that samples one is never dropped after, since the rebuild row of that step would hold a pad its host copies have no row for, and every row after it in the step would read the wrong encoder rows. A step of ids alone runs with no encoder output, so a pad in its rebuild row is embedded as the decode step embedded it.

Parameters:
check(node, graphs)[source]#

What node reads in each of walks besides ids_input, which the worker copies. Raise unless every walk runs node, some walk reads ids_input, copy_width names each walk where node reads more than ids_input, and some walk runs node inside loop. Also raise when node reads ids_input in a walk without outputting sample_output there, since such a walk can end a prompt.

Parameters:
Return type:

dict[str, frozenset[str]]

copy_width: Mapping[str, int]#
ids_input: str#
ids_walks(node, graphs)[source]#

The walks of walks in which node reads ids_input: those an ids span can be written in.

Parameters:
Return type:

frozenset[str]

loop: str#
loop_walks(node, graphs)[source]#

The walks in which loop runs node: those a request can be dropped in.

Parameters:
Return type:

frozenset[str]

pad_ids: frozenset[int] = frozenset({})#
sample_output: str#
walks: frozenset[str]#