mstar.model.rebuild#
How a node recomputes a dropped request’s KV, as the model declares it.
Classes
|
A request that prefilled in |
- class mstar.model.rebuild.RebuildPolicy(walks, ids_input, sample_output, loop, copy_width=<factory>, pad_ids=frozenset({}))[source]#
Bases:
objectA request that prefilled in
walksis rebuilt by one row per prefill step, each run in its step’s walk. The rows recompute the request’s prompt and every id it has sampled since, except the last, which the next decode step reads.ids_inputis the input a row’s ids go under, and the ids sampled since the prompt follow it in the last row. It needn’t carry all the prompt’s ids. Whatever else the node reads in a walk, such as the encoder’s output for a prompt with images, or the text before the audio in a Higgs Audio prompt, the worker copies to the host when the walk prefills a request, and the rebuild row reads it again.copy_widthgives, for each such walk, how many values its copies hold per KV slot, each in the dtype the node runs in.sample_outputis the output that carries the one id a row samples, andloopthe loop each of whose iterations samples one.pad_idsare the ids a step replaces with the encoders’ outputs, one row each in order. A request whose last prefill step read more than its ids and that samples one is never dropped after, since the rebuild row of that step would hold a pad its host copies have no row for, and every row after it in the step would read the wrong encoder rows. A step of ids alone runs with no encoder output, so a pad in its rebuild row is embedded as the decode step embedded it.- Parameters:
- check(node, graphs)[source]#
What
nodereads in each ofwalksbesidesids_input, which the worker copies. Raise unless every walk runsnode, some walk readsids_input,copy_widthnames each walk wherenodereads more thanids_input, and some walk runsnodeinsideloop. Also raise whennodereadsids_inputin a walk without outputtingsample_outputthere, since such a walk can end a prompt.
- ids_walks(node, graphs)[source]#
The walks of
walksin whichnodereadsids_input: those an ids span can be written in.