mstar.worker.rebuild#
What a worker keeps so a request dropped from a node’s KV can be rebuilt there, as the node’s RebuildPolicy declares, and how far each rebuild has run.
Functions
|
What each of a paged KV cache's |
|
The policies whose rebuild this worker runs, each with the walks this worker runs its node in: it runs the node in every walk its loop grows the KV in, and in one of the policy's walks. |
Classes
|
What Rebuilds.ingest_request decided for a request. |
|
A request's log on one node: the rows of its layout's prefill steps, a record of each step that has run, in order, and each id sampled once the last has. |
|
One prefill step of a request on a node, as its layout says: the walk it runs and the slots it writes. |
|
The walks a node runs on this worker, and the loops that grow its paged KV cache's labels in them. |
|
One row of a rebuild: the walk it runs, the slots it writes ( |
|
A prompt mid-prefill on a node: the walk it prefills in, the tokens its chunks have run, and its length. |
|
A device tensor's bytes still to be copied into |
|
A rebuild under way: its rows, the one running ( |
|
The log of each request on the nodes that declare a RebuildPolicy, checked against the graph when the worker starts, and the rebuilds under way. |
|
One prefill step of a request as it ran, kept so its rebuild can run the step again: host copies of every input the node read in it, as the step read it, in the request's slice, and the step's metadata. |
|
A rebuild row that has run, and the slots its chunks ran, which leave out a cached prefix. |
- class mstar.worker.rebuild.Ingested(rebuildable, unheld)[source]#
Bases:
NamedTupleWhat Rebuilds.ingest_request decided for a request.
- class mstar.worker.rebuild.KeptIds(rows, records=<factory>, sampled=<factory>, used=0, laid_out=0)[source]#
Bases:
objectA request’s log on one node: the rows of its layout’s prefill steps, a record of each step that has run, in order, and each id sampled once the last has.
usedcounts the bytes of its slice the records take, andlaid_outthe samples its KV’s layout already holds, which its last rebuild wrote.- Parameters:
- class mstar.worker.rebuild.LayoutRow(walk, length)[source]#
Bases:
NamedTupleOne prefill step of a request on a node, as its layout says: the walk it runs and the slots it writes.
- class mstar.worker.rebuild.NodeWrites(walks, loops)[source]#
Bases:
NamedTupleThe walks a node runs on this worker, and the loops that grow its paged KV cache’s labels in them.
- writes(layout)[source]#
- Parameters:
layout (StreamLayout)
- Return type:
- class mstar.worker.rebuild.PlannedRow(walk, total, length, record)[source]#
Bases:
NamedTupleOne row of a rebuild: the walk it runs, the slots it writes (
total) of an inputlengthlong, and the record it reads. A prompt’s step in progress has no record: it reads the input the engine keeps for the prompt, up to where the prompt stood.
- class mstar.worker.rebuild.Prefilling(walk, done, total)[source]#
Bases:
NamedTupleA prompt mid-prefill on a node: the walk it prefills in, the tokens its chunks have run, and its length.
- class mstar.worker.rebuild.QueuedCopy(src, into, after, record)[source]#
Bases:
NamedTupleA device tensor’s bytes still to be copied into
intoonceaftercompletes, forrecord.
- class mstar.worker.rebuild.RebuildCursor(rows, sampled, row=0, done=0, ran=0)[source]#
Bases:
objectA rebuild under way: its rows, the one running (
row), the end of that row’s chunks that have landed, 0 before the first, and the slots they ran (ran). The last row read from a record also writes every idsampledsince but the newest, which the next decode step reads.- property resumes_prefill: bool#
Whether the row running is its prompt’s step in progress, which goes on past the rebuild.
- rows: tuple[PlannedRow, ...]#
- class mstar.worker.rebuild.Rebuilds(policies, graphs, pool=None, allowance=None, node_dtypes=MappingProxyType({}), followed=frozenset())[source]#
Bases:
objectThe log of each request on the nodes that declare a RebuildPolicy, checked against the graph when the worker starts, and the rebuilds under way. Host copies of what a step read besides ids spend the step’s
allowance(flush).- Parameters:
policies (Mapping[str, RebuildPolicy])
graphs (Mapping[str, GraphSection])
pool (HostPool | None)
allowance (StepAllowance | None)
node_dtypes (Mapping[str, torch.dtype])
followed (Collection[str])
- can_rebuild(node, layouts)[source]#
Whether a rebuild on
nodecan write a request’s laid-out labels again: the node declares a policy, and its loop grows labels its rebuild rows can lay out again, written in the policy’s walks.- Parameters:
node (str)
layouts (Iterable[StreamLayout])
- Return type:
- fill(node, rids, chunk_ranges, inputs, infos)[source]#
Give each rebuild row of
ridsits row’s walk and its inputs, and return the events the step waits on before it reads them.A prompt’s step in progress runs every chunk from the input the engine keeps for the prompt, under the prompt’s own step metadata. Any other row runs under its step’s recorded metadata and reads its record, on the device when it runs its first chunk or runs whole, the last row’s ids with every id sampled since but the newest appended. A later chunk is cut from the input the engine prepared on the first.
- Parameters:
node (str)
rids (Collection[int])
chunk_ranges (Mapping[int, ChunkRange])
infos (MutableMapping[int, CurrentForwardPassInfo])
- Return type:
- flush()[source]#
Start the queued copies to the host, oldest first, up to what the step’s allowance has left, once the step’s outputs are sent or on a pass of the worker’s loop that runs no step (flush_idle). A copy that runs past the allowance is split there, and the rest waits for the next allowance. The device tensor a queued copy reads stays held until its copy starts.
- Return type:
None
- flush_idle()[source]#
flush on a pass of the worker’s loop that runs no step, with a step’s allowance once the last copy it started has landed. A drop waits for the queued copies, so with every request waiting, no step would ever start them. Waiting for the last copy keeps a burst of such passes from queuing more than one allowance.
- Return type:
None
- ingest_request(rid, writes, layouts, unpausable)[source]#
For each paged KV cache in
writes, whether a drop here can freerid’s pages there: rebuild rows can lay out the request’slayoutsin that cache again, and every node that writes those layouts can redo all of its writes. A node that writes nothing for the request isn’t asked. Also the nodes whose log the pool couldn’t hold, which rank 0 journals for its followers to refuse.A drop takes every cache of its node, so a node is judged by what it writes in all of them. A node keeps a log of
ridonly when a drop can freerid’s pages in every cache the node writes, andunpausableis False. A node takes the pool’s room for the log now, so a request the pool can’t hold is known before it is admitted and is never dropped on that node. The log’s rows are the layout’s prefill steps (StreamLayout.steps).- Parameters:
rid (int)
writes (Mapping[str, Mapping[str, NodeWrites]])
layouts (Mapping[str, Collection[StreamLayout]])
unpausable (bool)
- Return type:
- keep_samples(node_batch, cpu_outputs, host_rows, skip_rids=())[source]#
Append each kept row’s sample, from the stop check’s host copy, to its log, once its log holds every prefill step, or forget the log when the sample is one of the policy’s
pad_idsand the last step, whose rebuild row carries the samples, read more than the ids. Returns the requests whose logs it forgot.skip_ridsare rows whose sample is not real, as for Engine.step_finished.
- land_chunk(rid, node, chunk)[source]#
Record that
rid’s rebuild ranchunkof its row, or the rest of the row with no chunk, and return the row once it has run. The rebuild then moves to its next row, and ends after its last.- Parameters:
rid (int)
node (str)
chunk (ChunkRange | None)
- Return type:
RowRun | None
- record(node_batch, cached_infos=MappingProxyType({}))[source]#
Record each row of the step that runs the next prefill step of a request still prefilling here, on the row’s first chunk, which carries the step’s whole input: its ids into the request’s slice once the step completes, its other inputs queued to follow them (flush), and its step metadata. Once the last step is recorded the request keeps its samples, and the slice’s unused tail goes back to the pool.
A row the prefix cache served whole ran no forward, but a rebuild may find its pages evicted, so it is recorded from its inputs and its info in
cached_infos, which the engine takes out of the step.A step other than the one the request’s layout says comes next, by walk or by length, or one missing an input the node reads, as a row the prefix cache served past its encoder, leaves the request with no log, so it is never dropped.
- Parameters:
node_batch (ExecutingBatch)
cached_infos (Mapping[int, CurrentForwardPassInfo])
- Return type:
None
- refusal(rid, node, walk, loop_iters, prefilling)[source]#
Why
rid, inwalk, can’t be dropped onnodenow, or None. Its rebuild reruns each prefill step recorded before the one in progress from its record, which has to be on the host. A requestprefillingits prompt reruns the step in progress from the input the engine keeps for the prompt. Any other reruns every step its layout lays out, and its samples cover its KV when the prefill and each iteration of the policy’s loop sampled one id, since the KV holds all but the last.
- refuse(rid, node)[source]#
Give back
rid’s log onnode, which rank 0’s pool couldn’t hold, and return the paged KV cachesnodewrites forrid, where a drop then can’t freerid’s pages, as at rank 0’s ingest_request.
- start(rid, node, prefilling)[source]#
Open
rid’s rebuild onnodeover the slots tokens counts, which its drop checked, and return what it writes past its layout. A promptprefillingits step in progress reruns that step’s chunks last, and changes nothing the layout says, so this returns None for it.- Parameters:
rid (int)
node (str)
prefilling (Prefilling | None)
- Return type:
Rebuild | None
- class mstar.worker.rebuild.RowRecord(inputs, step_metadata, queued=0, landed=None)[source]#
Bases:
objectOne prefill step of a request as it ran, kept so its rebuild can run the step again: host copies of every input the node read in it, as the step read it, in the request’s slice, and the step’s metadata.
queuedcounts the copies that haven’t started, andlandedis recorded after the last that has, None where the copies run in place.- Parameters:
- class mstar.worker.rebuild.RowRun(row, ran)[source]#
Bases:
NamedTupleA rebuild row that has run, and the slots its chunks ran, which leave out a cached prefix.
- Parameters:
row (PlannedRow)
ran (int)
- row: PlannedRow#
Alias for field number 0
- mstar.worker.rebuild.node_writes(labels, nodes, my_walks)[source]#
What each of a paged KV cache’s
nodesthat runs here writes of the cache’slabels.
- mstar.worker.rebuild.policies_run_here(policies, graphs, my_walks, parallel_nodes)[source]#
The policies whose rebuild this worker runs, each with the walks this worker runs its node in: it runs the node in every walk its loop grows the KV in, and in one of the policy’s walks. A disaggregated decode node can’t run the prefill walk its rebuild needs.
Above one rank a policy keeps only the walks with nothing to copy, since rank 0 decides a drop from its own host copies, and a follower’s may not have landed.
- Parameters:
policies (Mapping[str, RebuildPolicy])
graphs (Mapping[str, GraphSection])
my_walks (Collection[tuple[str, str]])
parallel_nodes (Collection[str])
- Return type: