mstar.worker.host_copies#

Host copies of a step’s outputs: the pinned staging buffers they go through and the side stream that copies them without waiting on the next step.

The default stream has GPU(N+1) queued behind GPU(N)’s outputs once N+1 is launched, so syncing on it would also drain GPU(N+1) and erase the overlap. The side stream waits on output.completion_event (recorded after GPU(N)) and then runs an isolated D2H, so the main thread only blocks on the copy.

Functions

crop_to_spans(outputs)

Cut each row of a spanned output to its span along the last dim (NodeSubmodule.spanned_outputs).

rows_to_per_rid(host, request_ids)

Slice row-addressed buffers into the per-rid form check_stop reads: row i goes to request_ids[i].

Classes

HostCopies(device, device_capability)

class mstar.worker.host_copies.HostCopies(device, device_capability)[source]#

Bases: object

Parameters:
prematerialize_for_check_stop(outputs, completion_event, request_ids=None)[source]#

Side-stream D→H of the CUDA tensors in outputs so the subsequent check_stop reads (typically .item() on the sampled token) don’t trigger a default-stream sync. With same-thread async, GPU(N+1)’s kernels are already queued on default stream behind N’s outputs by the time we get here — a default-stream sync would block waiting for N+1 to finish, defeating the overlap.

Returns per-rid host outputs, plus HostRows when the submodule handed over row-addressed buffers, for a batched stop check. Without those, the outputs are a _LazyHostOutputs, which copies an output name for every rid the first time it is read. Skipped (returns outputs unchanged) when there’s no completion event (CPU execution) or when CUDA is unavailable.

The host tensors are views of pinned buffers the next step reuses. Their one other reader, a needs_cpu_tensor transport, gets them through the store and sends in _register_outputs later in the same postprocess, which then drops them from the store.

Parameters:
Return type:

tuple[Mapping[int, Mapping[str, list[Tensor]]], HostRows | None]

stage_host_rows(outputs)[source]#

Start the device-to-host copy of the step’s row-addressed stop buffers here on the gpu thread, on the execution stream right behind the clone that made them, so the completion event recorded next covers it. The main thread’s stop check then finds the rows landed once it has waited on that event: no side stream, no second sync, and one fewer GIL release on its critical path.

A small ring of pinned buffers: step N’s rows are read by the main thread while this thread already stages N+1’s (the pipeline is one step deep, the ring leaves slack).

The step’s output spans (BatchedModelOutput.output_spans) take the same path, so cutting the outputs to them costs no wait of its own.

Parameters:

outputs (BatchedModelOutput)

Return type:

None

mstar.worker.host_copies.crop_to_spans(outputs)[source]#

Cut each row of a spanned output to its span along the last dim (NodeSubmodule.spanned_outputs). Called once the step’s completion event has fired, so the spans staged behind the step are on the host. The cut is a view, and it replaces the tensor in its list in place, so a second name that postprocess or unpack_packed_outputs gave the same list is cut with it. Two names the forward returned on its own get their own lists from the engine and keep their own widths.

Returns rid -> the outputs whose span is empty, which the caller leaves out of the store as it does a withheld output.

Parameters:

outputs (BatchedModelOutput)

Return type:

dict[int, list[str]]

mstar.worker.host_copies.rows_to_per_rid(host, request_ids)[source]#

Slice row-addressed buffers into the per-rid form check_stop reads: row i goes to request_ids[i].

Parameters:
Return type:

dict[int, dict[str, list[Tensor]]]