mstar.worker.host_copies#
Host copies of a step’s outputs: the pinned staging buffers they go through and the side stream that copies them without waiting on the next step.
The default stream has GPU(N+1) queued behind GPU(N)’s outputs once N+1 is
launched, so syncing on it would also drain GPU(N+1) and erase the overlap. The
side stream waits on output.completion_event (recorded after GPU(N)) and then
runs an isolated D2H, so the main thread only blocks on the copy.
Functions
|
Cut each row of a spanned output to its span along the last dim ( |
|
Slice row-addressed buffers into the per-rid form |
Classes
|
- class mstar.worker.host_copies.HostCopies(device, device_capability)[source]#
Bases:
object- Parameters:
device (torch.device)
device_capability (DeviceCapability)
- prematerialize_for_check_stop(outputs, completion_event, request_ids=None)[source]#
Side-stream D→H of the CUDA tensors in
outputsso the subsequentcheck_stopreads (typically.item()on the sampled token) don’t trigger a default-stream sync. With same-thread async, GPU(N+1)’s kernels are already queued on default stream behind N’s outputs by the time we get here — a default-stream sync would block waiting for N+1 to finish, defeating the overlap.Returns per-rid host outputs, plus
HostRowswhen the submodule handed over row-addressed buffers, for a batched stop check. Without those, the outputs are a_LazyHostOutputs, which copies an output name for every rid the first time it is read. Skipped (returnsoutputsunchanged) when there’s no completion event (CPU execution) or when CUDA is unavailable.The host tensors are views of pinned buffers the next step reuses. Their one other reader, a
needs_cpu_tensortransport, gets them through the store and sends in_register_outputslater in the samepostprocess, which then drops them from the store.
- stage_host_rows(outputs)[source]#
Start the device-to-host copy of the step’s row-addressed stop buffers here on the gpu thread, on the execution stream right behind the clone that made them, so the completion event recorded next covers it. The main thread’s stop check then finds the rows landed once it has waited on that event: no side stream, no second sync, and one fewer GIL release on its critical path.
A small ring of pinned buffers: step N’s rows are read by the main thread while this thread already stages N+1’s (the pipeline is one step deep, the ring leaves slack).
The step’s output spans (
BatchedModelOutput.output_spans) take the same path, so cutting the outputs to them costs no wait of its own.- Parameters:
outputs (BatchedModelOutput)
- Return type:
None
- mstar.worker.host_copies.crop_to_spans(outputs)[source]#
Cut each row of a spanned output to its span along the last dim (
NodeSubmodule.spanned_outputs). Called once the step’s completion event has fired, so the spans staged behind the step are on the host. The cut is a view, and it replaces the tensor in its list in place, so a second name thatpostprocessorunpack_packed_outputsgave the same list is cut with it. Two names the forward returned on its own get their own lists from the engine and keep their own widths.Returns rid -> the outputs whose span is empty, which the caller leaves out of the store as it does a withheld output.
- Parameters:
outputs (BatchedModelOutput)
- Return type: