mstar.worker.host_pool#

Host memory a worker copies request state into, so a request can leave the device and come back.

Functions

aligned(nbytes)

nbytes rounded up to where the next slice can start.

Classes

HostCopy(view, landed)

A device tensor copied into the pool: a view of where it went, and the event that records the copy, None when it copied in place.

HostPool(nbytes, device)

nbytes of host memory allocated once, pinned where device takes pinned memory, and handed out first fit.

HostSlice(offset, nbytes)

class mstar.worker.host_pool.HostCopy(view, landed)[source]#

Bases: NamedTuple

A device tensor copied into the pool: a view of where it went, and the event that records the copy, None when it copied in place.

Parameters:
landed: Event | None#

Alias for field number 1

view: Tensor#

Alias for field number 0

class mstar.worker.host_pool.HostPool(nbytes, device)[source]#

Bases: object

nbytes of host memory allocated once, pinned where device takes pinned memory, and handed out first fit. The pool rounds nbytes up to a power of two, as the pinned allocator does, so it hands out all it pins. Copies run on streams of the pool’s own, one each way, so a copy never waits behind a later step, and a copy back to the device never waits behind copies to the host.

Parameters:
copy_in(view)[source]#

Copy view, a tensor in the pool whose copy from the device has landed, to the device, and return the copy with the event recorded once it lands. Where the device is the host, return view as it is and no event.

Parameters:

view (Tensor)

Return type:

tuple[Tensor, Event | None]

copy_out(src, into, after)[source]#

Copy src as it is to the start of into once after (the step that wrote it) completes, running nothing on the device but the copy.

Parameters:
Return type:

HostCopy

give(piece)[source]#
Parameters:

piece (HostSlice)

Return type:

None

property nbytes: int#
nbytes_used()[source]#
Return type:

int

take(nbytes, spill=False)[source]#

A free slice of nbytes, or None when no free slice holds it, unless spill, which then allocates memory for the slice alone and places it past the pool’s end.

Parameters:
Return type:

HostSlice | None

view(into, dtype, shape)[source]#

The start of into as a tensor of dtype and shape.

Parameters:
Return type:

Tensor

class mstar.worker.host_pool.HostSlice(offset, nbytes)[source]#

Bases: NamedTuple

Parameters:
nbytes: int#

Alias for field number 1

offset: int#

Alias for field number 0

mstar.worker.host_pool.aligned(nbytes)[source]#

nbytes rounded up to where the next slice can start.

Parameters:

nbytes (int)

Return type:

int