mstar.engine.resources.lora.backend#
Kernels that add a batch’s LoRA deltas: shrink h = x @ A^T, then expand y += s * h @ B^T.
Both read the step’s LoraAddressing from the pool, never a model-side index, so the eager
and captured paths address slots the same way. Shapes, per layer of one target:
x [T, in],y [T, out](written in place),h [T, slices * R]a [slots, slices * R, in],b [slots, out, R],ranks/scalings[slots](slot 0, the zero adapter, has rank 0)
out_slices are this rank’s widths of a fused projection’s outputs ((q, k, v)); slice
i reads h[:, i*R:(i+1)*R] and writes its own columns of y.
An embedding target’s shrink is a gather instead (gather_shrink): its input is a token
id, in is the vocabulary, and x @ A^T of the id’s one-hot is A’s column for it.
An experts target (apply_experts_) holds one A and B per (slot, expert), addressed per
output row by a combined id (slot - 1) * experts + expert; the zero adapter’s are not
stored, and any id past them is a row that gets nothing.
Functions
|
|
|
Plain torch for |
|
Plain torch for |
|
The custom ops ( |
|
|
Classes
- class mstar.engine.resources.lora.backend.LoraBackend[source]#
Bases:
ABC- abstractmethod align_experts(ids, num_ids)[source]#
What
apply_experts_reads to group the rows ofids([tokens, top_k]combined ids,num_idsmeaning none) by id, shared by every call over them; None where nothing groups them (experts_reference_).- Parameters:
ids (torch.Tensor)
num_ids (int)
- Return type:
MoeAlignment | None
- apply_(x, y, a, b, addressing, ranks, scalings, out_slices, embedding=False)[source]#
Shrink (a gather for an
embeddingtarget, whosexis token ids) then expand intoy; returnsy.- Parameters:
x (torch.Tensor)
y (torch.Tensor)
a (torch.Tensor)
b (torch.Tensor)
addressing (LoraAddressing)
ranks (torch.Tensor)
scalings (torch.Tensor)
embedding (bool)
- Return type:
- abstractmethod apply_experts_(x, y, a, b, ids, row_weight, top_k, out_slices, alignment=None)[source]#
y[m, slice i] += row_weight[m] * (x[m // top_k] @ a[id, slice i]^T) @ b[id, slice i]^Tfor each rowmofy,id = ids.flatten()[m]; a row whose id is pasta’s (a.shape[0]) gets nothing.a [ids, slices * R, in],b [ids, out, R];alignmentisalign_experts’ forids.- Parameters:
x (torch.Tensor)
y (torch.Tensor)
a (torch.Tensor)
b (torch.Tensor)
ids (torch.Tensor)
row_weight (torch.Tensor)
top_k (int)
alignment (MoeAlignment | None)
- Return type:
None
- apply_linear_(x, y, a, b, addressing, ranks, scalings, out_slices)[source]#
Shrink then expand into
y, for a linear target; a backend may fuse them.bas forexpand_fused.- Parameters:
x (torch.Tensor)
y (torch.Tensor)
a (torch.Tensor)
b (torch.Tensor)
addressing (LoraAddressing)
ranks (torch.Tensor)
scalings (torch.Tensor)
- Return type:
None
- abstractmethod expand_fused(h, b, addressing, ranks, scalings, y, out_slices)[source]#
y[t, slice i] += scalings[slot(t)] * h[t, slice i] @ b[slot(t), slice i]^T.bmust be finite everywhere, past a slot’s rank included: a backend may read it up to the max rank against zeros inh(the pool keeps it zero there).- Parameters:
h (torch.Tensor)
b (torch.Tensor)
addressing (LoraAddressing)
ranks (torch.Tensor)
scalings (torch.Tensor)
y (torch.Tensor)
- Return type:
None
- gather_shrink(ids, a, addressing, ranks)[source]#
h[t] = a[slot(t)][:, ids[t]], the shrink of a token’s one-hot; an id outside[0, in)(e.g. an image position’s-1) contributes nothing.- Parameters:
ids (torch.Tensor)
a (torch.Tensor)
addressing (LoraAddressing)
ranks (torch.Tensor)
- Return type:
- abstractmethod shrink(x, a, addressing, ranks, num_slices=1)[source]#
h[t] = x[t] @ a[slot(t)]^T. Rows of rank-0 slots, and columns past a slot’s rank, may hold anything: the expand does not read them.- Parameters:
x (torch.Tensor)
a (torch.Tensor)
addressing (LoraAddressing)
ranks (torch.Tensor)
num_slices (int)
- Return type:
- mstar.engine.resources.lora.backend.expand_reference_(y, h, b, token_slot, ranks, scalings, out_slices)[source]#
y[:, slice] += scaling * h[:, slice ranks] @ b[slot, slice]^Tin place, asshrink_referencecomputes it.
- mstar.engine.resources.lora.backend.experts_reference_(x, y, a, b, ids, row_weight, top_k, out_slices)[source]#
Plain torch for
LoraBackend.apply_experts_: gathers each row’s factors,_EXPERT_ROWSrows at a time so the gathered copies stay bounded. Fixed shapes and no host sync, so it compiles and captures; fine on a CPU, at a dense LoRA’s work per row.
- mstar.engine.resources.lora.backend.gather_shrink_reference(ids, a, token_slot, ranks)[source]#
Plain torch for
gather_shrink: fixed shapes and no host sync, so it compiles and captures on any device.
- mstar.engine.resources.lora.backend.make_backend(kind, device)[source]#
The custom ops (
ops.LoraOpsBackend) on every device, sotorch.compiletakes each kernel as one opaque node."triton"("auto"on CUDA) registers the Triton kernels for them;"torch"registers none. A deployment’s pools share onelora:block, so they choose alike, but registration is process-wide: the reference runs only while nothing else in the process has registered the Triton kernels.- Parameters:
- Return type:
- mstar.engine.resources.lora.backend.shrink_reference(x, a, token_slot, ranks, num_slices)[source]#
h [T, slices * R]forx [T, in]againsta [slots, slices * R, in], every token against every slot and then its own (token_slot [T]) selected: no host sync and no data-dependent shape, so it compiles and captures, atslotstimes a dense LoRA’s work. Zeros where the expand does not read (the zero adapter’s rows, columns past a slot’s rank).