mstar.engine.resources.lora.ops#

The LoRA shrink and expand as custom ops, so torch.compile sees one opaque node each, and lora_apply_, the two as one op, which is what a linear target runs; lora_experts_apply_ is an experts target’s shrink and expands over combined ids.

mstar::lora_expand_ adds into y in place and says so (mutates_args): functionalization then keeps every read of y on the right side of the add and never drops it, whatever Inductor reorders around it.

The kernel registered here is the plain-torch reference (backend.shrink_reference / expand_reference_ / experts_reference_), on any device, with no host sync and no data-dependent shape; triton_backend registers the CUDA kernels over it. Both read the same arguments with the same meaning, so the fake (shape) implementations and torch.library.opcheck can be checked on a CPU.

Rows of h the expand does not read (tokens on the zero adapter, columns past a slot’s rank) are unspecified; the reference writes zeros there.

b must be finite everywhere, including b[slot, :, rank:] past each slot’s rank: the CUDA expand reads it up to the max rank against zeros in h, so a NaN or inf there reaches the output (the reference’s one GEMM over every slot reads all of b). The pool keeps those columns zero.

Functions

experts_alignment(ids, num_ids)

ids grouped by id and padded to experts_config's BLOCK_SIZE_M (a MoeAlignment), as the fused-MoE kernel reads them.

experts_config(ids, num_ids)

The tiles of every LoRA GEMM over one layer's combined ids, which share one alignment and so one BLOCK_SIZE_M: the base fused-MoE kernel's defaults, which pick by token and id counts alone.

Classes

LoraOpsBackend()

The custom ops: on CUDA the Triton kernels once triton_backend is imported, elsewhere the reference above, so a CPU test runs exactly the graph a GPU compiles.

class mstar.engine.resources.lora.ops.LoraOpsBackend[source]#

Bases: LoraBackend

The custom ops: on CUDA the Triton kernels once triton_backend is imported, elsewhere the reference above, so a CPU test runs exactly the graph a GPU compiles.

align_experts(ids, num_ids)[source]#

What apply_experts_ reads to group the rows of ids ([tokens, top_k] combined ids, num_ids meaning none) by id, shared by every call over them; None where nothing groups them (experts_reference_).

apply_experts_(x, y, a, b, ids, row_weight, top_k, out_slices, alignment=None)[source]#

y[m, slice i] += row_weight[m] * (x[m // top_k] @ a[id, slice i]^T) @ b[id, slice i]^T for each row m of y, id = ids.flatten()[m]; a row whose id is past a’s (a.shape[0]) gets nothing. a [ids, slices * R, in], b [ids, out, R]; alignment is align_experts’ for ids.

apply_linear_(x, y, a, b, addressing, ranks, scalings, out_slices)[source]#

Shrink then expand into y, for a linear target; a backend may fuse them. b as for expand_fused.

Parameters:

addressing (LoraAddressing)

expand_fused(h, b, addressing, ranks, scalings, y, out_slices)[source]#

y[t, slice i] += scalings[slot(t)] * h[t, slice i] @ b[slot(t), slice i]^T. b must be finite everywhere, past a slot’s rank included: a backend may read it up to the max rank against zeros in h (the pool keeps it zero there).

Parameters:

addressing (LoraAddressing)

gather_shrink(ids, a, addressing, ranks)[source]#

h[t] = a[slot(t)][:, ids[t]], the shrink of a token’s one-hot; an id outside [0, in) (e.g. an image position’s -1) contributes nothing.

Parameters:

addressing (LoraAddressing)

shrink(x, a, addressing, ranks, num_slices=1)[source]#

h[t] = x[t] @ a[slot(t)]^T. Rows of rank-0 slots, and columns past a slot’s rank, may hold anything: the expand does not read them.

Parameters:

addressing (LoraAddressing)

mstar.engine.resources.lora.ops.experts_alignment(ids, num_ids)[source]#

ids grouped by id and padded to experts_config’s BLOCK_SIZE_M (a MoeAlignment), as the fused-MoE kernel reads them.

Parameters:
mstar.engine.resources.lora.ops.experts_config(ids, num_ids)[source]#

The tiles of every LoRA GEMM over one layer’s combined ids, which share one alignment and so one BLOCK_SIZE_M: the base fused-MoE kernel’s defaults, which pick by token and id counts alone.

Parameters:
Return type:

dict