mstar.engine.resources.lora.config#

What a model declares about per-request LoRA adapters.

Kept free of the pool and its kernels, so declaring a step imports neither.

Each declared target linear computes y += scaling[slot] * (x @ A[slot]^T) @ B[slot]^T next to its frozen base GEMM, with the slot chosen per row: rows of one batch may run different adapters, or none.

Functions

check_expert_ids(resource_key, max_slots, ...)

Refuse an experts target with more (slot, expert) ids, max_slots slots of its experts each, than the align op groups.

describe_load_error(exc)

What a failed adapter read tells a client: a refusal (a ValueError: rank, targets, shapes, weights changed on disk, a name already loaded) as itself, with server paths cut to base names as every route's errors are (scrub_paths), anything else (an I/O error naming the server's files, say) only that it could not be read.

lora_enabled(model_config)

Whether a deployment's YAML turns LoRA on (a resources: lora: block), with or without adapters named in it: adapters may also be loaded at runtime.

resolve_adapter_path(path)

The safetensors file an adapter path names: the path itself, or a PEFT directory's adapter_model.safetensors (what vLLM's lora_path names).

Classes

AdapterInfo(rank, unmapped[, misfits])

What validation needs to know about an adapter checkpoint, from its modules' shapes.

LoraModuleRef(target, layer[, shard_id, param])

Where one checkpoint module's adapter lands: a target, a stack layer, and the stacked-shard id of its slice (None for a single-slice target).

LoraReqConfig(adapter[, digest])

The adapter a request runs under.

LoraSpec(resource_key, nodes[, num_layers, ...])

Per-request LoRA adapters for the linears named in targets.

LoraTarget(in_features, out_slices[, ...])

One family of linears an adapter may touch, e.g. every layer's fused qkv_proj.

class mstar.engine.resources.lora.config.AdapterInfo(rank, unmapped, misfits=())[source]#

Bases: NamedTuple

What validation needs to know about an adapter checkpoint, from its modules’ shapes.

rank covers every LoRA module in the file, those left to another node’s pool included. misfits names modules whose shapes do not fit their target (an adapter trained on another size of the model).

Parameters:
misfits: tuple[str, ...]#

Alias for field number 2

rank: int#

Alias for field number 0

unmapped: tuple[str, ...]#

Alias for field number 1

class mstar.engine.resources.lora.config.LoraModuleRef(target, layer, shard_id=None, param=None)[source]#

Bases: NamedTuple

Where one checkpoint module’s adapter lands: a target, a stack layer, and the stacked-shard id of its slice (None for a single-slice target). param names the parameter of the bound module the target adapts, for a module with a target per parameter (MoE experts); None for a module that is one target.

Parameters:
layer: int#

Alias for field number 1

param: str | None#

Alias for field number 3

shard_id: str | int | None#

Alias for field number 2

target: str#

Alias for field number 0

class mstar.engine.resources.lora.config.LoraReqConfig(adapter, digest=None)[source]#

Bases: ResourceReqConfig

The adapter a request runs under. A base-model request carries none.

Which adapters are served is the API server’s to check (ServedModels), where a request names one; a pool refuses at ingest a name it does not serve, or serves under other weights than digest, which the request’s prompt was keyed under: the name was unloaded and loaded again since.

Parameters:
adapter: str#
digest: bytes | None = None#
validate(spec=None)[source]#

Raise ValueError if a client-supplied value is out of range, or one the resource built from spec cannot honour.

The conductor calls it at admission, so a bad request is a 400 before any worker sees it.

Parameters:

spec (LoraSpec | None)

Return type:

None

class mstar.engine.resources.lora.config.LoraSpec(resource_key, nodes, num_layers=<factory>, targets=<factory>, module_map=<factory>, max_slots=4, max_rank=16, cpu_capacity=16, adapters=<factory>, runtime_updates=False, runtime_update_timeout_s=120.0, adapter_digests=<factory>, dtype=None, backend='auto', convert_keys=None, ignored_modules=<factory>, dropped_targets=<factory>, declare=None)[source]#

Bases: NodeResourceSpec

Per-request LoRA adapters for the linears named in targets.

max_slots adapters are resident on the device at once, beside a zero adapter that base-model rows use, and up to cpu_capacity are kept pinned on the host. adapters maps the names requests ask for to safetensors paths. module_map places each checkpoint module on a target; a node’s spec comes from module_lora_spec (mstar.model.components.lora.modules), whose LoraMap is what binds the node’s modules, and a pool built with any other resolver is refused at bind. convert_keys rewrites a model-native checkpoint layout first. declare gives num_layers, targets, module_map and ignored_modules in place of passing them, on their first read: module_lora_spec walks a node’s modules there, which a deployment that never builds the spec skips.

Opt-in (NodeResourceSpec.opt_in): built only for a deployment whose serving config has a resources: lora: block. One spec covers every node an adapter reaches (an LLM and its vision tower), and each node gets a pool of its own (per_node): the nodes may sit in different lockstep instances, whose ranks must make the same slot decisions.

Parameters:
adapter_digests: dict[str, bytes]#
adapters: dict[str, str]#
apply_yaml_overrides(max_slots=None, max_rank=None, cpu_capacity=None, adapters=None, adapter_digests=None, runtime_updates=None, runtime_update_timeout_s=None, targets=None)[source]#

How many adapters stay resident, the largest rank served, and which adapters this deployment serves. targets serves just those of the model’s declared targets, the experts targets included (served only when named, dropped_targets): dropping the LM head and embedding saves their vocabulary-sized slots, a node left with none gets no pool, and an adapter that touches a dropped target is refused. Without targets the served set stays as it is.

Every value is checked before any is set, so a refused override leaves the spec as declared.

Parameters:
  • max_slots (int | None)

  • max_rank (int | None)

  • cpu_capacity (int | None)

  • adapters (dict[str, str] | None)

  • adapter_digests (dict[str, str] | None)

  • runtime_updates (bool | None)

  • runtime_update_timeout_s (float | None)

  • targets (list[str] | None)

backend: Literal['auto', 'torch', 'triton'] = 'auto'#
check_adapter(name, info)[source]#

Refuse an adapter this deployment cannot serve: no LoRA layers, a rank above max_rank, or a module outside the declared targets.

Parameters:
Return type:

None

convert_keys: Callable | None = None#
cpu_capacity: int = 16#
declare: Callable[[], dict[str, Any]] | None = None#
dropped_targets: dict[str, LoraTarget]#
dtype: dtype | None = None#
for_node(node)[source]#

node’s pool: the targets on it, keyed lora@<node>; None when the YAML’s targets left it none. Placement stays the whole spec’s, so a pool reads an adapter file as the API server checked it, leaving the modules another node’s targets place to that node’s pool.

Parameters:

node (str)

Return type:

LoraSpec | None

ignored_modules: Callable[[str], bool] | None#
max_rank: int = 16#
max_slots: int = 4#
module_map: Callable[[str], LoraModuleRef | None] | None#
num_layers: int#
property num_slots: int#

Device slots, the zero adapter’s included.

opt_in: ClassVar[bool] = True#
per_node: ClassVar[bool] = True#
request_config(model_kwargs)[source]#

The adapter the lora_adapter kwarg names, and the digest it was selected under; None for a base-model request, which lets the pool run lean (Resource.lean_capture).

Parameters:

model_kwargs (dict | None)

Return type:

LoraReqConfig | None

property resource_class: type[Resource]#

What builds this spec. Imported inside the property, so declaring a resource stays free of the manager and its kernels.

The builder, not necessarily the class built: an attention spec names AttentionManager, whose build picks a backend subclass.

runtime_update_timeout_s: float = 120.0#
runtime_updates: bool = False#
split_step_refusals(node, shard, tp, sp, specs)[source]#

What this resource can’t run on node, whose steps shard splits over sp SP ranks (tp TP ranks each), among specs (every spec, by key); each a sentence.

Parameters:
Return type:

list[str]

targets: dict[str, LoraTarget]#
class mstar.engine.resources.lora.config.LoraTarget(in_features, out_slices, parallel='column', slice_names=(), slice_units=None, slice_packs=None, slice_align=None, layers=None, node=None, kind='linear', per_segment=False, token_fold=1, experts=1, expert_split=None)[source]#

Bases: object

One family of linears an adapter may touch, e.g. every layer’s fused qkv_proj.

Shapes are the unsharded linear’s. out_slices lists the output widths of a fused projection in order ((q, k, v)); each slice has its own A and B, named in a checkpoint by the stacked-shard id in slice_names. A single linear has one slice.

parallel says how the base linear is split across tensor-parallel ranks:

  • "column": output rows are split, so A is replicated and B is sharded by output rows of each slice. The delta lands on this rank’s rows of the output.

  • "row": input columns are split, so A is sharded along the input and B replicated. x @ A_r^T is then a partial sum over this rank’s input columns, and so is the expanded delta: add it to the partial base output before the layer’s all-reduce, which sums both. LoRA adds no collective.

  • "none": every rank holds the whole adapter and adds the whole delta to an output that is already whole (after the base layer’s own collective).

slice_units counts the shardable units (heads) of each slice. A slice with fewer units than ranks is replicated across groups of ranks rather than split, as QKVParallelLinear does with GQA’s K/V heads. None treats every row as a unit.

slice_packs names checkpoint modules that pack several slices into one tensor: a gated delta-net’s in_proj_qkv is [q | k | v], three outputs the base shards across ranks on their own. Such a module has one A, which each of its slices takes, and a B whose rows are those slices’ in order. A pack named None is the module itself as a checkpoint stores it fused, with no stacked-shard id.

slice_align rounds each slice’s rank-local width up to a multiple, for a base linear that pads its fused output per rank so the next slice starts aligned. The padding rows follow the slice’s own, and B is zero there.

layers names the stack layer indices this target exists on (a hybrid stack has attention on some layers only); None means all of the spec’s num_layers.

node is the graph node the target’s linears run on, which a spec over several nodes names for every target (each node’s pool holds its own, LoraSpec.for_node); None for a spec over one node.

kind="embedding" is a token embedding: in_features is the vocabulary, a row’s input is its token id, and the shrink gathers A’s column for it (x @ A^T of the token’s one-hot), as PEFT’s lora_embedding_A / _B define it.

Which rows the linear sees, for the per-row adapter choice: one per token of the step (the default), one per segment (per_segment, e.g. an LM head over each row’s last token), or one per token_fold consecutive tokens of a segment (a patch merger that folds 2x2 patches into one row).

experts counts the groups of the weight: 1 for a linear, E for one stacked parameter of an MoE layer’s routed experts ([E, out, in]), where each row runs its token’s slot’s adapter on the expert the router picked for it. Each expert has its own A and B per slice, named in a checkpoint by the stacked-shard id "<slice>:<expert>" ("gate:3"). expert_split is (first, count) of the experts one rank’s module holds when expert parallelism splits them; None when it holds every expert.

Parameters:
expert_of(shard_id)[source]#

The expert a checkpoint module of an experts target holds, from its "<slice>:<expert>" shard id; 0 for any other target.

Parameters:

shard_id (str | int | None)

Return type:

int

expert_split: tuple[int, int] | None = None#
experts: int = 1#
in_features: int#
input_cols(rank, world_size)[source]#

(start, size) of rank’s input columns.

Parameters:
Return type:

tuple[int, int]

kind: Literal['linear', 'embedding'] = 'linear'#
layers: tuple[int, ...] | None = None#
local_width(slice_idx, world_size)[source]#

Rows of slice slice_idx in one rank’s base output, padding included.

Parameters:
  • slice_idx (int)

  • world_size (int)

Return type:

int

module_slices(shard_id)[source]#

The slices a checkpoint module lands on, in the order its B’s rows hold them: several for a slice_packs name, else the one slice_index gives.

Parameters:

shard_id (str | int | None)

Return type:

tuple[int, …]

node: str | None = None#
property num_slices: int#
property out_features: int#
out_slices: tuple[int, ...]#
parallel: Literal['column', 'row', 'none'] = 'column'#
per_segment: bool = False#
property row_grouping: int#

1 per token, k per k folded tokens, 0 per segment.

Type:

Which addressing the linear reads

shard(world_size)[source]#

This target as one rank of world_size sees it: its slices the widths they have in that rank’s base output, alignment padding included.

Parameters:

world_size (int)

Return type:

LoraTarget

shard_pair(slice_idx, a, b, rank, world_size)[source]#

rank’s share of one slice’s a [r, in] and b [out_slice, r].

Parameters:
Return type:

tuple[Tensor, Tensor]

slice_align: tuple[int, ...] | None = None#
slice_index(shard_id)[source]#

Which slice a checkpoint module lands on, from its stacked-shard id.

Parameters:

shard_id (str | int | None)

Return type:

int

slice_names: tuple[str | int, ...] = ()#
slice_packs: tuple[tuple[str | int | None, tuple[str | int, ...]], ...] | None = None#
slice_rows(slice_idx, rank, world_size)[source]#

(start, size) of rank’s output rows within slice slice_idx.

Parameters:
  • slice_idx (int)

  • rank (int)

  • world_size (int)

Return type:

tuple[int, int]

slice_units: tuple[int, ...] | None = None#
token_fold: int = 1#
mstar.engine.resources.lora.config.check_expert_ids(resource_key, max_slots, targets)[source]#

Refuse an experts target with more (slot, expert) ids, max_slots slots of its experts each, than the align op groups.

Parameters:
Return type:

None

mstar.engine.resources.lora.config.describe_load_error(exc)[source]#

What a failed adapter read tells a client: a refusal (a ValueError: rank, targets, shapes, weights changed on disk, a name already loaded) as itself, with server paths cut to base names as every route’s errors are (scrub_paths), anything else (an I/O error naming the server’s files, say) only that it could not be read. Deterministic, so every rank of an instance says the same. The caller logs exc in full.

Parameters:

exc (BaseException)

Return type:

str

mstar.engine.resources.lora.config.lora_enabled(model_config)[source]#

Whether a deployment’s YAML turns LoRA on (a resources: lora: block), with or without adapters named in it: adapters may also be loaded at runtime. A resources: value that is not a mapping turns nothing on: the conductor’s validate_deployment reports it with the deployment’s other problems.

Parameters:

model_config (Mapping[str, Any])

Return type:

bool

mstar.engine.resources.lora.config.resolve_adapter_path(path)[source]#

The safetensors file an adapter path names: the path itself, or a PEFT directory’s adapter_model.safetensors (what vLLM’s lora_path names).

Parameters:

path (str)

Return type:

str