mstar.engine.resources.lora.config#
What a model declares about per-request LoRA adapters.
Kept free of the pool and its kernels, so declaring a step imports neither.
Each declared target linear computes y += scaling[slot] * (x @ A[slot]^T) @ B[slot]^T next
to its frozen base GEMM, with the slot chosen per row: rows of one batch may run different
adapters, or none.
Functions
|
Refuse an experts target with more (slot, expert) ids, |
|
What a failed adapter read tells a client: a refusal (a |
|
Whether a deployment's YAML turns LoRA on (a |
|
The safetensors file an adapter path names: the path itself, or a PEFT directory's |
Classes
|
What validation needs to know about an adapter checkpoint, from its modules' shapes. |
|
Where one checkpoint module's adapter lands: a target, a stack layer, and the stacked-shard id of its slice (None for a single-slice target). |
|
The adapter a request runs under. |
|
Per-request LoRA adapters for the linears named in |
|
One family of linears an adapter may touch, e.g. every layer's fused |
- class mstar.engine.resources.lora.config.AdapterInfo(rank, unmapped, misfits=())[source]#
Bases:
NamedTupleWhat validation needs to know about an adapter checkpoint, from its modules’ shapes.
rankcovers every LoRA module in the file, those left to another node’s pool included.misfitsnames modules whose shapes do not fit their target (an adapter trained on another size of the model).
- class mstar.engine.resources.lora.config.LoraModuleRef(target, layer, shard_id=None, param=None)[source]#
Bases:
NamedTupleWhere one checkpoint module’s adapter lands: a target, a stack layer, and the stacked-shard id of its slice (None for a single-slice target).
paramnames the parameter of the bound module the target adapts, for a module with a target per parameter (MoE experts); None for a module that is one target.
- class mstar.engine.resources.lora.config.LoraReqConfig(adapter, digest=None)[source]#
Bases:
ResourceReqConfigThe adapter a request runs under. A base-model request carries none.
Which adapters are served is the API server’s to check (
ServedModels), where a request names one; a pool refuses at ingest a name it does not serve, or serves under other weights thandigest, which the request’s prompt was keyed under: the name was unloaded and loaded again since.
- class mstar.engine.resources.lora.config.LoraSpec(resource_key, nodes, num_layers=<factory>, targets=<factory>, module_map=<factory>, max_slots=4, max_rank=16, cpu_capacity=16, adapters=<factory>, runtime_updates=False, runtime_update_timeout_s=120.0, adapter_digests=<factory>, dtype=None, backend='auto', convert_keys=None, ignored_modules=<factory>, dropped_targets=<factory>, declare=None)[source]#
Bases:
NodeResourceSpecPer-request LoRA adapters for the linears named in
targets.max_slotsadapters are resident on the device at once, beside a zero adapter that base-model rows use, and up tocpu_capacityare kept pinned on the host.adaptersmaps the names requests ask for to safetensors paths.module_mapplaces each checkpoint module on a target; a node’s spec comes frommodule_lora_spec(mstar.model.components.lora.modules), whoseLoraMapis what binds the node’s modules, and a pool built with any other resolver is refused at bind.convert_keysrewrites a model-native checkpoint layout first.declaregivesnum_layers,targets,module_mapandignored_modulesin place of passing them, on their first read:module_lora_specwalks a node’s modules there, which a deployment that never builds the spec skips.Opt-in (
NodeResourceSpec.opt_in): built only for a deployment whose serving config has aresources: lora:block. One spec covers every node an adapter reaches (an LLM and its vision tower), and each node gets a pool of its own (per_node): the nodes may sit in different lockstep instances, whose ranks must make the same slot decisions.- Parameters:
resource_key (str)
num_layers (int)
targets (dict[str, LoraTarget])
module_map (Callable[[str], LoraModuleRef | None] | None)
max_slots (int)
max_rank (int)
cpu_capacity (int)
runtime_updates (bool)
runtime_update_timeout_s (float)
dtype (dtype | None)
backend (Literal['auto', 'torch', 'triton'])
convert_keys (Callable | None)
dropped_targets (dict[str, LoraTarget])
- apply_yaml_overrides(max_slots=None, max_rank=None, cpu_capacity=None, adapters=None, adapter_digests=None, runtime_updates=None, runtime_update_timeout_s=None, targets=None)[source]#
How many adapters stay resident, the largest rank served, and which adapters this deployment serves.
targetsserves just those of the model’s declared targets, the experts targets included (served only when named,dropped_targets): dropping the LM head and embedding saves their vocabulary-sized slots, a node left with none gets no pool, and an adapter that touches a dropped target is refused. Withouttargetsthe served set stays as it is.Every value is checked before any is set, so a refused override leaves the spec as declared.
- check_adapter(name, info)[source]#
Refuse an adapter this deployment cannot serve: no LoRA layers, a rank above
max_rank, or a module outside the declared targets.- Parameters:
name (str)
info (AdapterInfo)
- Return type:
None
- dropped_targets: dict[str, LoraTarget]#
- for_node(node)[source]#
node’s pool: the targets on it, keyedlora@<node>; None when the YAML’stargetsleft it none. Placement stays the whole spec’s, so a pool reads an adapter file as the API server checked it, leaving the modules another node’s targets place to that node’s pool.
- request_config(model_kwargs)[source]#
The adapter the
lora_adapterkwarg names, and the digest it was selected under; None for a base-model request, which lets the pool run lean (Resource.lean_capture).- Parameters:
model_kwargs (dict | None)
- Return type:
LoraReqConfig | None
- property resource_class: type[Resource]#
What builds this spec. Imported inside the property, so declaring a resource stays free of the manager and its kernels.
The builder, not necessarily the class built: an attention spec names AttentionManager, whose build picks a backend subclass.
- split_step_refusals(node, shard, tp, sp, specs)[source]#
What this resource can’t run on
node, whose stepsshardsplits overspSP ranks (tpTP ranks each), amongspecs(every spec, by key); each a sentence.- Parameters:
node (str)
shard (SeqShardSpec)
tp (int)
sp (int)
specs (Mapping[str, NodeResourceSpec])
- Return type:
- targets: dict[str, LoraTarget]#
- class mstar.engine.resources.lora.config.LoraTarget(in_features, out_slices, parallel='column', slice_names=(), slice_units=None, slice_packs=None, slice_align=None, layers=None, node=None, kind='linear', per_segment=False, token_fold=1, experts=1, expert_split=None)[source]#
Bases:
objectOne family of linears an adapter may touch, e.g. every layer’s fused
qkv_proj.Shapes are the unsharded linear’s.
out_sliceslists the output widths of a fused projection in order ((q, k, v)); each slice has its own A and B, named in a checkpoint by the stacked-shard id inslice_names. A single linear has one slice.parallelsays how the base linear is split across tensor-parallel ranks:"column": output rows are split, so A is replicated and B is sharded by output rows of each slice. The delta lands on this rank’s rows of the output."row": input columns are split, so A is sharded along the input and B replicated.x @ A_r^Tis then a partial sum over this rank’s input columns, and so is the expanded delta: add it to the partial base output before the layer’s all-reduce, which sums both. LoRA adds no collective."none": every rank holds the whole adapter and adds the whole delta to an output that is already whole (after the base layer’s own collective).
slice_unitscounts the shardable units (heads) of each slice. A slice with fewer units than ranks is replicated across groups of ranks rather than split, asQKVParallelLineardoes with GQA’s K/V heads. None treats every row as a unit.slice_packsnames checkpoint modules that pack several slices into one tensor: a gated delta-net’sin_proj_qkvis[q | k | v], three outputs the base shards across ranks on their own. Such a module has one A, which each of its slices takes, and a B whose rows are those slices’ in order. A pack named None is the module itself as a checkpoint stores it fused, with no stacked-shard id.slice_alignrounds each slice’s rank-local width up to a multiple, for a base linear that pads its fused output per rank so the next slice starts aligned. The padding rows follow the slice’s own, and B is zero there.layersnames the stack layer indices this target exists on (a hybrid stack has attention on some layers only); None means all of the spec’snum_layers.nodeis the graph node the target’s linears run on, which a spec over several nodes names for every target (each node’s pool holds its own,LoraSpec.for_node); None for a spec over one node.kind="embedding"is a token embedding:in_featuresis the vocabulary, a row’s input is its token id, and the shrink gathers A’s column for it (x @ A^Tof the token’s one-hot), as PEFT’slora_embedding_A/_Bdefine it.Which rows the linear sees, for the per-row adapter choice: one per token of the step (the default), one per segment (
per_segment, e.g. an LM head over each row’s last token), or one pertoken_foldconsecutive tokens of a segment (a patch merger that folds 2x2 patches into one row).expertscounts the groups of the weight: 1 for a linear,Efor one stacked parameter of an MoE layer’s routed experts ([E, out, in]), where each row runs its token’s slot’s adapter on the expert the router picked for it. Each expert has its own A and B per slice, named in a checkpoint by the stacked-shard id"<slice>:<expert>"("gate:3").expert_splitis(first, count)of the experts one rank’s module holds when expert parallelism splits them; None when it holds every expert.- Parameters:
- expert_of(shard_id)[source]#
The expert a checkpoint module of an experts target holds, from its
"<slice>:<expert>"shard id; 0 for any other target.
- local_width(slice_idx, world_size)[source]#
Rows of slice
slice_idxin one rank’s base output, padding included.
- module_slices(shard_id)[source]#
The slices a checkpoint module lands on, in the order its B’s rows hold them: several for a
slice_packsname, else the oneslice_indexgives.
- property row_grouping: int#
1 per token,
kperkfolded tokens, 0 per segment.- Type:
Which addressing the linear reads
- shard(world_size)[source]#
This target as one rank of
world_sizesees it: its slices the widths they have in that rank’s base output, alignment padding included.- Parameters:
world_size (int)
- Return type:
- shard_pair(slice_idx, a, b, rank, world_size)[source]#
rank’s share of one slice’sa [r, in]andb [out_slice, r].
- mstar.engine.resources.lora.config.check_expert_ids(resource_key, max_slots, targets)[source]#
Refuse an experts target with more (slot, expert) ids,
max_slotsslots of its experts each, than the align op groups.- Parameters:
resource_key (str)
max_slots (int)
targets (Mapping[str, LoraTarget])
- Return type:
None
- mstar.engine.resources.lora.config.describe_load_error(exc)[source]#
What a failed adapter read tells a client: a refusal (a
ValueError: rank, targets, shapes, weights changed on disk, a name already loaded) as itself, with server paths cut to base names as every route’s errors are (scrub_paths), anything else (an I/O error naming the server’s files, say) only that it could not be read. Deterministic, so every rank of an instance says the same. The caller logsexcin full.- Parameters:
exc (BaseException)
- Return type:
- mstar.engine.resources.lora.config.lora_enabled(model_config)[source]#
Whether a deployment’s YAML turns LoRA on (a
resources: lora:block), with or without adapters named in it: adapters may also be loaded at runtime. Aresources:value that is not a mapping turns nothing on: the conductor’svalidate_deploymentreports it with the deployment’s other problems.