mstar.engine.resources.sequence_shard#
Sequence parallelism: whether a step’s tokens split over a node’s SP ranks, and how attention reaches the tokens it needs when they do.
A node’s SP ranks share each step one of two ways. Its tokens are split (each rank runs the
token-local layers on its share) or copied (every rank runs them all). The sequence shard
resource (SeqShardSpec) decides which, per step, from the step the model declares, and gives
each listed label its per-rank ranges (SeqRange): contiguous, or on a node whose attention
runs the ring, striped by KV position. A model brackets its layer stack with scatter /
gather on that resource; nothing else in model code changes.
The attention resources over the node depend on it (shard_key, taken in their specs’
derive) and pick an exchange per label from its plan: for a split label, the shard’s
exchange over their storage; for a copied one, what attends it at the node’s head geometry.
The exchanges are the functions at the bottom of this module, each wrapped around the resource’s
own run:
ulysses: one all-to-all from tokens to heads,run, and one back (head-stored KV, a cacheless attention planning a label the shard lists,RaggedAttentionSpec.labels, or a recurrent layer’s prompt rows);all_gather: every rank’s q, k and v gathered,runas on a copied step, this rank’s share of the output (KV stored by position,context_parallel);ring: a striped label’s own tokens, whose K/V this rank stores,runon them with their indices in the step; the plan passes queries round the partners (KV stored by position,RingAttentionWrapper);head_slice: a copied step at atp * sphead geometry: this rank’s heads,run, gather the heads;sharded_key_merge: copied queries over split keys (cacheless cross-attention): each rank’s partial with its log-sum-exp, merged;kv_gather: every rank’s k and v gathered,runwrites them and attends this rank’s queries (KV every rank stores whole: a sliding-window attention’s layers of a pool stored by position).
A recurrent layer’s resource takes its conv’s left context from the earlier ranks (conv_halo),
and its prompt rows through ulysses where its heads split over the SP ranks or else
state_passing, the state a row enters a rank with (the spec picks from its head counts).
sequence_parallel_refusals is what a node on SP ranks may not declare; the engine and
validate_deployment both run it.
Functions
|
Every rank's q, k and v gathered whole in one collective, |
|
The shard plan a dependent of |
|
A causal conv over a split step's rows, every channel, on this rank's tokens |
|
Rank |
|
A copied step at a |
|
KV every SP rank stores whole (a sliding-window attention's layers of a pool stored by position): every rank's k and v gathered in one collective, and |
|
A striped label ( |
|
What |
|
How a split step reaches |
|
Copied queries over split keys: |
|
|
|
The state a row enters this rank with, from every rank's hand-off of the last row it holds: that piece's map from the state it starts with, |
|
Set |
|
Split tokens to split heads: one all-to-all of each of |
Classes
|
One rank's conv windows for a split step ( |
|
One label's split over the SP ranks: each rank's count of its tokens (of each row's, for a |
|
A step's split: whether its tokens split, and each split label's ranges. |
|
A node's steps split over its SP ranks, or copied. |
|
Plans each step split or copied ( |
- class mstar.engine.resources.sequence_shard.ConvHalo(tails, initial, final, rows, spans)[source]#
Bases:
NamedTupleOne rank’s conv windows for a split step (
conv_halo), as indices.tails([R, W - 1]into this rank’s tokens, the zero row past them where a piece is shorter) is each row’s last inputs here, right-aligned;initial([L, W - 1]) the window each row with a piece here starts from, andfinal([R, W - 1]) the window each row ends with, both into the rows’ stored windows followed by every rank’s tails.rows([L]) are the rows with a piece here andspanstheir pieces, which the local conv runs over. The three indices are host arrays as planned (conv_halo_plan), and on the device the conv runs on once uploaded (to).- Parameters:
- rows: ndarray#
Alias for field number 3
- spans: ndarray#
Alias for field number 4
- class mstar.engine.resources.sequence_shard.SeqRange(group, counts, rows, order=None)[source]#
Bases:
NamedTupleOne label’s split over the SP ranks: each rank’s count of its tokens (of each row’s, for a
[B, S, ...]layout), and which they are: contiguous in rank order, ororder.- group: CommGroup#
Alias for field number 0
- local(x, dim=0)[source]#
This rank’s share of
x(the whole label) alongdim: a view of a contiguous range, which a captured graph reads in place; a copy of a striped one, as a kernel takes it.
- property local_tokens: int#
The tokens this rank holds of the label, as a packed tensor’s first dim.
- row_pieces(spans)[source]#
Each rank’s piece of each row of a packed label laid out by
spans:[P, R]token counts, a row’s pieces in rank order summing to its span. Contiguous ranges only.
- class mstar.engine.resources.sequence_shard.SeqShardPlan(split, ranges)[source]#
Bases:
NamedTupleA step’s split: whether its tokens split, and each split label’s ranges.
- class mstar.engine.resources.sequence_shard.SeqShardSpec(resource_key, nodes, labels, axis, walks=None, *, min_tokens=0, exchange='auto', exchanges=True)[source]#
Bases:
NodeResourceSpecA node’s steps split over its SP ranks, or copied.
labelsare the step labels whose tokens split.axis0 splits a label’s packed tokens into contiguous ranges;axis1 splits each row of a[B, S, ...]layout alike (every row of a label one length).walkslists the walks that may split (None: every walk); the others plan copied. A step splits only when it is not one token a row, its listed tokens are at leastmax(sp, min_tokens), and every listed label has at leastsptokens (a row, foraxis1), so no rank gets an empty range.min_tokensis aresources:override, a floor above the one the node sets itself (below).exchange, also aresources:override, is how a split step’s tokens reach the node’s paged attentions, from the storage of the node’s pools (NodeResourceSpec.storage):auto(the default): Ulysses over KV stored by head; over KV stored by position (context_parallel) the node runs copied, so splitting there is the deployment’s to opt into;ulysses(by head),all_gatherorring(by position; the ring stripes each step by stored position): split;none: never split.
Where the node splits, a resource over state every rank stores whole takes its own exchange whatever this one is: a window attention gathers the step’s keys (
kv_gather), a recurrent layer the one its head counts allow. So does a ragged attention, by Ulysses wherever it plans a label this shard lists (RaggedAttentionSpec.labels); a shard that never splits stops them all splitting.exchanges=Falsedeclares that no op on the node mixes a listed label’s units (a video decoder’s independent clips): nothing exchanges them, so the node needs no attention, and a rank may get an empty range (fewer units than ranks still splits;gathertakes it).The attention and position resources on the node take this as a dependency in their own
derive. A model declares no step for it: its step is the node’s segments andcombined_labels(Resource.steps_from_segments), so a combined plan label gets a range over its sources too; at one SP rank it plans copied at no cost.Two fields are derived from
exchangeand the resources on its nodes (derive), from their hooks alone:splits, whether the node’s steps may split (not whereexchangeresolves tonone, or where no resource there takes split steps andexchangesis True);stripe_pool, the pool a ring attention reads, by whose stored positions each step is striped; andeager, whether a resource there splits its steps eagerly only (NodeResourceSpec.splits_eagerly: a ring,kv_gather, a recurrent layer). Such a node never splits a captured step: at load the shard sets, for each walk it splits, a floor one past that walk’s largest captured bucket (SequenceShard.check_captures), the same on every rank, so a step that fits a captured bucket runs copied in its graph and a bigger one runs eagerly and splits (min_tokensraises the floor, never lowers it).- Parameters:
- apply_yaml_overrides(min_tokens=None, exchange=None)[source]#
The step size below which a node runs copied (
min_tokens), a deployment’s trade between the exchanges’ cost and the token-local layers’ saving; and how a split step reaches its attentions (exchange).
- depends_on()[source]#
Keys whose specs this one builds against; the engine resolves them into
EngineResourceInfo.dependencies.
- derive(specs)[source]#
Set what this spec takes from the specs that read it (a KV pool keeps the window its attentions declare), after a deployment’s overrides; return what is wrong with it. Idempotent: it runs wherever the specs resolve (resolve_spec_dependencies).
- Parameters:
specs (Mapping[str, NodeResourceSpec])
- Return type:
- exchange: SpExchange | Literal['auto'] = 'auto'#
- exchange_over(by_position)[source]#
exchangefor KV stored by position or by head,autoresolved.- Parameters:
by_position (bool)
- Return type:
SpExchange
- node_exchange(specs)[source]#
exchangeover the storage of the pools on its nodes, from raw fields.- Parameters:
specs (Mapping[str, NodeResourceSpec])
- Return type:
SpExchange
- class mstar.engine.resources.sequence_shard.SequenceShard(labels, axis, walks, min_tokens, group, splits=True, stripe_pool=None, device=torch.device('cpu'), exchanges=True, eager=False)[source]#
Bases:
ResourcePlans each step split or copied (
SeqShardSpec) and brackets the layer stack withscatter/gather. Holds no per-request state and no device buffers: its plan is from the declared step, the lease and (striped) the pool’s plan, the same on every rank; a striped step’sorderis copied to the device with it. Off its defaults:prefix_skip_safe(it plans from the step a prefix hit already trimmed),supports_preplan(its dependents pre-plan),supports_context_parallel(a striped shard reads a placed pool’s plan),resident_refusal(None: a one-token-a-row step, what resident planning replays, is never split) and somax_repeats(any); it keepscan_captureTrue andforce_double_bufferFalse (a leased step, the only kind a pre-plan stages, never stripes: an eager node’s floors sit above its buckets,SeqShardSpec.eager).- Parameters:
- classmethod build(spec, info)[source]#
- Parameters:
spec (SeqShardSpec)
info (EngineResourceInfo)
- Return type:
- check_bypass(label, what, by_head=False)[source]#
Refuse a paged
what(run or a KV write) reached outside an exchange: of a split label it would see this rank’s share of the tokens as the whole step, and over a pool stored by head (by_head) any step’s q, k and v at the node’s TP heads where the pool holds this rank’stp * spslice.
- check_captures(node, configs, piecewise)[source]#
Refuse captures this node can’t replay when it splits: a piecewise region (its inner steps aren’t bracketed, and its replay resets padding rows, so
reset_request, inside the forward), a graph replayed under a walk that decides otherwise than the walk it was captured under, and a packed capture whose padding has no one label to join.On an eager node (
SeqShardSpec.eager) it also sets the floors: no captured step splits. Every rank sees the same configs, so the floors agree.
- check_scattered(label)[source]#
Refuse an exchange of split
labelbefore the layer stack scattered it. Checked against the label, which every rank shares, so every rank refuses alike before the exchange’s collectives (each rank’s token count can pass on one rank alone).- Parameters:
label (str)
- Return type:
None
- commit(step, ctx)[source]#
record step consumption
- Parameters:
step (ResourceStep)
ctx (StepContext)
- Return type:
None
- gather(label, *xs, dim=None, walk=None)[source]#
scatter’s inverse: every rank’s share of each ofxsback whole.
- max_repeats(step, ctx)[source]#
How many times this step may replay with no host work between replays, for a resource the step declares. One means it needs the host each time, which is what a resource says until it knows better: whatever it stages per step would go stale. Asked per step, and once at load, on a walk’s template step, to refuse a walk that could never repeat.
- Return type:
- may_split(walk)[source]#
Whether a step of
walkmay split (None: any walk), though_decidemay still plan it copied. In O(1) from what is fixed at bind, so a compiled frame specializes on its walk.
- plan(step, ctx)[source]#
ret is immutable and opaque to runner; only gives to ctx.plan_results
- Parameters:
step (ResourceStep)
ctx (StepContext)
- Return type:
- reset_request(rid, free=False)[source]#
Clear a dummy rid’s state: during accelerator graph capture, after each replayed step and when a pre-plan is discarded.
- resident_refusal()[source]#
Why a node holding this resource can’t run on resident tables, or None if it can.
- Return type:
str | None
- scatter(label, *xs, dim=None, walk=None)[source]#
This rank’s share of each of
xs(whole on every rank) alongdim(the spec’s axis by default), when this step splitslabel(SeqRange.local), contiguous; the inputs unchanged otherwise. One tensor in, one out; several in, a tuple out. None passes through.Contiguous because eager pointwise ops keep their first input’s layout: a view of a channel-major stream (Wan2.2’s patchified latents) would carry that layout through every block’s norms and residual adds.
walk, the step’s walk where the caller’s frame is compiled per walk: a walk the shard never splits (outsidewalks) traces straight through, as at one SP rank, rather than breaking the frame.
- split_range(label)[source]#
label’s ranges if this step splits it; None when copied, unlisted or outside a step.
- supports_drop = True#
- property supports_preplan#
- mstar.engine.resources.sequence_shard.all_gather(run, q, k, v, r)[source]#
Every rank’s q, k and v gathered whole in one collective,
runon them as on a copied step, then this rank’s share of the output.qNone: k and v alone, for the write.
- mstar.engine.resources.sequence_shard.check_shard_plan(owner, shard_key, ctx)[source]#
The shard plan a dependent of
shard_keyreads; raises if the step has none (it declares no segments).- Parameters:
owner (str)
shard_key (str)
ctx (StepContext)
- Return type:
- mstar.engine.resources.sequence_shard.conv_halo(conv, x, stored, halo, group)[source]#
A causal conv over a split step’s rows, every channel, on this rank’s tokens
x([n, C]): one all-gather of every rank’stails([P, R, W - 1, C]) gives each row with a piece here its window from the earlier ranks and the stored one (stored,[R, C, W - 1]in the pool’s dtype, zeros for a row without state), and every row’s final window, the same bytes on every rank.conv(x, initial)runs the conv overhalo.rowsfrominitial([L, C, W - 1]). Returns its output and the final windows[R, C, W - 1]. The conv kernels read both in the pool’s dtype, and so do the windows here.halo’s indices are onx’s device (ConvHalo.to).
- mstar.engine.resources.sequence_shard.conv_halo_plan(pieces, rank, taps)[source]#
Rank
rank’sConvHalofor rows splitpieces(SeqRange.row_pieces) under a conv keepingtapsinputs. The same tails, and the samefinal, on every rank.
- mstar.engine.resources.sequence_shard.head_slice(run, q, k, v, group)[source]#
A copied step at a
tp * sphead geometry: this rank’s block of q, k and v’s heads (the block an all-to-all routes it),run, and every rank’s heads of the output gathered.- Parameters:
run (Callable)
group (CommGroup)
- mstar.engine.resources.sequence_shard.kv_gather(run, q, k, v, r)[source]#
KV every SP rank stores whole (a sliding-window attention’s layers of a pool stored by position): every rank’s k and v gathered in one collective, and
runwrites the whole step and attends this rank’s own queries, which the resource planned over each row’s keys up to its own piece (SeqRange.row_pieces).
- mstar.engine.resources.sequence_shard.ring(run, q, k, v, r)[source]#
A striped label (
SeqRange.order): this rank holds the tokens whose K/V it stores, so nothing moves here.runwrites them into its own slots and attends its queries, which the plan passes round the partners (RingAttentionWrapper).
- mstar.engine.resources.sequence_shard.sequence_parallel_refusals(specs, node, tp, sp, moe_parallel=None)[source]#
What
node, ontpTP andspSP ranks, declares that its SP ranks can’t run; each a sentence. Each resource answers for itself (NodeResourceSpec.split_step_refusals); the checks on built objects (supports_split_steps,check_captures) are the engine’s.
- mstar.engine.resources.sequence_shard.shard_exchange(spec, specs)[source]#
How a split step reaches
spec’s node: itsSeqShardSpec.node_exchange, from raw fields;nonewithout one shard (two are refused,take_shard_edge).- Parameters:
spec (NodeResourceSpec)
specs (Mapping[str, NodeResourceSpec])
- Return type:
SpExchange
- mstar.engine.resources.sequence_shard.sharded_key_merge(run, q, k, v, r)[source]#
Copied queries over split keys:
run(which returns(out, lse)) over this rank’s keys, and the partials merged by log-sum-exp after one gather (gather_merge).
- mstar.engine.resources.sequence_shard.shards_over(spec, specs)[source]#
- Parameters:
spec (NodeResourceSpec)
specs (Mapping[str, NodeResourceSpec])
- Return type:
- mstar.engine.resources.sequence_shard.state_passing(group, transition, offset, compose)[source]#
The state a row enters this rank with, from every rank’s hand-off of the last row it holds: that piece’s map from the state it starts with,
X -> compose(X, transition) + offset. One fixed-size fp32 all-gather of every rank’s record, then zeros composed through ranks0 .. rank - 1in order. A rank where the row starts hands off a zero transition, so the ranks before it drop out; one whose last row ends there, the identity and zeros.
- mstar.engine.resources.sequence_shard.take_shard_edge(spec, specs)[source]#
Set
spec.shard_keyto theSeqShardSpecover its nodes, if any (aderive). A shard sharing only some ofspec’s nodes is a problem: the spec would split on some of its nodes and not others.- Parameters:
spec (NodeResourceSpec)
specs (Mapping[str, NodeResourceSpec])
- Return type:
- mstar.engine.resources.sequence_shard.ulysses(run, *xs, r)[source]#
Split tokens to split heads: one all-to-all of each of
xs([n, H, ...], this rank’s tokens, heads on dim 1; None passed through) to[N, H / P, ...], one exchange per dtype,runon them, and its output back by token. None whererunreturns None (a K/V write alone). Attention passes q, k and v; a recurrent layer its projections and gates.Eager, as main’s exchange is (CommGroup.all_to_all): a traced variant measured no faster.