mstar.engine.resources.sequence_shard

Contents

mstar.engine.resources.sequence_shard#

Sequence parallelism: whether a step’s tokens split over a node’s SP ranks, and how attention reaches the tokens it needs when they do.

A node’s SP ranks share each step one of two ways. Its tokens are split (each rank runs the token-local layers on its share) or copied (every rank runs them all). The sequence shard resource (SeqShardSpec) decides which, per step, from the step the model declares, and gives each listed label its per-rank ranges (SeqRange): contiguous, or on a node whose attention runs the ring, striped by KV position. A model brackets its layer stack with scatter / gather on that resource; nothing else in model code changes.

The attention resources over the node depend on it (shard_key, taken in their specs’ derive) and pick an exchange per label from its plan: for a split label, the shard’s exchange over their storage; for a copied one, what attends it at the node’s head geometry. The exchanges are the functions at the bottom of this module, each wrapped around the resource’s own run:

  • ulysses: one all-to-all from tokens to heads, run, and one back (head-stored KV, a cacheless attention planning a label the shard lists, RaggedAttentionSpec.labels, or a recurrent layer’s prompt rows);

  • all_gather: every rank’s q, k and v gathered, run as on a copied step, this rank’s share of the output (KV stored by position, context_parallel);

  • ring: a striped label’s own tokens, whose K/V this rank stores, run on them with their indices in the step; the plan passes queries round the partners (KV stored by position, RingAttentionWrapper);

  • head_slice: a copied step at a tp * sp head geometry: this rank’s heads, run, gather the heads;

  • sharded_key_merge: copied queries over split keys (cacheless cross-attention): each rank’s partial with its log-sum-exp, merged;

  • kv_gather: every rank’s k and v gathered, run writes them and attends this rank’s queries (KV every rank stores whole: a sliding-window attention’s layers of a pool stored by position).

A recurrent layer’s resource takes its conv’s left context from the earlier ranks (conv_halo), and its prompt rows through ulysses where its heads split over the SP ranks or else state_passing, the state a row enters a rank with (the spec picks from its head counts).

sequence_parallel_refusals is what a node on SP ranks may not declare; the engine and validate_deployment both run it.

Functions

all_gather(run, q, k, v, r)

Every rank's q, k and v gathered whole in one collective, run on them as on a copied step, then this rank's share of the output.

check_shard_plan(owner, shard_key, ctx)

The shard plan a dependent of shard_key reads; raises if the step has none (it declares no segments).

conv_halo(conv, x, stored, halo, group)

A causal conv over a split step's rows, every channel, on this rank's tokens x ([n, C]): one all-gather of every rank's tails ([P, R, W - 1, C]) gives each row with a piece here its window from the earlier ranks and the stored one (stored, [R, C, W - 1] in the pool's dtype, zeros for a row without state), and every row's final window, the same bytes on every rank.

conv_halo_plan(pieces, rank, taps)

Rank rank's ConvHalo for rows split pieces (SeqRange.row_pieces) under a conv keeping taps inputs.

head_slice(run, q, k, v, group)

A copied step at a tp * sp head geometry: this rank's block of q, k and v's heads (the block an all-to-all routes it), run, and every rank's heads of the output gathered.

kv_gather(run, q, k, v, r)

KV every SP rank stores whole (a sliding-window attention's layers of a pool stored by position): every rank's k and v gathered in one collective, and run writes the whole step and attends this rank's own queries, which the resource planned over each row's keys up to its own piece (SeqRange.row_pieces).

ring(run, q, k, v, r)

A striped label (SeqRange.order): this rank holds the tokens whose K/V it stores, so nothing moves here.

sequence_parallel_refusals(specs, node, tp, sp)

What node, on tp TP and sp SP ranks, declares that its SP ranks can't run; each a sentence.

shard_exchange(spec, specs)

How a split step reaches spec's node: its SeqShardSpec.node_exchange, from raw fields; none without one shard (two are refused, take_shard_edge).

sharded_key_merge(run, q, k, v, r)

Copied queries over split keys: run (which returns (out, lse)) over this rank's keys, and the partials merged by log-sum-exp after one gather (gather_merge).

shards_over(spec, specs)

state_passing(group, transition, offset, compose)

The state a row enters this rank with, from every rank's hand-off of the last row it holds: that piece's map from the state it starts with, X -> compose(X, transition) + offset.

take_shard_edge(spec, specs)

Set spec.shard_key to the SeqShardSpec over its nodes, if any (a derive).

ulysses(run, *xs, r)

Split tokens to split heads: one all-to-all of each of xs ([n, H, ...], this rank's tokens, heads on dim 1; None passed through) to [N, H / P, ...], one exchange per dtype, run on them, and its output back by token.

Classes

ConvHalo(tails, initial, final, rows, spans)

One rank's conv windows for a split step (conv_halo), as indices.

SeqRange(group, counts, rows[, order])

One label's split over the SP ranks: each rank's count of its tokens (of each row's, for a [B, S, ...] layout), and which they are: contiguous in rank order, or order.

SeqShardPlan(split, ranges)

A step's split: whether its tokens split, and each split label's ranges.

SeqShardSpec(resource_key, nodes, labels, axis)

A node's steps split over its SP ranks, or copied.

SequenceShard(labels, axis, walks, ...[, ...])

Plans each step split or copied (SeqShardSpec) and brackets the layer stack with scatter / gather.

class mstar.engine.resources.sequence_shard.ConvHalo(tails, initial, final, rows, spans)[source]#

Bases: NamedTuple

One rank’s conv windows for a split step (conv_halo), as indices. tails ([R, W - 1] into this rank’s tokens, the zero row past them where a piece is shorter) is each row’s last inputs here, right-aligned; initial ([L, W - 1]) the window each row with a piece here starts from, and final ([R, W - 1]) the window each row ends with, both into the rows’ stored windows followed by every rank’s tails. rows ([L]) are the rows with a piece here and spans their pieces, which the local conv runs over. The three indices are host arrays as planned (conv_halo_plan), and on the device the conv runs on once uploaded (to).

Parameters:
  • tails (ndarray | Tensor)

  • initial (ndarray | Tensor)

  • final (ndarray | Tensor)

  • rows (ndarray)

  • spans (ndarray)

final: ndarray | Tensor#

Alias for field number 2

initial: ndarray | Tensor#

Alias for field number 1

rows: ndarray#

Alias for field number 3

spans: ndarray#

Alias for field number 4

tails: ndarray | Tensor#

Alias for field number 0

to(device)[source]#

The indices on device, one non-blocking upload each, for conv_halo.

Parameters:

device (device)

Return type:

ConvHalo

class mstar.engine.resources.sequence_shard.SeqRange(group, counts, rows, order=None)[source]#

Bases: NamedTuple

One label’s split over the SP ranks: each rank’s count of its tokens (of each row’s, for a [B, S, ...] layout), and which they are: contiguous in rank order, or order.

Parameters:
property count: int#

Return number of occurrences of value.

counts: tuple[int, ...]#

Alias for field number 1

group: CommGroup#

Alias for field number 0

property index: Tensor#

this rank’s tokens’ indices in the step.

Type:

Striped

local(x, dim=0)[source]#

This rank’s share of x (the whole label) along dim: a view of a contiguous range, which a captured graph reads in place; a copy of a striped one, as a kernel takes it.

Parameters:
Return type:

Tensor

property local_tokens: int#

The tokens this rank holds of the label, as a packed tensor’s first dim.

order: Tensor | None#

Alias for field number 3

row_pieces(spans)[source]#

Each rank’s piece of each row of a packed label laid out by spans: [P, R] token counts, a row’s pieces in rank order summing to its span. Contiguous ranges only.

Parameters:

spans (Sequence[int])

Return type:

ndarray

rows: int | None#

Alias for field number 2

property start: int#

Where this rank’s share begins among the label’s tokens (of each row’s), contiguous ranges in rank order; a striped one’s are index.

whole(x, dim=0)[source]#

local’s inverse: every rank’s share of x gathered, in step order.

Parameters:
Return type:

Tensor

class mstar.engine.resources.sequence_shard.SeqShardPlan(split, ranges)[source]#

Bases: NamedTuple

A step’s split: whether its tokens split, and each split label’s ranges.

Parameters:
ranges: Mapping[str, SeqRange]#

Alias for field number 1

split: bool#

Alias for field number 0

class mstar.engine.resources.sequence_shard.SeqShardSpec(resource_key, nodes, labels, axis, walks=None, *, min_tokens=0, exchange='auto', exchanges=True)[source]#

Bases: NodeResourceSpec

A node’s steps split over its SP ranks, or copied.

labels are the step labels whose tokens split. axis 0 splits a label’s packed tokens into contiguous ranges; axis 1 splits each row of a [B, S, ...] layout alike (every row of a label one length). walks lists the walks that may split (None: every walk); the others plan copied. A step splits only when it is not one token a row, its listed tokens are at least max(sp, min_tokens), and every listed label has at least sp tokens (a row, for axis 1), so no rank gets an empty range. min_tokens is a resources: override, a floor above the one the node sets itself (below).

exchange, also a resources: override, is how a split step’s tokens reach the node’s paged attentions, from the storage of the node’s pools (NodeResourceSpec.storage):

  • auto (the default): Ulysses over KV stored by head; over KV stored by position (context_parallel) the node runs copied, so splitting there is the deployment’s to opt into;

  • ulysses (by head), all_gather or ring (by position; the ring stripes each step by stored position): split;

  • none: never split.

Where the node splits, a resource over state every rank stores whole takes its own exchange whatever this one is: a window attention gathers the step’s keys (kv_gather), a recurrent layer the one its head counts allow. So does a ragged attention, by Ulysses wherever it plans a label this shard lists (RaggedAttentionSpec.labels); a shard that never splits stops them all splitting.

exchanges=False declares that no op on the node mixes a listed label’s units (a video decoder’s independent clips): nothing exchanges them, so the node needs no attention, and a rank may get an empty range (fewer units than ranks still splits; gather takes it).

The attention and position resources on the node take this as a dependency in their own derive. A model declares no step for it: its step is the node’s segments and combined_labels (Resource.steps_from_segments), so a combined plan label gets a range over its sources too; at one SP rank it plans copied at no cost.

Two fields are derived from exchange and the resources on its nodes (derive), from their hooks alone: splits, whether the node’s steps may split (not where exchange resolves to none, or where no resource there takes split steps and exchanges is True); stripe_pool, the pool a ring attention reads, by whose stored positions each step is striped; and eager, whether a resource there splits its steps eagerly only (NodeResourceSpec.splits_eagerly: a ring, kv_gather, a recurrent layer). Such a node never splits a captured step: at load the shard sets, for each walk it splits, a floor one past that walk’s largest captured bucket (SequenceShard.check_captures), the same on every rank, so a step that fits a captured bucket runs copied in its graph and a bigger one runs eagerly and splits (min_tokens raises the floor, never lowers it).

Parameters:
  • resource_key (str)

  • nodes (set[str])

  • labels (tuple[str, ...])

  • axis (int)

  • walks (Collection[str] | None)

  • min_tokens (int)

  • exchange (SpExchange | Literal['auto'])

  • exchanges (bool)

apply_yaml_overrides(min_tokens=None, exchange=None)[source]#

The step size below which a node runs copied (min_tokens), a deployment’s trade between the exchanges’ cost and the token-local layers’ saving; and how a split step reaches its attentions (exchange).

Parameters:
  • min_tokens (int | None)

  • exchange (str | None)

axis: int#
depends_on()[source]#

Keys whose specs this one builds against; the engine resolves them into EngineResourceInfo.dependencies.

Return type:

set[str]

derive(specs)[source]#

Set what this spec takes from the specs that read it (a KV pool keeps the window its attentions declare), after a deployment’s overrides; return what is wrong with it. Idempotent: it runs wherever the specs resolve (resolve_spec_dependencies).

Parameters:

specs (Mapping[str, NodeResourceSpec])

Return type:

list[str]

eager: bool = False#
exchange: SpExchange | Literal['auto'] = 'auto'#
exchange_over(by_position)[source]#

exchange for KV stored by position or by head, auto resolved.

Parameters:

by_position (bool)

Return type:

SpExchange

exchanges: bool = True#
labels: tuple[str, ...]#
min_tokens: int = 0#
node_exchange(specs)[source]#

exchange over the storage of the pools on its nodes, from raw fields.

Parameters:

specs (Mapping[str, NodeResourceSpec])

Return type:

SpExchange

property resource_class: type[Resource]#

What builds this spec. Imported inside the property, so declaring a resource stays free of the manager and its kernels.

The builder, not necessarily the class built: an attention spec names AttentionManager, whose build picks a backend subclass.

splits: bool = True#
stripe_pool: str | None = None#
walks: Collection[str] | None = None#
class mstar.engine.resources.sequence_shard.SequenceShard(labels, axis, walks, min_tokens, group, splits=True, stripe_pool=None, device=torch.device('cpu'), exchanges=True, eager=False)[source]#

Bases: Resource

Plans each step split or copied (SeqShardSpec) and brackets the layer stack with scatter / gather. Holds no per-request state and no device buffers: its plan is from the declared step, the lease and (striped) the pool’s plan, the same on every rank; a striped step’s order is copied to the device with it. Off its defaults: prefix_skip_safe (it plans from the step a prefix hit already trimmed), supports_preplan (its dependents pre-plan), supports_context_parallel (a striped shard reads a placed pool’s plan), resident_refusal (None: a one-token-a-row step, what resident planning replays, is never split) and so max_repeats (any); it keeps can_capture True and force_double_buffer False (a leased step, the only kind a pre-plan stages, never stripes: an eager node’s floors sit above its buckets, SeqShardSpec.eager).

Parameters:
classmethod build(spec, info)[source]#
Parameters:
Return type:

SequenceShard

check_bypass(label, what, by_head=False)[source]#

Refuse a paged what (run or a KV write) reached outside an exchange: of a split label it would see this rank’s share of the tokens as the whole step, and over a pool stored by head (by_head) any step’s q, k and v at the node’s TP heads where the pool holds this rank’s tp * sp slice.

Parameters:
Return type:

None

check_captures(node, configs, piecewise)[source]#

Refuse captures this node can’t replay when it splits: a piecewise region (its inner steps aren’t bracketed, and its replay resets padding rows, so reset_request, inside the forward), a graph replayed under a walk that decides otherwise than the walk it was captured under, and a packed capture whose padding has no one label to join.

On an eager node (SeqShardSpec.eager) it also sets the floors: no captured step splits. Every rank sees the same configs, so the floors agree.

Parameters:
Return type:

None

check_scattered(label)[source]#

Refuse an exchange of split label before the layer stack scattered it. Checked against the label, which every rank shares, so every rank refuses alike before the exchange’s collectives (each rank’s token count can pass on one rank alone).

Parameters:

label (str)

Return type:

None

clear_preplan()[source]#
commit(step, ctx)[source]#

record step consumption

Parameters:
Return type:

None

depends_on()[source]#
Return type:

set[str]

floors: dict[str, int]#
gather(label, *xs, dim=None, walk=None)[source]#

scatter’s inverse: every rank’s share of each of xs back whole.

Parameters:
max_repeats(step, ctx)[source]#

How many times this step may replay with no host work between replays, for a resource the step declares. One means it needs the host each time, which is what a resource says until it knows better: whatever it stages per step would go stale. Asked per step, and once at load, on a walk’s template step, to refuse a walk that could never repeat.

Return type:

int

may_split(walk)[source]#

Whether a step of walk may split (None: any walk), though _decide may still plan it copied. In O(1) from what is fixed at bind, so a compiled frame specializes on its walk.

Parameters:

walk (str | None)

Return type:

bool

plan(step, ctx)[source]#

ret is immutable and opaque to runner; only gives to ctx.plan_results

Parameters:
Return type:

SeqShardPlan

prefix_skip_safe: bool = True#
reset_request(rid, free=False)[source]#

Clear a dummy rid’s state: during accelerator graph capture, after each replayed step and when a pre-plan is discarded.

Parameters:
Return type:

None

resident_refusal()[source]#

Why a node holding this resource can’t run on resident tables, or None if it can.

Return type:

str | None

scatter(label, *xs, dim=None, walk=None)[source]#

This rank’s share of each of xs (whole on every rank) along dim (the spec’s axis by default), when this step splits label (SeqRange.local), contiguous; the inputs unchanged otherwise. One tensor in, one out; several in, a tuple out. None passes through.

Contiguous because eager pointwise ops keep their first input’s layout: a view of a channel-major stream (Wan2.2’s patchified latents) would carry that layout through every block’s norms and residual adds.

walk, the step’s walk where the caller’s frame is compiled per walk: a walk the shard never splits (outside walks) traces straight through, as at one SP rank, rather than breaking the frame.

Parameters:
split_range(label)[source]#

label’s ranges if this step splits it; None when copied, unlisted or outside a step.

Parameters:

label (str)

Return type:

SeqRange | None

steps_from_segments: bool = True#
supports_context_parallel: bool = True#
supports_drop = True#
property supports_preplan#
supports_split_steps: bool = True#
mstar.engine.resources.sequence_shard.all_gather(run, q, k, v, r)[source]#

Every rank’s q, k and v gathered whole in one collective, run on them as on a copied step, then this rank’s share of the output. q None: k and v alone, for the write.

Parameters:
mstar.engine.resources.sequence_shard.check_shard_plan(owner, shard_key, ctx)[source]#

The shard plan a dependent of shard_key reads; raises if the step has none (it declares no segments).

Parameters:
Return type:

SeqShardPlan

mstar.engine.resources.sequence_shard.conv_halo(conv, x, stored, halo, group)[source]#

A causal conv over a split step’s rows, every channel, on this rank’s tokens x ([n, C]): one all-gather of every rank’s tails ([P, R, W - 1, C]) gives each row with a piece here its window from the earlier ranks and the stored one (stored, [R, C, W - 1] in the pool’s dtype, zeros for a row without state), and every row’s final window, the same bytes on every rank. conv(x, initial) runs the conv over halo.rows from initial ([L, C, W - 1]). Returns its output and the final windows [R, C, W - 1]. The conv kernels read both in the pool’s dtype, and so do the windows here. halo’s indices are on x’s device (ConvHalo.to).

Parameters:
mstar.engine.resources.sequence_shard.conv_halo_plan(pieces, rank, taps)[source]#

Rank rank’s ConvHalo for rows split pieces (SeqRange.row_pieces) under a conv keeping taps inputs. The same tails, and the same final, on every rank.

Parameters:
  • pieces (ndarray)

  • rank (int)

  • taps (int)

Return type:

ConvHalo

mstar.engine.resources.sequence_shard.head_slice(run, q, k, v, group)[source]#

A copied step at a tp * sp head geometry: this rank’s block of q, k and v’s heads (the block an all-to-all routes it), run, and every rank’s heads of the output gathered.

Parameters:
mstar.engine.resources.sequence_shard.kv_gather(run, q, k, v, r)[source]#

KV every SP rank stores whole (a sliding-window attention’s layers of a pool stored by position): every rank’s k and v gathered in one collective, and run writes the whole step and attends this rank’s own queries, which the resource planned over each row’s keys up to its own piece (SeqRange.row_pieces).

Parameters:
mstar.engine.resources.sequence_shard.ring(run, q, k, v, r)[source]#

A striped label (SeqRange.order): this rank holds the tokens whose K/V it stores, so nothing moves here. run writes them into its own slots and attends its queries, which the plan passes round the partners (RingAttentionWrapper).

Parameters:
mstar.engine.resources.sequence_shard.sequence_parallel_refusals(specs, node, tp, sp, moe_parallel=None)[source]#

What node, on tp TP and sp SP ranks, declares that its SP ranks can’t run; each a sentence. Each resource answers for itself (NodeResourceSpec.split_step_refusals); the checks on built objects (supports_split_steps, check_captures) are the engine’s.

Parameters:
Return type:

list[str]

mstar.engine.resources.sequence_shard.shard_exchange(spec, specs)[source]#

How a split step reaches spec’s node: its SeqShardSpec.node_exchange, from raw fields; none without one shard (two are refused, take_shard_edge).

Parameters:
Return type:

SpExchange

mstar.engine.resources.sequence_shard.sharded_key_merge(run, q, k, v, r)[source]#

Copied queries over split keys: run (which returns (out, lse)) over this rank’s keys, and the partials merged by log-sum-exp after one gather (gather_merge).

Parameters:
mstar.engine.resources.sequence_shard.shards_over(spec, specs)[source]#
Parameters:
Return type:

list[SeqShardSpec]

mstar.engine.resources.sequence_shard.state_passing(group, transition, offset, compose)[source]#

The state a row enters this rank with, from every rank’s hand-off of the last row it holds: that piece’s map from the state it starts with, X -> compose(X, transition) + offset. One fixed-size fp32 all-gather of every rank’s record, then zeros composed through ranks 0 .. rank - 1 in order. A rank where the row starts hands off a zero transition, so the ranks before it drop out; one whose last row ends there, the identity and zeros.

Parameters:
Return type:

Tensor

mstar.engine.resources.sequence_shard.take_shard_edge(spec, specs)[source]#

Set spec.shard_key to the SeqShardSpec over its nodes, if any (a derive). A shard sharing only some of spec’s nodes is a problem: the spec would split on some of its nodes and not others.

Parameters:
Return type:

list[str]

mstar.engine.resources.sequence_shard.ulysses(run, *xs, r)[source]#

Split tokens to split heads: one all-to-all of each of xs ([n, H, ...], this rank’s tokens, heads on dim 1; None passed through) to [N, H / P, ...], one exchange per dtype, run on them, and its output back by token. None where run returns None (a K/V write alone). Attention passes q, k and v; a recurrent layer its projections and gates.

Eager, as main’s exchange is (CommGroup.all_to_all): a traced variant measured no faster.

Parameters: