mstar.engine.resources.context_parallel#
Context parallelism: a paged KV pool’s token positions placed over a group of ranks that would otherwise hold the same KV heads.
Page ids stay logical and are the same on every rank, so allocation, prefix keys,
offload and admission need nothing new. Within a page, logical slot s belongs to
partner s % P and sits at its physical slot s // P, so each rank stores
page_size // P slots a page. PagedKVConfig.place resolves the placement. A pool that keeps
a sliding window places its first group of layers alone; the window’s stay whole on every partner
(PagedKVConfig.placed_groups).
Attention over such a pool (ContextParallelWrapper) runs on each rank over its own slots and
returns the partial with its log-sum-exp; the partners exchange partials and merge them,
exactly.
A step split over SP partners by stored position (stripe_owners) attends by passing queries
round the partners instead (RingAttentionWrapper): each rank attends every partner’s queries
over its own slots, and the partials go back to their queries’ rank.
Functions
|
A query replicated on every rank of |
|
A plan's logical paged indptrs as this rank's ( |
|
Attention over the union of disjoint key sets from each set's partial: each out |
|
|
|
|
|
A ring hop's |
|
Which of |
|
A zigzag share's kernel captured for a bucket of |
|
Rank |
Classes
|
This rank's place in its KV pool's placement: which of a page's logical slots it stores. |
|
A step's attention over a placed pool from a backend's two wrappers: each rank attends its own slots, and the partners exchange the partials and merge them, as its placement's subclass does ( |
|
Rows of a step's own queries, each over a list of the step's own tokens as its keys: what an |
|
An |
Attention of rows of a step's own queries over lists of its own tokens as keys ( |
|
|
A split step's attention over an SP-placed pool by striped pass-Q. |
|
Attention over an |
|
Placed over the node's SP group, whose ranks hold the same heads: the partners are the whole group, in rank order ( |
|
What the kernel of a captured step's own tokens is sized for. |
|
Attention over a |
|
Placed over the block of the node's TP ranks that hold one KV head, each with different query heads of it ( |
|
A step's own tokens under SP placement, attended for this rank's zigzag chunks alone on |
- class mstar.engine.resources.context_parallel.ContextParallel(size, index, group)[source]#
Bases:
objectThis rank’s place in its KV pool’s placement: which of a page’s logical slots it stores.
SPPlacementorTPPlacementsays over which ranks.- group: CommGroup#
- class mstar.engine.resources.context_parallel.ContextParallelWrapper(placement, page_size, dtype, context, new, device, batch_size=None, max_total_tokens=None)[source]#
Bases:
ABCA step’s attention over a placed pool from a backend’s two wrappers: each rank attends its own slots, and the partners exchange the partials and merge them, as its placement’s subclass does (
TPContextParallelWrapper,SPContextParallelWrapper). A one-token-a-row step attends the pages after its write (newNone); any other step attends its stored context (not causal) oncontextand its own tokens, from the keys the layer passes torun, onnew.page_sizeis the logical one.- Parameters:
placement (ContextParallel)
page_size (int)
dtype (dtype)
context (PagedKernel)
new (OwnTokensKernel | ZigzagWrapper | None)
device (device)
batch_size (int | None)
max_total_tokens (int | None)
- attend(q, kv_cache_layer, k, v)[source]#
Attention over the whole placed pool, in the query’s dtype (merged in fp32):
q[N, H, D], and the step’s own keyskandv(read only on a step that isn’t one token a row). Every rank runs the same collectives.
- forget_plan()[source]#
Replan the own-token wrapper on the next
planwhatever its layout, for a dropped pre-plan: a ragged one skips a layout it already holds (_RaggedPrefillBase.forget_plan).- Return type:
None
- static local_geometry(placement, page_size, num_qo_heads)[source]#
A paged attention wrapper’s
page_sizeandnum_qo_headsover this rank’s slots: the slots a page it stores, and its own query heads.
- abstractmethod classmethod own_tokens(kernels, placement, device, workspace, kwargs, max_total_tokens)[source]#
The part a multi-token step attends its own tokens on, from the backend’s
kernels(_plan_ownplans it,attendruns it).- Parameters:
kernels (PlacedAttention)
placement (ContextParallel)
device (device)
workspace (Tensor)
kwargs (dict)
max_total_tokens (int | None)
- Return type:
- plan(kv_out, causal=True, real_rows=None)[source]#
kv_out: the step’s logicalKVPlanOutputfor this label. The context attention reads its indptrs for a one-token-a-row step, its stored context alone otherwise.real_rows: a replay’s real requests when it has padding rows (StepContext.real_rows), which follow the real rows in each label. The own-token part then plans the real rows alone, so the padding’s queries come out 0, lse -inf, and its keys lie past the plan: FA3’s ragged kernel (FlashInfer’s TP own tokens) reads keys past a row’s end to its KV tile’s, whererunzeroes them, so a NaN the model’s layers left in a padding row’s keys would otherwise reach the real rows before it.
- class mstar.engine.resources.context_parallel.KeyRows(qo_indptr, kv_indptr, kv_tokens)[source]#
Bases:
NamedTupleRows of a step’s own queries, each over a list of the step’s own tokens as its keys: what an
OwnTokensKernelplans. Token ids, not pages.- Parameters:
qo_indptr (ndarray)
kv_indptr (ndarray)
kv_tokens (ndarray)
- kv_indptr: ndarray#
Alias for field number 1
- kv_tokens: ndarray#
Alias for field number 2
- qo_indptr: ndarray#
Alias for field number 0
- class mstar.engine.resources.context_parallel.OneSlotPages(kernel, dtype, captured=None)[source]#
Bases:
OwnTokensKernelAn
OwnTokensKernelon a backend’s paged kernel over pages of one slot, pageithe step’s tokeni: rows whose key lists overlap (a zigzag share’s two chunks) plan as rows of one call with no copy of the keys. Captured, it pads each plan as the paged planner’s fixed buffers read it: the spare queries one row, the rows left over empty, each over token 0. The padding row’s one key costs O(pad): under causal alignment all but its last query see none.- Parameters:
kernel (PagedKernel)
dtype (dtype)
captured (ShareCapture | None)
- class mstar.engine.resources.context_parallel.OwnTokensKernel[source]#
Bases:
ABCAttention of rows of a step’s own queries over lists of its own tokens as keys (
KeyRows), from the keys the layer passesrun: a placed pool’s own-token part, TP’s whole rows and SP’s zigzag share alike (PlacedAttention.own_tokens_kernelsupplies it). Nothing here assumes pages: a paged kernel reads the keys in place (OneSlotPages); a ragged one reads contiguous runs (FlashInfer’s for whole rows), or would gather the keys bykv_tokens.A captured kernel is run on
captured.queriesqueries every replay, however few its plan has (ZigzagWrappergathers that many; TP passes the bucket’s). A kernel that declareswhole_rows_onlyis refused where any other rows would be planned on it.- captured: ShareCapture | None = None#
- forget_plan()[source]#
Every
planplans the kernel, so there is nothing to forget.- Return type:
None
- abstractmethod plan(rows, causal)[source]#
Plans
rows;causalaligns each row’s queries to the end of its keys.
- class mstar.engine.resources.context_parallel.RingAttentionWrapper(placement, page_size, dtype, paged, device)[source]#
Bases:
objectA split step’s attention over an SP-placed pool by striped pass-Q. The step’s tokens are striped over the partners by KV position (
stripe_owners), so each rank holds the queries whose K/V it writes into its own slots. In roundia rank attends partner(r - i) % P’s queries over its slots, the stored context and the step’s own tokens together, with one call (_ring_rows); the partial goes back to that partner, which merges it into its running result (merge_lse).One causal call is exact because a partner’s queries in a row sit at KV positions
p0 + aP, and this rank’s keys at or belowpnumber(p - r) // P + 1: one more per query, so the kernel’s bottom-right causal mask over the local length to the row’s last query is the right one. Only a row’s first query can see no key here (p0 < r); it plans as a row of its own over one sink slot, masked out of the merge (_masked), and its own rank always holds a key of it. Not causal, each row’s queries see its whole length, and a row with no slot here is masked the same way. A row this partner holds no query of is left out of its plan.pagedis one eager kernel over this rank’s slots per partner, all planned per step. Eager only: the node never splits a captured step (SeqShardSpec.eager). The partner plans differ by rank and no admission bounds them: FlashInfer’s paged manager, the one backend that takes a placed pool (PlacedAttention), so the one that plans a ring, bounds no plan at admission, as no refusal of its planner on lengths is known. A backend whose planner may refuse on lengths (MLA’s FA planner) would have to bound every rank’s partner plans at admission before it could plan a ring, or one rank alone could refuse a lockstep step.- Parameters:
placement (SPPlacement)
page_size (int)
dtype (dtype)
paged (list[PagedKernel])
device (device)
- attend(q, kv_cache_layer, k, v)[source]#
This rank’s queries’ attention over the whole pool, in their dtype, as
ContextParallelWrapper.attendtakes them; the step’s own tokens are already written, sokandvare unread.2(P - 1)exchanges, the same on every rank, each awaited before its result is read.
- forget_plan()[source]#
Every
planplans every kernel, so there is nothing to forget.- Return type:
None
- plan(kv_out, counts, causal=True)[source]#
kv_out: the step’s logicalKVPlanOutputfor the label;counts: each partner’s queries, as the sequence shard striped them from the same plan (SeqRange.counts);causal: the step’s.
- reads_step_kv = False#
- class mstar.engine.resources.context_parallel.SPContextParallelWrapper(placement, page_size, dtype, context, new, device, batch_size=None, max_total_tokens=None)[source]#
Bases:
ContextParallelWrapperAttention over an
SPPlacement: the partners hold the same heads, so each attends its own slots and its zigzag share of the own tokens (ZigzagWrapper) for its own queries, merges the two, and one gather merges every partner’s.- Parameters:
placement (ContextParallel)
page_size (int)
dtype (dtype)
context (PagedKernel)
new (ZigzagWrapper | None)
device (device)
batch_size (int | None)
max_total_tokens (int | None)
- new: ZigzagWrapper | None#
- class mstar.engine.resources.context_parallel.SPPlacement(size, index, group)[source]#
Bases:
ContextParallelPlaced over the node’s SP group, whose ranks hold the same heads: the partners are the whole group, in rank order (
SPContextParallelWrapper).
Bases:
NamedTupleWhat the kernel of a captured step’s own tokens is sized for.
Alias for field number 1
Alias for field number 0
Alias for field number 2
- class mstar.engine.resources.context_parallel.TPContextParallelWrapper(placement, page_size, dtype, context, new, device, batch_size=None, max_total_tokens=None)[source]#
Bases:
ContextParallelWrapperAttention over a
TPPlacement: the partners hold different query heads of one KV head, so each attends its slots for every partner’s heads (_exchange_queries), sends each partner its heads’ partial back (_exchange_partials), and merges them with the own tokens, which it attends whole for its own heads.- Parameters:
placement (ContextParallel)
page_size (int)
dtype (dtype)
context (PagedKernel)
new (OwnTokensKernel | None)
device (device)
batch_size (int | None)
max_total_tokens (int | None)
- static local_geometry(placement, page_size, num_qo_heads)[source]#
And every partner’s query heads, which it attends for them.
- new: OwnTokensKernel | None#
- class mstar.engine.resources.context_parallel.TPPlacement(size, index, group, partners)[source]#
Bases:
ContextParallelPlaced over the block of the node’s TP ranks that hold one KV head, each with different query heads of it (
TPContextParallelWrapper). The exchange is an all-to-all over the whole TP group with nothing sent to the ranks outside the block, so no group of its own is needed.
- class mstar.engine.resources.context_parallel.ZigzagWrapper(placement, kernel, device)[source]#
Bases:
objectA step’s own tokens under SP placement, attended for this rank’s zigzag chunks alone on
kernel, out 0 and lse -inf off them. On a captured kernel it gathers the same number of queries every replay: its share, then padding, which gathers token 0 and scatters to a rowrundrops.- Parameters:
placement (SPPlacement)
kernel (OwnTokensKernel)
device (device)
- mstar.engine.resources.context_parallel.gather_merge(group, out, lse)[source]#
A query replicated on every rank of
groupattending keys sharded across them, from this rank’sout[..., H, D]and natural-loglse[..., H]over its share: one gather of both, thenmerge. Moves query-sized results rather than the keys.
- mstar.engine.resources.context_parallel.local_indptrs(placement, indptrs, page_size)[source]#
A plan’s logical paged indptrs as this rank’s (
page_sizethe logical one), and per row whether it holds any of this rank’s slots.Every row keeps a length of at least 1 and a last page holding at least one slot, as on one rank: FlashInfer’s split-KV planner counts a row’s pages where its kernel counts its tokens, and a last page with none of this rank’s slots puts the two out of step. Such a page is dropped from its row. A row with no slot here attends one slot of SINK_PAGE instead, and its partial is masked before the merge (
_masked).- Parameters:
placement (ContextParallel)
indptrs (PagedIndptrs)
page_size (int)
- Return type:
tuple[PagedIndptrs, ndarray]
- mstar.engine.resources.context_parallel.merge(outs, lses)[source]#
Attention over the union of disjoint key sets from each set’s partial: each out
[..., D](a[K, ..., D]stack is a sequence too) and their natural-log lses stacked,[K, ...]; fp32.
- mstar.engine.resources.context_parallel.merge_lse(outs, lses)[source]#
merge, and the union’s natural-log lse, fp32, to merge again.
- mstar.engine.resources.context_parallel.position_pools(specs, node)[source]#
node’s resources that hold token positions, each to whether it places them (NodeResourceSpec.storageby position or by head; a copied one goes with either). Raises when one is placed and another isn’t.
- mstar.engine.resources.context_parallel.ring_hop(size, rank, offset, send, recv)[source]#
A ring hop’s
all_to_all_rowscounts atoffset:sendrows to rankrank + offset,recvback from rankrank - offset, ofsize.
- mstar.engine.resources.context_parallel.stripe_owners(qo_indptr, stored, size)[source]#
Which of
sizepartners stores each of a step’s tokens: a row’s tokentgoes to KV positionstored + t(storedper row, the row’s KV before the step), whose slot partner(stored + t) % sizeholds, assizedivides the page.- Parameters:
qo_indptr (ndarray)
stored (ndarray)
size (int)
- Return type:
ndarray
- mstar.engine.resources.context_parallel.zigzag_capture(size, rows, tokens)[source]#
A zigzag share’s kernel captured for a bucket of
rowsrows andtokenstokens: its fixed rows (at most 2 chunks a row, and one more for the padding queries) and queries (at mostceil(T / 2P)a chunk).- Parameters:
- Return type:
- mstar.engine.resources.context_parallel.zigzag_rows(qo_indptr, size, index, causal=True)[source]#
Rank
indexofsize’s zigzag chunks of the rowsqo_indptrlays out (chunksrand2P - 1 - rof each row’s2P; see the context-parallel docs): the step’s tokens it attends, chunk by chunk, and its chunks as rows over their keys, each row’s first token to the chunk’s last (or the row’s, not causal).