mstar.engine.resources.context_parallel#

Context parallelism: a paged KV pool’s token positions placed over a group of ranks that would otherwise hold the same KV heads.

Page ids stay logical and are the same on every rank, so allocation, prefix keys, offload and admission need nothing new. Within a page, logical slot s belongs to partner s % P and sits at its physical slot s // P, so each rank stores page_size // P slots a page. PagedKVConfig.place resolves the placement. A pool that keeps a sliding window places its first group of layers alone; the window’s stay whole on every partner (PagedKVConfig.placed_groups).

Attention over such a pool (ContextParallelWrapper) runs on each rank over its own slots and returns the partial with its log-sum-exp; the partners exchange partials and merge them, exactly.

A step split over SP partners by stored position (stripe_owners) attends by passing queries round the partners instead (RingAttentionWrapper): each rank attends every partner’s queries over its own slots, and the partials go back to their queries’ rank.

Functions

gather_merge(group, out, lse)

A query replicated on every rank of group attending keys sharded across them, from this rank's out [..., H, D] and natural-log lse [..., H] over its share: one gather of both, then merge.

local_indptrs(placement, indptrs, page_size)

A plan's logical paged indptrs as this rank's (page_size the logical one), and per row whether it holds any of this rank's slots.

merge(outs, lses)

Attention over the union of disjoint key sets from each set's partial: each out [..., D] (a [K, ..., D] stack is a sequence too) and their natural-log lses stacked, [K, ...]; fp32.

merge_lse(outs, lses)

merge, and the union's natural-log lse, fp32, to merge again.

position_pools(specs, node)

node's resources that hold token positions, each to whether it places them (NodeResourceSpec.storage by position or by head; a copied one goes with either).

ring_hop(size, rank, offset, send, recv)

A ring hop's all_to_all_rows counts at offset: send rows to rank rank + offset, recv back from rank rank - offset, of size.

stripe_owners(qo_indptr, stored, size)

Which of size partners stores each of a step's tokens: a row's token t goes to KV position stored + t (stored per row, the row's KV before the step), whose slot partner (stored + t) % size holds, as size divides the page.

zigzag_capture(size, rows, tokens)

A zigzag share's kernel captured for a bucket of rows rows and tokens tokens: its fixed rows (at most 2 chunks a row, and one more for the padding queries) and queries (at most ceil(T / 2P) a chunk).

zigzag_rows(qo_indptr, size, index[, causal])

Rank index of size's zigzag chunks of the rows qo_indptr lays out (chunks r and 2P - 1 - r of each row's 2P; see the context-parallel docs): the step's tokens it attends, chunk by chunk, and its chunks as rows over their keys, each row's first token to the chunk's last (or the row's, not causal).

Classes

ContextParallel(size, index, group)

This rank's place in its KV pool's placement: which of a page's logical slots it stores.

ContextParallelWrapper(placement, page_size, ...)

A step's attention over a placed pool from a backend's two wrappers: each rank attends its own slots, and the partners exchange the partials and merge them, as its placement's subclass does (TPContextParallelWrapper, SPContextParallelWrapper).

KeyRows(qo_indptr, kv_indptr, kv_tokens)

Rows of a step's own queries, each over a list of the step's own tokens as its keys: what an OwnTokensKernel plans.

OneSlotPages(kernel, dtype[, captured])

An OwnTokensKernel on a backend's paged kernel over pages of one slot, page i the step's token i: rows whose key lists overlap (a zigzag share's two chunks) plan as rows of one call with no copy of the keys.

OwnTokensKernel()

Attention of rows of a step's own queries over lists of its own tokens as keys (KeyRows), from the keys the layer passes run: a placed pool's own-token part, TP's whole rows and SP's zigzag share alike (PlacedAttention.own_tokens_kernel supplies it).

RingAttentionWrapper(placement, page_size, ...)

A split step's attention over an SP-placed pool by striped pass-Q.

SPContextParallelWrapper(placement, ...[, ...])

Attention over an SPPlacement: the partners hold the same heads, so each attends its own slots and its zigzag share of the own tokens (ZigzagWrapper) for its own queries, merges the two, and one gather merges every partner's.

SPPlacement(size, index, group)

Placed over the node's SP group, whose ranks hold the same heads: the partners are the whole group, in rank order (SPContextParallelWrapper).

ShareCapture(rows, queries, tokens)

What the kernel of a captured step's own tokens is sized for.

TPContextParallelWrapper(placement, ...[, ...])

Attention over a TPPlacement: the partners hold different query heads of one KV head, so each attends its slots for every partner's heads (_exchange_queries), sends each partner its heads' partial back (_exchange_partials), and merges them with the own tokens, which it attends whole for its own heads.

TPPlacement(size, index, group, partners)

Placed over the block of the node's TP ranks that hold one KV head, each with different query heads of it (TPContextParallelWrapper).

ZigzagWrapper(placement, kernel, device)

A step's own tokens under SP placement, attended for this rank's zigzag chunks alone on kernel, out 0 and lse -inf off them.

class mstar.engine.resources.context_parallel.ContextParallel(size, index, group)[source]#

Bases: object

This rank’s place in its KV pool’s placement: which of a page’s logical slots it stores. SPPlacement or TPPlacement says over which ranks.

Parameters:
  • size (int)

  • index (int)

  • group (CommGroup)

group: CommGroup#
index: int#
size: int#
slot_range(start, end)[source]#

The physical slots of a page holding this rank’s share of logical slots [start, end).

Parameters:
Return type:

tuple[int, int]

write_slots(pages, offsets)[source]#

Each token’s write address on this rank, from its logical (page, offset): an owned token goes to (page, offset // P), the rest to (SINK_PAGE, 0). Device tensors stay on the device; anything else comes back as numpy arrays.

class mstar.engine.resources.context_parallel.ContextParallelWrapper(placement, page_size, dtype, context, new, device, batch_size=None, max_total_tokens=None)[source]#

Bases: ABC

A step’s attention over a placed pool from a backend’s two wrappers: each rank attends its own slots, and the partners exchange the partials and merge them, as its placement’s subclass does (TPContextParallelWrapper, SPContextParallelWrapper). A one-token-a-row step attends the pages after its write (new None); any other step attends its stored context (not causal) on context and its own tokens, from the keys the layer passes to run, on new. page_size is the logical one.

Parameters:
attend(q, kv_cache_layer, k, v)[source]#

Attention over the whole placed pool, in the query’s dtype (merged in fp32): q [N, H, D], and the step’s own keys k and v (read only on a step that isn’t one token a row). Every rank runs the same collectives.

Parameters:
Return type:

Tensor

forget_plan()[source]#

Replan the own-token wrapper on the next plan whatever its layout, for a dropped pre-plan: a ragged one skips a layout it already holds (_RaggedPrefillBase.forget_plan).

Return type:

None

static local_geometry(placement, page_size, num_qo_heads)[source]#

A paged attention wrapper’s page_size and num_qo_heads over this rank’s slots: the slots a page it stores, and its own query heads.

Parameters:
Return type:

dict[str, int]

abstractmethod classmethod own_tokens(kernels, placement, device, workspace, kwargs, max_total_tokens)[source]#

The part a multi-token step attends its own tokens on, from the backend’s kernels (_plan_own plans it, attend runs it).

Parameters:
Return type:

OwnTokensKernel | ZigzagWrapper

plan(kv_out, causal=True, real_rows=None)[source]#

kv_out: the step’s logical KVPlanOutput for this label. The context attention reads its indptrs for a one-token-a-row step, its stored context alone otherwise.

real_rows: a replay’s real requests when it has padding rows (StepContext.real_rows), which follow the real rows in each label. The own-token part then plans the real rows alone, so the padding’s queries come out 0, lse -inf, and its keys lie past the plan: FA3’s ragged kernel (FlashInfer’s TP own tokens) reads keys past a row’s end to its KV tile’s, where run zeroes them, so a NaN the model’s layers left in a padding row’s keys would otherwise reach the real rows before it.

Parameters:
property reads_step_kv: bool#

Whether the step attends the keys run is passed (PlacedAttention.reads_step_kv): any step but one token a row.

class mstar.engine.resources.context_parallel.KeyRows(qo_indptr, kv_indptr, kv_tokens)[source]#

Bases: NamedTuple

Rows of a step’s own queries, each over a list of the step’s own tokens as its keys: what an OwnTokensKernel plans. Token ids, not pages.

Parameters:
  • qo_indptr (ndarray)

  • kv_indptr (ndarray)

  • kv_tokens (ndarray)

kv_indptr: ndarray#

Alias for field number 1

kv_tokens: ndarray#

Alias for field number 2

qo_indptr: ndarray#

Alias for field number 0

classmethod whole(qo_indptr)[source]#

Each row of qo_indptr over its own tokens: TP placement’s own tokens.

Parameters:

qo_indptr (ndarray)

Return type:

KeyRows

class mstar.engine.resources.context_parallel.OneSlotPages(kernel, dtype, captured=None)[source]#

Bases: OwnTokensKernel

An OwnTokensKernel on a backend’s paged kernel over pages of one slot, page i the step’s token i: rows whose key lists overlap (a zigzag share’s two chunks) plan as rows of one call with no copy of the keys. Captured, it pads each plan as the paged planner’s fixed buffers read it: the spare queries one row, the rows left over empty, each over token 0. The padding row’s one key costs O(pad): under causal alignment all but its last query see none.

Parameters:
plan(rows, causal)[source]#

Plans rows; causal aligns each row’s queries to the end of its keys.

Parameters:
Return type:

None

run(q, k, v)[source]#

(out, lse) (lse natural-log fp32) of the planned rows’ queries q, in plan order, over the step’s keys k and v. A row for every query passed; one past the plan (a captured bucket’s padding) holds whatever the kernel left, which reaches no real row.

class mstar.engine.resources.context_parallel.OwnTokensKernel[source]#

Bases: ABC

Attention of rows of a step’s own queries over lists of its own tokens as keys (KeyRows), from the keys the layer passes run: a placed pool’s own-token part, TP’s whole rows and SP’s zigzag share alike (PlacedAttention.own_tokens_kernel supplies it). Nothing here assumes pages: a paged kernel reads the keys in place (OneSlotPages); a ragged one reads contiguous runs (FlashInfer’s for whole rows), or would gather the keys by kv_tokens.

A captured kernel is run on captured.queries queries every replay, however few its plan has (ZigzagWrapper gathers that many; TP passes the bucket’s). A kernel that declares whole_rows_only is refused where any other rows would be planned on it.

captured: ShareCapture | None = None#
forget_plan()[source]#

Every plan plans the kernel, so there is nothing to forget.

Return type:

None

abstractmethod plan(rows, causal)[source]#

Plans rows; causal aligns each row’s queries to the end of its keys.

Parameters:
Return type:

None

abstractmethod run(q, k, v)[source]#

(out, lse) (lse natural-log fp32) of the planned rows’ queries q, in plan order, over the step’s keys k and v. A row for every query passed; one past the plan (a captured bucket’s padding) holds whatever the kernel left, which reaches no real row.

Parameters:
Return type:

tuple[Tensor, Tensor]

whole_rows_only: bool = False#
class mstar.engine.resources.context_parallel.RingAttentionWrapper(placement, page_size, dtype, paged, device)[source]#

Bases: object

A split step’s attention over an SP-placed pool by striped pass-Q. The step’s tokens are striped over the partners by KV position (stripe_owners), so each rank holds the queries whose K/V it writes into its own slots. In round i a rank attends partner (r - i) % P’s queries over its slots, the stored context and the step’s own tokens together, with one call (_ring_rows); the partial goes back to that partner, which merges it into its running result (merge_lse).

One causal call is exact because a partner’s queries in a row sit at KV positions p0 + aP, and this rank’s keys at or below p number (p - r) // P + 1: one more per query, so the kernel’s bottom-right causal mask over the local length to the row’s last query is the right one. Only a row’s first query can see no key here (p0 < r); it plans as a row of its own over one sink slot, masked out of the merge (_masked), and its own rank always holds a key of it. Not causal, each row’s queries see its whole length, and a row with no slot here is masked the same way. A row this partner holds no query of is left out of its plan.

paged is one eager kernel over this rank’s slots per partner, all planned per step. Eager only: the node never splits a captured step (SeqShardSpec.eager). The partner plans differ by rank and no admission bounds them: FlashInfer’s paged manager, the one backend that takes a placed pool (PlacedAttention), so the one that plans a ring, bounds no plan at admission, as no refusal of its planner on lengths is known. A backend whose planner may refuse on lengths (MLA’s FA planner) would have to bound every rank’s partner plans at admission before it could plan a ring, or one rank alone could refuse a lockstep step.

Parameters:
attend(q, kv_cache_layer, k, v)[source]#

This rank’s queries’ attention over the whole pool, in their dtype, as ContextParallelWrapper.attend takes them; the step’s own tokens are already written, so k and v are unread. 2(P - 1) exchanges, the same on every rank, each awaited before its result is read.

Parameters:
Return type:

Tensor

forget_plan()[source]#

Every plan plans every kernel, so there is nothing to forget.

Return type:

None

plan(kv_out, counts, causal=True)[source]#

kv_out: the step’s logical KVPlanOutput for the label; counts: each partner’s queries, as the sequence shard striped them from the same plan (SeqRange.counts); causal: the step’s.

Parameters:
Return type:

None

reads_step_kv = False#
class mstar.engine.resources.context_parallel.SPContextParallelWrapper(placement, page_size, dtype, context, new, device, batch_size=None, max_total_tokens=None)[source]#

Bases: ContextParallelWrapper

Attention over an SPPlacement: the partners hold the same heads, so each attends its own slots and its zigzag share of the own tokens (ZigzagWrapper) for its own queries, merges the two, and one gather merges every partner’s.

Parameters:
new: ZigzagWrapper | None#
classmethod own_tokens(kernels, placement, device, workspace, kwargs, max_total_tokens)[source]#

The part a multi-token step attends its own tokens on, from the backend’s kernels (_plan_own plans it, attend runs it).

class mstar.engine.resources.context_parallel.SPPlacement(size, index, group)[source]#

Bases: ContextParallel

Placed over the node’s SP group, whose ranks hold the same heads: the partners are the whole group, in rank order (SPContextParallelWrapper).

Parameters:
  • size (int)

  • index (int)

  • group (CommGroup)

class mstar.engine.resources.context_parallel.ShareCapture(rows, queries, tokens)[source]#

Bases: NamedTuple

What the kernel of a captured step’s own tokens is sized for.

Parameters:
queries: int#

Alias for field number 1

rows: int#

Alias for field number 0

tokens: int#

Alias for field number 2

class mstar.engine.resources.context_parallel.TPContextParallelWrapper(placement, page_size, dtype, context, new, device, batch_size=None, max_total_tokens=None)[source]#

Bases: ContextParallelWrapper

Attention over a TPPlacement: the partners hold different query heads of one KV head, so each attends its slots for every partner’s heads (_exchange_queries), sends each partner its heads’ partial back (_exchange_partials), and merges them with the own tokens, which it attends whole for its own heads.

Parameters:
static local_geometry(placement, page_size, num_qo_heads)[source]#

And every partner’s query heads, which it attends for them.

Parameters:
Return type:

dict[str, int]

new: OwnTokensKernel | None#
classmethod own_tokens(kernels, placement, device, workspace, kwargs, max_total_tokens)[source]#

The part a multi-token step attends its own tokens on, from the backend’s kernels (_plan_own plans it, attend runs it).

class mstar.engine.resources.context_parallel.TPPlacement(size, index, group, partners)[source]#

Bases: ContextParallel

Placed over the block of the node’s TP ranks that hold one KV head, each with different query heads of it (TPContextParallelWrapper). The exchange is an all-to-all over the whole TP group with nothing sent to the ranks outside the block, so no group of its own is needed.

Parameters:
partners: tuple[int, ...]#
class mstar.engine.resources.context_parallel.ZigzagWrapper(placement, kernel, device)[source]#

Bases: object

A step’s own tokens under SP placement, attended for this rank’s zigzag chunks alone on kernel, out 0 and lse -inf off them. On a captured kernel it gathers the same number of queries every replay: its share, then padding, which gathers token 0 and scatters to a row run drops.

Parameters:
forget_plan()[source]#
Return type:

None

plan(qo_indptr, causal=True)[source]#
Parameters:
Return type:

None

run(q, k, v)[source]#

This rank’s share of the own tokens’ (out, lse) for every token of the step.

Parameters:
Return type:

tuple[Tensor, Tensor]

mstar.engine.resources.context_parallel.gather_merge(group, out, lse)[source]#

A query replicated on every rank of group attending keys sharded across them, from this rank’s out [..., H, D] and natural-log lse [..., H] over its share: one gather of both, then merge. Moves query-sized results rather than the keys.

Parameters:
Return type:

Tensor

mstar.engine.resources.context_parallel.local_indptrs(placement, indptrs, page_size)[source]#

A plan’s logical paged indptrs as this rank’s (page_size the logical one), and per row whether it holds any of this rank’s slots.

Every row keeps a length of at least 1 and a last page holding at least one slot, as on one rank: FlashInfer’s split-KV planner counts a row’s pages where its kernel counts its tokens, and a last page with none of this rank’s slots puts the two out of step. Such a page is dropped from its row. A row with no slot here attends one slot of SINK_PAGE instead, and its partial is masked before the merge (_masked).

Parameters:
Return type:

tuple[PagedIndptrs, ndarray]

mstar.engine.resources.context_parallel.merge(outs, lses)[source]#

Attention over the union of disjoint key sets from each set’s partial: each out [..., D] (a [K, ..., D] stack is a sequence too) and their natural-log lses stacked, [K, ...]; fp32.

Parameters:
Return type:

Tensor

mstar.engine.resources.context_parallel.merge_lse(outs, lses)[source]#

merge, and the union’s natural-log lse, fp32, to merge again.

Parameters:
Return type:

tuple[Tensor, Tensor]

mstar.engine.resources.context_parallel.position_pools(specs, node)[source]#

node’s resources that hold token positions, each to whether it places them (NodeResourceSpec.storage by position or by head; a copied one goes with either). Raises when one is placed and another isn’t.

Parameters:
Return type:

dict[str, bool]

mstar.engine.resources.context_parallel.ring_hop(size, rank, offset, send, recv)[source]#

A ring hop’s all_to_all_rows counts at offset: send rows to rank rank + offset, recv back from rank rank - offset, of size.

Parameters:
Return type:

tuple[list[int], list[int]]

mstar.engine.resources.context_parallel.stripe_owners(qo_indptr, stored, size)[source]#

Which of size partners stores each of a step’s tokens: a row’s token t goes to KV position stored + t (stored per row, the row’s KV before the step), whose slot partner (stored + t) % size holds, as size divides the page.

Parameters:
  • qo_indptr (ndarray)

  • stored (ndarray)

  • size (int)

Return type:

ndarray

mstar.engine.resources.context_parallel.zigzag_capture(size, rows, tokens)[source]#

A zigzag share’s kernel captured for a bucket of rows rows and tokens tokens: its fixed rows (at most 2 chunks a row, and one more for the padding queries) and queries (at most ceil(T / 2P) a chunk).

Parameters:
Return type:

ShareCapture

mstar.engine.resources.context_parallel.zigzag_rows(qo_indptr, size, index, causal=True)[source]#

Rank index of size’s zigzag chunks of the rows qo_indptr lays out (chunks r and 2P - 1 - r of each row’s 2P; see the context-parallel docs): the step’s tokens it attends, chunk by chunk, and its chunks as rows over their keys, each row’s first token to the chunk’s last (or the row’s, not causal).

Parameters:
  • qo_indptr (ndarray)

  • size (int)

  • index (int)

  • causal (bool)

Return type:

tuple[ndarray, KeyRows]