mstar.utils.knobs

Environment knobs read by more than one package.

Functions

device_loopback_enabled()	MSTAR_DEVICE_LOOPBACK (default 1): whether a node may keep its loop-back token on the device (the sampler's slot master) instead of routing a tensor per request; also whether the sampler persists the last token at all.
ingraph_decode_rope()	MSTAR_INGRAPH_DECODE_ROPE (default 0): a captured decode step builds its rotary tables inside the graph from the position resource's planned positions instead of staging them before the replay.
kv_chain_lazy_steps()	MSTAR_KV_CHAIN_LAZY_STEPS (default 16): how many decode steps of sampled tokens the KV manager batches before extending the requests' prefix chains, one extend per request per batch instead of one per step (the per-row Python of that extension was 0.3-0.4 ms a step at 128 rows).
kv_pinned_indptrs()	MSTAR_KV_PINNED_INDPTRS (default 0): the KV plan writes FlashInfer's index arrays (qo_indptr, kv_indptr, kv_indices, last_page_len) into a ring of page-locked host buffers, and the decode wrapper copies the indices straight into its captured device buffer from there.
kv_plan_cache()	MSTAR_KV_PLAN_CACHE (default 0): the KV manager plans a leased decode step from the previous step's plan when the batch is the same rows in the same order and nothing but that step's commit touched the streams: lengths advance by one, pages change only where a row crossed a page boundary, and the five index arrays come from numpy instead of a per-row Python pass.
sampler_ingraph_scatter()	MSTAR_SAMPLER_INGRAPH_SCATTER (default 0): the captured sampler gathers its RNG offsets from the slot masters and scatters the advanced offsets and the sampled tokens back inside the graph, so a replayed step issues none of those launches on the gpu thread (each is a GIL release the main thread's postprocess can win).
sampler_slots()	MSTAR_SAMPLER_SLOTS (default 4096): master rows allocated up front when the in-graph scatter is on (see sampler_ingraph_scatter).

mstar.utils.knobs.device_loopback_enabled()

MSTAR_DEVICE_LOOPBACK (default 1): whether a node may keep its loop-back token on the device (the sampler’s slot master) instead of routing a tensor per request; also whether the sampler persists the last token at all.

Return type:

bool

mstar.utils.knobs.ingraph_decode_rope()

MSTAR_INGRAPH_DECODE_ROPE (default 0): a captured decode step builds its rotary tables inside the graph from the position resource’s planned positions instead of staging them before the replay.

Return type:

bool

mstar.utils.knobs.kv_chain_lazy_steps()

MSTAR_KV_CHAIN_LAZY_STEPS (default 16): how many decode steps of sampled tokens the KV manager batches before extending the requests’ prefix chains, one extend per request per batch instead of one per step (the per-row Python of that extension was 0.3-0.4 ms a step at 128 rows). A page key then appears up to that many steps late, and a later commit indexes it, which the single-request path already allows for. 1 (the default) extends every step: 16 measured as noise at c64/c128 on 0.8B (the extension is a small part of the stop check’s tail).

Return type:

int

mstar.utils.knobs.kv_pinned_indptrs()

MSTAR_KV_PINNED_INDPTRS (default 0): the KV plan writes FlashInfer’s index arrays (qo_indptr, kv_indptr, kv_indices, last_page_len) into a ring of page-locked host buffers, and the decode wrapper copies the indices straight into its captured device buffer from there. The copies are then really asynchronous; from pageable memory the driver stages each one, and FlashInfer makes the indices copy blocking. Off by default: no measurable gain at c32-c128 on 0.8B. The cost is one CUDA event per plan, which a buffer set waits on before it is rewritten.

Return type:

bool

mstar.utils.knobs.kv_plan_cache()

MSTAR_KV_PLAN_CACHE (default 0): the KV manager plans a leased decode step from the previous step’s plan when the batch is the same rows in the same order and nothing but that step’s commit touched the streams: lengths advance by one, pages change only where a row crossed a page boundary, and the five index arrays come from numpy instead of a per-row Python pass. 0 (the default until the reworked path is measured) plans every step from the streams; 2 does both and logs any difference (a correctness check for runs).

Return type:

int

mstar.utils.knobs.sampler_ingraph_scatter()

MSTAR_SAMPLER_INGRAPH_SCATTER (default 0): the captured sampler gathers its RNG offsets from the slot masters and scatters the advanced offsets and the sampled tokens back inside the graph, so a replayed step issues none of those launches on the gpu thread (each is a GIL release the main thread’s postprocess can win). Padding rows then address a trash master row, and the masters are allocated once for MSTAR_SAMPLER_SLOTS (default 4096) concurrent requests, since a captured graph holds their addresses.

Return type:

bool

mstar.utils.knobs.sampler_slots()

MSTAR_SAMPLER_SLOTS (default 4096): master rows allocated up front when the in-graph scatter is on (see sampler_ingraph_scatter).

Return type:

int