mstar.utils.knobs#

Engine knobs read by more than one package.

Each knob is set per deployment in the serving config’s knobs: block (configure installs it once per worker process, before the engine is built) and can be overridden for a run by its environment variable: a non-empty MSTAR_* value wins over the config, which wins over the default.

Functions

check(settings)

settings (a knobs: block) in the env vars' string format.

compile_deterministic()

compile_deterministic / MSTAR_COMPILE_DETERMINISTIC (default off): Qwen3.5's decode capture compiles in Inductor's deterministic mode, as its prefill captures always do, so greedy output reproduces across server processes and the cold start shrinks, at the cost of one-row decode speed.

configure(settings)

Install the deployment config's knobs: block for this process.

device_loopback_enabled()

device_loopback / MSTAR_DEVICE_LOOPBACK (default on): whether a node may keep its loop-back token on the device (the sampler's slot master) instead of routing a tensor per request; also whether the sampler persists the last token at all.

gc_gen0_threshold()

gc_gen0_threshold / MSTAR_GC_GEN0_THRESHOLD (default 0, keep CPython's 700): the generation-0 threshold a worker sets after warmup.

idle_hold()

idle_hold / MSTAR_IDLE_HOLD (default off): qos has the API server hold a PM QoS CPU latency request for its life (cpus.hold_idle), which keeps every CPU of the host out of deep idle states.

ingraph_decode_rope()

ingraph_decode_rope / MSTAR_INGRAPH_DECODE_ROPE (default off): a captured text prefill or mixed step builds its rotary tables inside the graph from the position resource's planned positions instead of building them before the replay.

kv_chain_lazy_steps()

kv_chain_lazy_steps / MSTAR_KV_CHAIN_LAZY_STEPS (default 1): how many decode steps of sampled tokens the KV manager batches before extending the requests' prefix chains, one extend per request per batch instead of one per step (the per-row Python of that extension was 0.3-0.4 ms a step at 128 rows).

kv_pinned_indptrs()

kv_pinned_indptrs / MSTAR_KV_PINNED_INDPTRS (default off): the KV plan writes FlashInfer's index arrays (qo_indptr, kv_indptr, kv_indices, last_page_len) into a ring of page-locked host buffers, and the decode wrapper copies the indices straight into its captured device buffer from there.

pre_plan_enabled()

pre_plan / MSTAR_PRE_PLAN_SPEC (default on): the worker pre-plans the speculative batch on a plan thread while the previous step replays, and the engine double-buffers the resources that support it.

resident_planning()

resident_planning / MSTAR_RESIDENT_PLANNING (default auto): a captured decode step reads its rows off per-request tables held on the device instead of having them rebuilt on the plan thread (mstar.engine.resources.resident).

sampler_ingraph_scatter()

sampler_ingraph_scatter / MSTAR_SAMPLER_INGRAPH_SCATTER (default on): the captured sampler gathers its RNG offsets from the slot masters and scatters the advanced offsets and the sampled tokens back inside the graph, so a replayed step issues none of those launches on the gpu thread (each is a GIL release the main thread's postprocess can win).

sampler_slots()

sampler_slots / MSTAR_SAMPLER_SLOTS (default 4096): request rows of the masters a captured graph indexes when the in-graph scatter is on (see sampler_ingraph_scatter), so the most requests admitted at once on its node.

step_thread_core()

step_thread_core / MSTAR_STEP_THREAD_CORE (default auto): pin each worker's main, GPU and plan threads, and the API server's result threads, to CPUs of their own.

step_threads()

step_threads / MSTAR_STEP_THREADS (default threaded): where a worker runs a step's host work.

mstar.utils.knobs.check(settings)[source]#

settings (a knobs: block) in the env vars’ string format.

Raises on a key that names no knob, so a misspelled setting fails at startup instead of being silently ignored.

Parameters:

settings (Mapping[str, object] | None)

Return type:

dict[str, str]

mstar.utils.knobs.compile_deterministic()[source]#

compile_deterministic / MSTAR_COMPILE_DETERMINISTIC (default off): Qwen3.5’s decode capture compiles in Inductor’s deterministic mode, as its prefill captures always do, so greedy output reproduces across server processes and the cold start shrinks, at the cost of one-row decode speed. Qwen3.5’s text walks are the only reader.

Return type:

bool

mstar.utils.knobs.configure(settings)[source]#

Install the deployment config’s knobs: block for this process.

Parameters:

settings (Mapping[str, object] | None)

Return type:

None

mstar.utils.knobs.device_loopback_enabled()[source]#

device_loopback / MSTAR_DEVICE_LOOPBACK (default on): whether a node may keep its loop-back token on the device (the sampler’s slot master) instead of routing a tensor per request; also whether the sampler persists the last token at all.

Return type:

bool

mstar.utils.knobs.gc_gen0_threshold()[source]#

gc_gen0_threshold / MSTAR_GC_GEN0_THRESHOLD (default 0, keep CPython’s 700): the generation-0 threshold a worker sets after warmup. A decode step at 128 rows keeps more than 700 young container objects alive at its peak, so at 700 a collection runs almost every step and escalates to a full one (about 10 ms) every few hundred; the time lands in whichever span is allocating. 50000 removed them on Qwen3.5 0.8B at c=128 (8% shorter decode step) for at most that many young objects held between collections.

Return type:

int

mstar.utils.knobs.idle_hold()[source]#

idle_hold / MSTAR_IDLE_HOLD (default off): qos has the API server hold a PM QoS CPU latency request for its life (cpus.hold_idle), which keeps every CPU of the host out of deep idle states.

Return type:

str

mstar.utils.knobs.ingraph_decode_rope()[source]#

ingraph_decode_rope / MSTAR_INGRAPH_DECODE_ROPE (default off): a captured text prefill or mixed step builds its rotary tables inside the graph from the position resource’s planned positions instead of building them before the replay. A captured decode step always does.

Return type:

bool

mstar.utils.knobs.kv_chain_lazy_steps()[source]#

kv_chain_lazy_steps / MSTAR_KV_CHAIN_LAZY_STEPS (default 1): how many decode steps of sampled tokens the KV manager batches before extending the requests’ prefix chains, one extend per request per batch instead of one per step (the per-row Python of that extension was 0.3-0.4 ms a step at 128 rows). A page key then appears up to that many steps late, and a later commit indexes it, which the single-request path already allows for. 1 extends every step: 16 measured as noise at c64/c128 on 0.8B (the extension is a small part of the stop check’s tail).

Return type:

int

mstar.utils.knobs.kv_pinned_indptrs()[source]#

kv_pinned_indptrs / MSTAR_KV_PINNED_INDPTRS (default off): the KV plan writes FlashInfer’s index arrays (qo_indptr, kv_indptr, kv_indices, last_page_len) into a ring of page-locked host buffers, and the decode wrapper copies the indices straight into its captured device buffer from there. The copies are then really asynchronous; from pageable memory the driver stages each one, and FlashInfer makes the indices copy blocking. Off by default: no measurable gain at c32-c128 on 0.8B. The cost is one CUDA event per plan, which a buffer set waits on before it is rewritten.

Return type:

bool

mstar.utils.knobs.pre_plan_enabled()[source]#

pre_plan / MSTAR_PRE_PLAN_SPEC (default on): the worker pre-plans the speculative batch on a plan thread while the previous step replays, and the engine double-buffers the resources that support it. Off, each step plans inline on the GPU thread. It wins on a deep decoder (Whisper large-v3, Qwen3-ASR) and loses on a shallow one (Whisper turbo’s 4 layers), where the step is too short to hide the planning in and the extra thread is just GIL contention.

Return type:

bool

mstar.utils.knobs.resident_planning()[source]#

resident_planning / MSTAR_RESIDENT_PLANNING (default auto): a captured decode step reads its rows off per-request tables held on the device instead of having them rebuilt on the plan thread (mstar.engine.resources.resident). auto turns it on for a node whose submodule names resident walks and every resource of which is resident-capable, and says why at startup when one is not; off keeps the host plan everywhere.

Return type:

str

mstar.utils.knobs.sampler_ingraph_scatter()[source]#

sampler_ingraph_scatter / MSTAR_SAMPLER_INGRAPH_SCATTER (default on): the captured sampler gathers its RNG offsets from the slot masters and scatters the advanced offsets and the sampled tokens back inside the graph, so a replayed step issues none of those launches on the gpu thread (each is a GIL release the main thread’s postprocess can win). Padding rows then address a trash master row, and the offset and last-token masters are allocated once for sampler_slots concurrent requests, since a captured graph holds their addresses; a request takes its row at admission and waits while none is free.

Return type:

bool

mstar.utils.knobs.sampler_slots()[source]#

sampler_slots / MSTAR_SAMPLER_SLOTS (default 4096): request rows of the masters a captured graph indexes when the in-graph scatter is on (see sampler_ingraph_scatter), so the most requests admitted at once on its node.

Return type:

int

mstar.utils.knobs.step_thread_core()[source]#

step_thread_core / MSTAR_STEP_THREAD_CORE (default auto): pin each worker’s main, GPU and plan threads, and the API server’s result threads, to CPUs of their own. always places them by the GPU’s rank on its NUMA node, a worker’s three on three cores of their own; siblings places them the same way with a worker’s three on one core’s hardware threads; a number is the first worker’s core. auto pins nothing and logs why for this host (see mstar.utils.cpus).

Return type:

str | int

mstar.utils.knobs.step_threads()[source]#

step_threads / MSTAR_STEP_THREADS (default threaded): where a worker runs a step’s host work. threaded uses its GPU and plan threads; serial runs that work on the main thread, keeping CPU-GPU overlap and giving up CPU-CPU overlap, which the GIL mostly denies a Python-heavy step anyway.

Return type:

str