mstar.utils.knobs#
Engine knobs read by more than one package.
Each knob is set per deployment in the serving config’s knobs: block
(configure installs it once per worker process, before the engine is
built) and can be overridden for a run by its environment variable: a
non-empty MSTAR_* value wins over the config, which wins over the default.
Functions
|
|
|
|
|
Install the deployment config's |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- mstar.utils.knobs.check(settings)[source]#
settings(aknobs:block) in the env vars’ string format.Raises on a key that names no knob, so a misspelled setting fails at startup instead of being silently ignored.
- mstar.utils.knobs.compile_deterministic()[source]#
compile_deterministic/MSTAR_COMPILE_DETERMINISTIC(default off): Qwen3.5’s decode capture compiles in Inductor’s deterministic mode, as its prefill captures always do, so greedy output reproduces across server processes and the cold start shrinks, at the cost of one-row decode speed. Qwen3.5’s text walks are the only reader.- Return type:
- mstar.utils.knobs.configure(settings)[source]#
Install the deployment config’s
knobs:block for this process.
- mstar.utils.knobs.device_loopback_enabled()[source]#
device_loopback/MSTAR_DEVICE_LOOPBACK(default on): whether a node may keep its loop-back token on the device (the sampler’s slot master) instead of routing a tensor per request; also whether the sampler persists the last token at all.- Return type:
- mstar.utils.knobs.gc_gen0_threshold()[source]#
gc_gen0_threshold/MSTAR_GC_GEN0_THRESHOLD(default 0, keep CPython’s 700): the generation-0 threshold a worker sets after warmup. A decode step at 128 rows keeps more than 700 young container objects alive at its peak, so at 700 a collection runs almost every step and escalates to a full one (about 10 ms) every few hundred; the time lands in whichever span is allocating. 50000 removed them on Qwen3.5 0.8B at c=128 (8% shorter decode step) for at most that many young objects held between collections.- Return type:
- mstar.utils.knobs.idle_hold()[source]#
idle_hold/MSTAR_IDLE_HOLD(defaultoff):qoshas the API server hold a PM QoS CPU latency request for its life (cpus.hold_idle), which keeps every CPU of the host out of deep idle states.- Return type:
- mstar.utils.knobs.ingraph_decode_rope()[source]#
ingraph_decode_rope/MSTAR_INGRAPH_DECODE_ROPE(default off): a captured text prefill or mixed step builds its rotary tables inside the graph from the position resource’s planned positions instead of building them before the replay. A captured decode step always does.- Return type:
- mstar.utils.knobs.kv_chain_lazy_steps()[source]#
kv_chain_lazy_steps/MSTAR_KV_CHAIN_LAZY_STEPS(default 1): how many decode steps of sampled tokens the KV manager batches before extending the requests’ prefix chains, one extend per request per batch instead of one per step (the per-row Python of that extension was 0.3-0.4 ms a step at 128 rows). A page key then appears up to that many steps late, and a later commit indexes it, which the single-request path already allows for.1extends every step: 16 measured as noise at c64/c128 on 0.8B (the extension is a small part of the stop check’s tail).- Return type:
- mstar.utils.knobs.kv_pinned_indptrs()[source]#
kv_pinned_indptrs/MSTAR_KV_PINNED_INDPTRS(default off): the KV plan writes FlashInfer’s index arrays (qo_indptr, kv_indptr, kv_indices, last_page_len) into a ring of page-locked host buffers, and the decode wrapper copies the indices straight into its captured device buffer from there. The copies are then really asynchronous; from pageable memory the driver stages each one, and FlashInfer makes the indices copy blocking. Off by default: no measurable gain at c32-c128 on 0.8B. The cost is one CUDA event per plan, which a buffer set waits on before it is rewritten.- Return type:
- mstar.utils.knobs.pre_plan_enabled()[source]#
pre_plan/MSTAR_PRE_PLAN_SPEC(default on): the worker pre-plans the speculative batch on a plan thread while the previous step replays, and the engine double-buffers the resources that support it. Off, each step plans inline on the GPU thread. It wins on a deep decoder (Whisper large-v3, Qwen3-ASR) and loses on a shallow one (Whisper turbo’s 4 layers), where the step is too short to hide the planning in and the extra thread is just GIL contention.- Return type:
- mstar.utils.knobs.resident_planning()[source]#
resident_planning/MSTAR_RESIDENT_PLANNING(defaultauto): a captured decode step reads its rows off per-request tables held on the device instead of having them rebuilt on the plan thread (mstar.engine.resources.resident).autoturns it on for a node whose submodule names resident walks and every resource of which is resident-capable, and says why at startup when one is not;offkeeps the host plan everywhere.- Return type:
- mstar.utils.knobs.sampler_ingraph_scatter()[source]#
sampler_ingraph_scatter/MSTAR_SAMPLER_INGRAPH_SCATTER(default on): the captured sampler gathers its RNG offsets from the slot masters and scatters the advanced offsets and the sampled tokens back inside the graph, so a replayed step issues none of those launches on the gpu thread (each is a GIL release the main thread’s postprocess can win). Padding rows then address a trash master row, and the offset and last-token masters are allocated once forsampler_slotsconcurrent requests, since a captured graph holds their addresses; a request takes its row at admission and waits while none is free.- Return type:
- mstar.utils.knobs.sampler_slots()[source]#
sampler_slots/MSTAR_SAMPLER_SLOTS(default 4096): request rows of the masters a captured graph indexes when the in-graph scatter is on (seesampler_ingraph_scatter), so the most requests admitted at once on its node.- Return type:
- mstar.utils.knobs.step_thread_core()[source]#
step_thread_core/MSTAR_STEP_THREAD_CORE(defaultauto): pin each worker’s main, GPU and plan threads, and the API server’s result threads, to CPUs of their own.alwaysplaces them by the GPU’s rank on its NUMA node, a worker’s three on three cores of their own;siblingsplaces them the same way with a worker’s three on one core’s hardware threads; a number is the first worker’s core.autopins nothing and logs why for this host (seemstar.utils.cpus).
- mstar.utils.knobs.step_threads()[source]#
step_threads/MSTAR_STEP_THREADS(defaultthreaded): where a worker runs a step’s host work.threadeduses its GPU and plan threads;serialruns that work on the main thread, keeping CPU-GPU overlap and giving up CPU-CPU overlap, which the GIL mostly denies a Python-heavy step anyway.- Return type: