Environment variables#

Runtime knobs M* reads from the environment. New variables should be documented here as they are introduced.

Communication#

Variable

Default

Meaning

MSTAR_WIRE_CODEC

msgpack

How messages are encoded on every communicator edge (see mstar.communication.codec.default_codec()). msgpack: the typed encoding in mstar.communication.wire, language-neutral so an edge can terminate in a Rust process. pickle is no longer supported and fails at startup. Loosely typed metadata uses data-only extensions instead of pickle. Upgrade all processes in a deployment together: older processes cannot decode the new extensions, and updated processes reject legacy pickle messages and metadata.

MSTAR_ZMQ_TRANSPORT

constructor’s protocol

Overrides the communicator protocol (IPC or TCP) for a process, e.g. to run entities on separate hosts.

MSTAR_ZMQ_TCP_HOST

127.0.0.1

Host used to build peer endpoints when the protocol is TCP.

MSTAR_ZMQ_TCP_BASE_PORT

19000

Base of the deterministic entity-id → TCP port map (api_server = base, conductor = base+1, worker_<rank> = base+100+rank).

MSTAR_REQUIRE_ACCELERATOR_GRAPHS

0

1 makes a worker fail at startup when any accelerator graph bucket could not be captured, instead of serving that bucket eagerly at 10-20x the latency. Off by default: a failed capture is logged at ERROR with a per-runner summary and the rest keeps running.

MSTAR_DEPLOYMENT_TOKEN

random per launch

Mixed into the namespace of the SHM tensor files and arena segments, so no two launches share names: not a restart on the same socket prefix, and not two deployments that share /dev/shm from different containers (--ipc=host, hostIPC) with the same socket prefix and ports. The API server mints a new one at every launch unless a launcher set it (one an API server minted is replaced, also when inherited), and the conductor and workers inherit it. Set it only when a launcher starts a deployment’s processes separately: the same value in all of them, and a new value per launch (a dead run’s files are swept only once no process holds its namespace, and a stopping worker unlinks its own files even after its conductor is gone, which a reused value would let it do to the next launch’s files). The API server logs a warning whenever it keeps a value the launcher set.

MSTAR_KV_SHM_DIR

/dev/shm (or /tmp if unavailable)

Base directory for host-staged KV snapshots when remote KV transfer uses SHM (make_deployment_kv_shm_dir). Each deployment creates a private child directory here, named under its tensor-file namespace (see MSTAR_DEPLOYMENT_TOKEN), and holds that namespace’s lock file here while it runs; request cleanup removes snapshots, and startup removes this user’s directories whose lock no live process holds. All workers sharing KV snapshots must see the same filesystem. The lock is flock, which some cluster filesystems (GPFS, for one) keep per node, so do not point deployments on different nodes at one shared directory there: each would sweep the other’s live directories.

MSTAR_SHM_ARENA_SEGMENT_MB

256, less on a small or crowded /dev/shm

Size of each arena segment. The arena grows segment by segment; existing segments never move (registrations stay valid). A segment’s /dev/shm pages are committed when it is created (pinned segments always were, by cudaHostRegister), so the whole segment counts against /dev/shm, as do spill files, which land in /dev/shm by default. A segment /dev/shm cannot hold counts as arena full (sends spill, see MSTAR_SHM_ARENA_SPILL).

MSTAR_SHM_ARENA_MAX_SEGMENTS

32, fewer on a small or crowded /dev/shm

Growth cap PER ENTITY. Every entity (workers + the api-server data worker) on CUDA or CPU creates its own arena, so node-wide /dev/shm demand can reach MAX_SEGMENTS x SEGMENT_MB x num_entities — size against df -h /dev/shm (tmpfs defaults to ~50% of RAM). With neither this nor MSTAR_SHM_ARENA_SEGMENT_MB set, the API server sizes every arena of the deployment once, before any exists (see MSTAR_SHM_ARENA_PLAN): /dev/shm’s free space split evenly among the arenas, plus one share left for spill files. Each gets 256 MiB segments, 1 to 32 of them, or where a share is under 256 MiB, one segment of the largest power-of-two MiB that fits (16 MiB at least), logged at startup. An entity started without the API server takes a quarter of the free space it finds. Where a share is under 16 MiB, or when its first segment cannot be created, the entity has no arena and sends every tensor as a file (it still reads its peers’ arenas); a warning says why. Entities on other accelerators (ROCm, XPU) do the same until the arena is measured there. Set by hand, construction fails fast if one entity’s ceiling exceeds /dev/shm. At the cap, sends spill (see MSTAR_SHM_ARENA_SPILL).

MSTAR_SHM_ARENA_PLAN

set by the API server

Internal: <segment MiB>x<max segments> for every arena of the deployment (0x0: none, send by files), written by mstar.communication.arena.export_arena_plan() before any arena exists, measured after a dead run’s files are swept, and inherited by the conductor and workers, so each arena gets the same size whatever order the processes start in. Read by _arena_sizing; MSTAR_SHM_ARENA_SEGMENT_MB and _MAX_SEGMENTS take precedence. Not meant to be set by hand.

MSTAR_SHM_ARENA_FULL_TIMEOUT_S

30

Strict mode only (MSTAR_SHM_ARENA_SPILL=0): how long a send backpressures on a full arena before failing.

MSTAR_SHM_ARENA_SPILL

1

Degrade gracefully at the segment cap: stage the tensor through the per-uuid file protocol instead — slower, never fails, matching the file transport’s saturation behavior. 0 restores strict backpressure + timeout — only meaningful where ANOTHER thread drains consumer ACKs (the threaded api-server); on a worker the ACKs arrive on the very thread that would be waiting. An entity without an arena spills every send either way.

MSTAR_SHM_ARENA_SPILL_AFTER_S

0

Optional grace before spilling, for deployments where another thread frees slots concurrently. Default 0: spill immediately (a worker cannot receive ACKs while it waits).

MSTAR_SHM_ARENA_PIN

1

cudaHostRegister each mapped segment (both sides) so D2H/H2D copies through the side streams run at page-locked bandwidth and stay asynchronous. 0 disables (pageable copies).

MSTAR_SHM_ARENA_PIN_MAX_MB

4096

Budget for TOTAL pinned host memory PER PROCESS, distinct from the segment cap (pinned pages come out of the OS’s pageable pool system-wide). Node-wide pinned demand is approx PIN_MAX_MB x num_entities — a consumer pins peer segments too, so one process can pin more than its own arena holds. Segments past the budget stay unpinned: copies work, without async overlap.

MSTAR_SHM_ARENA_GATHER_MAX_BYTES

16384

Largest item (bytes) a consumer gathers with others of the same shape, dtype and segment into one pinned buffer and one H2D copy, instead of a copy per item (_read_many_from_arena). The default is the crossover measured on an H200 host; 0 disables the gather. Read in ArenaShmCommunicationManager.__init__.

MSTAR_SHM_ARENA_GATHER_WORDS_MAX_BYTES

512

Up to this item size (bytes, 8-byte aligned) the gather indexes 8-byte words in one pass; larger items are copied one slice each. Measured on an H200 host. Read in ArenaShmCommunicationManager.__init__.

MSTAR_SHM_ARENA_SLOT_TTL_S

0

TTL backstop for abort-orphaned slots (a request aborted after staging but before all consumer ACKs defers reclaim forever). A slot older than the request timeout cannot have a legitimate reader, so a bound safely above it (recommend >= 2x the request timeout) cannot race a real consumer. 0 disables (default, pending review discussion); reclaims run under capacity pressure and with the periodic stats sweep, logging loudly.

MSTAR_SHM_ARENA_STATS_INTERVAL_S

60

Under --log-stats: how often the arena logs its occupancy / fragmentation snapshot (segments, free bytes, largest contiguous free block, pinned bytes).

NCCL_SOCKET_IFNAME

lo

NCCL’s own variable. Every worker runs on the conductor’s host, so the conductor sets lo for its workers and NCCL’s bootstrap sockets listen on loopback only. A value you set, or a set NCCL_SOCKET_FAMILY, is kept as is.

NCCL_GRAPH_REGISTER

0

NCCL’s own variable. On, NCCL registers the buffers a CUDA graph captures, which succeeds on expandable_segments memory, and Cosmos3 SP4 with batched captures then gave wrong images (#897). Workers on CUDA set 0 unless you set it.

GLOO_SOCKET_IFNAME

lo

PyTorch’s variable for the Gloo groups next to each TP/SP group. Same rule: the conductor sets lo for its workers unless you set it, so Gloo does not listen on the address the hostname resolves to.

Models#

Per-model knobs, read when the submodule is built. They are scoped to one model, so they are named for it.

Variable

Default

Meaning

MSTAR_VIT_BATCHING

0

BAGEL: batch several requests through the ViT encoder in one forward. Off by default because flash-attn’s varlen reductions across packed images produce small bf16 drift, which at greedy temperature=0 can flip a downstream LLM argmax. When off, prefill_vit runs one request at a time.

MSTAR_VIT_ACCELERATOR_GRAPH

0

BAGEL: accelerator graph capture of the ViT block loop, over the node’s ragged attention resource (see Adding a New Model). Requires the deployment option model_kwargs.accelerator_graph to be true; BAGEL defaults this option to true (see Supported Models). Off by default: capture costs one graph per (batch size, token bucket) and the eager flash-attn path is already fast. The win is removing per-layer launch overhead on small images.

MSTAR_VIT_ACCELERATOR_GRAPH_TOKEN_BUCKETS

512,1024,2048,4096,4900

Comma-separated token-count buckets to capture, when MSTAR_VIT_ACCELERATOR_GRAPH=1. A batch longer than the largest bucket runs eagerly. 4900 is 70*70, the exact length the vllm preprocess option emits.

MSTAR_VIT_ACCELERATOR_GRAPH_BATCH_SIZES

1,2,4

Comma-separated batch sizes to capture. Read only when MSTAR_VIT_BATCHING=1; otherwise only batch size 1 is captured.

COSMOS3_DISABLE_ACCELERATOR_GRAPH

unset

Cosmos3: any nonempty value disables denoise, prefill and reasoner decode capture. The deployment option model_kwargs.accelerator_graph: false also disables all these captures and cannot be overridden by this variable.

COSMOS3_DISABLE_PREFILL_ACCELERATOR_GRAPH

unset

Cosmos3: any nonempty value disables only the understanding-tower prefill capture. Denoise and reasoner decode captures remain eligible when model_kwargs.accelerator_graph is true.

MSTAR_OMNIVOICE_MODEL_PATH

unset

OmniVoice: load weights from this local directory instead of the checkpoint the registry names. For serving a fine-tune of the same architecture off a mounted volume, where the registry entry would otherwise pull the public weights from the Hub.

MSTAR_ASR_DATA

data/asr

test/asr (GPU parity tests): the set benchmark.asr_data built.

MSTAR_WHISPER_REPO

openai/whisper-large-v3-turbo

test/asr/test_whisper_parity.py: the Whisper checkpoint to compare.

MSTAR_QWEN_ASR_REPO

Qwen/Qwen3-ASR-1.7B

test/asr/test_qwen3_asr_parity.py: the Qwen3-ASR checkpoint to compare.

MSTAR_QWEN_ASR_REF

refs/Qwen3-ASR

test/asr/test_qwen3_asr_parity.py: a checkout of the qwen-asr SDK, the reference transcripts come from.

MSTAR_QWEN_ASR_REF_SITE

unset

test/asr/test_qwen3_asr_parity.py: a directory holding transformers 4.57, put first on the path of the subprocess that runs the SDK (whose model code does not run on transformers 5). Unset under transformers 5, the SDK comparison skips.

Serving (Python frontend)#

Read by mstar-serve / mstar.api_server.entrypoint.

Variable

Default

Meaning

MSTAR_MAX_BODY_MB

128

Request body limit on every route, as in the Rust frontend: a larger body gets a 413, refused on its Content-Length or, for a chunked upload, while it is read. /v1/audio/transcriptions keeps its own 512 MiB file cap when this is lower. A body refused on its Content-Length is answered before it is read, so a client that writes the whole body before reading the reply can see a connection reset instead of the 413; one that sends Expect: 100-continue gets the 413 before it sends the body.

MSTAR_ROOT_PATH

""

Mount the app under a URL sub-path. Set it when an ingress routes the deployment at a prefix and does not strip that prefix before the request reaches the pod (Run:AI serves a workload at /<project>/<job-name>/, for instance); without it FastAPI 404s every route. Empty by default, so a direct deployment is unaffected.

MSTAR_MAX_IMAGES_PER_REQUEST

10

Largest n one /v1/images/generations request may ask for (see mstar.api_server.openai.serving_images.max_images_per_request()); n outside 1 to this value is a 400 before anything is submitted. The Rust frontend and the Dynamo bridge apply the same variable and default.

MSTAR_API_KEY

unset

API key (read by mstar.api_server.auth.resolve_api_key(); --api-key overrides it). When set, every route except /health and /metrics needs Authorization: Bearer <key>, WebSocket handshakes included; unset or empty turns the check off. The launcher passes it on to the Rust frontend only (the conductor and workers do not inherit it), and mstar.client.MStarClient sends it when not given api_key. See API key.

MSTAR_SILERO_VAD_PATH

unset

Silero VAD ONNX file (silero_vad.onnx) for /v1/realtime server_vad turn detection, read by realtime_vad.silero_weights. Unset, the copy bundled with the silero-vad package (mstar[vad]) is used; with neither, or with a file whose inputs or outputs differ (the package’s other exports), server_vad sessions are refused. Never downloaded.

MSTAR_SMART_TURN_PATH

unset

Pipecat Smart Turn v3 ONNX file for /v1/realtime semantic_vad, read by realtime_vad.smart_turn_weights. Unset, the pipecat-ai/smart-turn-v3 smart-turn-v3.2-cpu.onnx file in the local Hugging Face cache is used if present; with neither, or with a file whose inputs or outputs differ, semantic_vad sessions are refused. Never downloaded.

Serving (Rust frontend)#

Read by the mstar-server binary and its bridge (mstar-serve --rust-frontend; see Installation).

Variable

Default

Meaning

MSTAR_SERVER_BIN

unset

Path to the mstar-server binary. Fallback order: --rust-frontend-bin, this variable, the running environment’s scripts directory (where the Linux wheels install it), $PATH, then the in-repo rust/server/target/release build. In a checkout the fallbacks skip a binary built from other sources, while this one is used with a warning (find_server_binary).

MSTAR_API_KEY

unset

Same rule as the Python server (rust/server/src/auth.rs, read once at startup): every route except /health and /metrics needs Authorization: Bearer <key>. mstar-serve --rust-frontend sets it from --api-key / its own environment.

MSTAR_REQUEST_TIMEOUT_S

600

Per-request budget in the Rust frontend; on expiry the client gets an error and the request is aborted in the backend.

MSTAR_SAMPLE_RATE

24000

Sample rate stamped on /v1/audio/speech WAV output.

MSTAR_ALLOW_REMOTE

0

Allow http(s) media URLs in requests (fetched server-side, 30 s timeout). Off by default.

MSTAR_MAX_CONCURRENT_REQUESTS

256

Admission cap on in-flight generation requests; beyond it clients get an immediate 503 instead of queueing into the request timeout. /health, /metrics and /v1/models bypass the cap.

MSTAR_MAX_BODY_MB

128

Request body limit (multipart uploads included).

MSTAR_TOKENIZE_PARALLEL

1

When a long prompt is encoded as parallel chunks, set TOKENIZERS_PARALLELISM=true for the process first: the Rust tokenizer only uses its thread pool while that variable reads true, and serving environments often export it false. 0 leaves the variable as it is (the chunked encode then runs serially).

MSTAR_OPENAI_ADAPTER

set by the launcher

The class name of the model’s adapter in ADAPTER_REGISTRY (openai_adapter_class in mstar/api_server/rust_frontend.py), which picks the OpenAI translation mstar-server uses (Adapter::from_python). mstar-serve --rust-frontend always sets it; a mstar-server started by hand without it answers 404 on /v1/*.

MSTAR_TOKENIZER

unset

Path to a HuggingFace tokenizer.json enabling frontend tokenization. Leave unset with the Python backend — its preprocess worker owns tokenization, and the bridge rejects pre-tokenized ingest.

Rust frontend limitations#

The Rust frontend serves no /generate/ws, so a model’s live client input (appends to a running request, see Using a Server) needs the Python frontend; over /generate such a model’s input ends when the request is submitted.

It also carries no audio encoder or media muxer (the Python frontend uses soundfile / ffmpeg), so two surfaces degrade:

  • /v1/audio/speech produces only wav and pcm. Any other response_format (mp3, opus, flac, …) is rejected with a 400 rather than silently returned as WAV. Run the Python frontend for compressed containers.

  • /v1/audio/speech refuses ref_audio (Qwen3-TTS voice cloning) with a 400: the Python frontend decodes the clip and checks its length before submitting it, and the Rust frontend has no decoder to do that with.

  • /v1/videos/generations (Cosmos3) returns the video-only mp4. When a request sets generate_sound, Cosmos3 generates a separate audio track that mstar muxes into the mp4 as an AAC track; the Rust frontend cannot mux it, so the audio is dropped and a warning is logged. Do not set generate_sound on the Rust frontend (it spends compute on a track the client won’t receive), or run the Python frontend for sound video.

Request media#

Read by the model’s media loaders in the data worker’s preprocess thread, behind either frontend. All four are also checked once when the data worker starts (mstar.model.media_limits.check_media_limits()), so an invalid value stops the server instead of failing every media request. The pixel and audio caps hold for a request’s files together (mstar.model.media_limits.RequestMediaBudget), so attaching more files does not multiply them.

Variable

Default

Meaning

MSTAR_MAX_IMAGE_PIXELS

89478485

Pixels a request’s images may decode to, summed over them (see mstar.model.media_limits.max_image_pixels()), checked from each header before decoding; over it, the request gets a 400. PNG, JPEG, WebP and GIF are read (FLUX.2 klein, which decodes with Pillow, also reads BMP and TIFF); other formats get a 400. A GIF or an animated WebP is read as its first frame.

MSTAR_MAX_VIDEO_PIXELS

1073741824

Pixels a request’s videos may decode to (see mstar.model.media_limits.max_video_pixels()), summed over the frames the model’s loader decodes, each frame counting as at least 64x64; over it, the request gets a 400. At 1080p30 the default refuses clips over about 17 s on Cosmos3, which decodes every frame, and over about 259 s on Qwen3-Omni and Qwen3.5, which decode 2 frames a second (Qwen3-Omni decodes every frame, as Cosmos3, when the clip’s header gives no frame count).

MSTAR_MAX_MEDIA_FILES

16

Images, audio clips and videos one request may attach, together (see mstar.model.media_limits.max_media_files()); more get a 400 before any is decoded. A model can take fewer of one kind (mstar.model.base.Model.max_input_files(); Qwen3-TTS, Chatterbox, Whisper and Higgs-Audio take one audio clip).

MSTAR_MAX_AUDIO_SECONDS

3600

Seconds a request’s audio clips may decode to, summed over them (see mstar.model.media_limits.max_audio_seconds()), checked from each header before the model decodes it and again from the decoded clip (a file whose header gives no length, such as a FLAC with no sample count, is counted as it decodes, or from its size for a WAV with no data length); over it, the request gets a 400.

Worker scheduling#

Variable

Default

Meaning

MSTAR_TP_ASYNC_SCHED

config’s tp_async_sched, else 1

Async scheduling for lockstep-parallel (TP / SP) nodes; overrides the deployment config’s tp_async_sched. 1 (or true): the instance leader speculates step N+1 of the parallel node during forward N (the existing single-worker speculation machinery, gate opened) and broadcasts it at once as a speculative ScheduleTPNode; followers rebuild the identical batch during their own forward N and every rank submits N+1 the moment N completes. Removes the per-step serial build/plan from the group’s critical path. Voids (allocation failure, per-rid failure, a continuing request without loop-back output) are derived by each rank from replicated state, never signalled. A comma-separated list of node names (thinker,talker) enables it for those parallel nodes only. 0 (or false): the serial path — leader schedules after N, followers rebuild after the broadcast. Set it identically on every rank of an instance: the workers compare it at startup and refuse to start on a mismatch. Leave MSTAR_ENGINE_STEP_SYNC at 0 with it: that throttle holds the GPU thread until step N drains, which serialises the very overlap this flag buys.

MSTAR_PRE_PLAN_SPEC

config’s knobs: pre_plan, else 1

Pre-plan the speculative batch’s attention on a dedicated thread while the previous replay runs. 0 plans inline on the GPU thread, which Whisper turbo’s serve config sets: its 4-layer steps are too short to hide the planning in. Read by knobs.pre_plan_enabled.

MSTAR_STEP_THREAD_CORE

config’s knobs: step_thread_core, else auto

Pin each worker’s main, GPU and plan threads, and the API server’s message thread and Rust frontend bridge on a CPU of their own. A step hands work between the three worker threads several times and every step’s tokens pass from the message thread to the bridge; on a host whose idle cores drop into deep C-states, each handoff to a sleeping core waits for it to wake. off pins nothing. always gives each worker cpus.STEP_CORES (3) cores, one hardware thread of each, and the API server one core; no two pinned threads share a core and the siblings of the CPUs taken stay unpinned. siblings puts each worker’s three threads on all hardware threads of one core instead (two cores of one L2 cluster, topology/cluster_cpus_list, where a core has one CPU in the mask); cpus._siblings has its numbers. On a deep-idle Xeon 6767P it beat no pin and three separate cores; on two H200 hosts with SMT it lost to no pin. auto pins nothing and logs one line saying why for this host, since no host it can tell apart has a measurement in which a layout beats no pin: idle_hold is set (with the cores held awake pinning measured no faster), the CPU vendor isn’t Intel (cpus.AUTO_VENDORS, by /proc/cpuinfo’s vendor_id), no enabled idle state of 100 µs or more shows on a CPU of the process mask (/sys/devices/system/cpu/cpu*/cpuidle/state*/latency, cpus.deepest_idle, cpus.DEEP_IDLE_US), or, on a deep-idle Intel host, three separate cores measured no faster than no pin. A number names the first worker’s core (its hardware threads in the mask); the next workers take the next cores up and the API server the last core of the mask. A number outside the mask, or too near its top for every worker, stops the server at startup. always and siblings place by the GPU’s rank among the host’s GPUs on its NUMA node (sysfs /sys/bus/pci/devices: the PCI functions of the GPU’s vendor and class in address order, numa_node and local_cpulist), so servers on different GPUs take different CPUs without knowing about each other: the API server the core at its first worker’s GPU’s rank from the top of the node’s cores in the process mask, a worker its cores at that rank from the bottom of the rest; further workers of one deployment on one GPU step by the node’s GPU count. A GPU whose node has no CPU in the mask picks, by its rank among the host’s GPUs in the same case, from the cores the node-local GPUs leave. Under always the cores a GPU picks from must hold four per GPU that picks from them (two under siblings). A mask too small for that across the host may do for the GPUs this container has a device node for, since only those can share it, and those rank instead (NVIDIA only; a 2-GPU slice of an 8-GPU host fits an 8-core mask this way). Short of that, the GPUs this process was given (CUDA_VISIBLE_DEVICES, which --gpus sets) rank, and the mask is cut into as many core sets as it holds for them: a one-GPU server under an 8-core job mask on a node of four GPUs gets three cores and an API core, and the mask holds two such sets. The log line names the rule that sized the plan. Short of that too, or with a mask too small to leave a CPU unpinned, nothing is pinned and a warning says why. Servers on one GPU (two MIG instances share its address) would take the same CPUs, so the first always or siblings server holds /tmp/mstar-gpu-<PCI address>.lock for its life and a second one pins nothing, with a warning. Every server that pins, a CPU number included, also holds /tmp/mstar-core-<CPU>.lock for each physical core it pins (named by the core’s first CPU, with its pid inside): one that finds a core held logs the holder’s pid and takes the mask’s next core set, or pins nothing with a warning when none is free. The locks go with the process however it exits, and a file naming a pid that has exited is taken over with an info line. They only reach servers that share /tmp, so servers in separate containers on one GPU or under one mask still take the same CPUs. A split config’s second worker can also take a core that a server on a GPU without allowed CPUs takes under the same mask. Threads and pools a pinned thread starts keep the process mask (mstar.utils.threads), and the API server’s intra-op and encode pools and FFmpeg’s per-decode threads (media_limits.video_decode_threads) are sized from the CPUs the pins leave (CpuPlan in mstar.utils.cpus). Each worker and the API server log their CPUs and why. See Host idle states for measurements. Read by knobs.step_thread_core; no effect where os.sched_setaffinity is missing (macOS, Windows).

MSTAR_STEP_CORE_SET

set by the API server

Which of the mask’s core sets the API server locked under step_thread_core (cpus.claim_cores), for the conductor and workers it starts: they build the same plan, but only the API server knows which set was free. Not meant to be set by hand. Read by cpus.devices_plan.

MSTAR_IDLE_HOLD

config’s knobs: idle_hold, else off

qos has the API server open /dev/cpu_dma_latency, write a 10 µs latency and keep the fd open for its life, which keeps every CPU of the host out of idle states slower than that (C1 and C1E stay allowed). The log says what happened, and a device that is absent or not writable only costs the hold: startup carries on. Containers usually hide the device. The request is host-wide and holds cores awake whether or not the server is busy, so it spends power on idle hosts; it is off by default. auto placement stands down while it is set: with every server core held awake by a SCHED_IDLE spinner (the stand-in where the device is hidden), one core’s hardware threads lost to no pin and three separate cores were level with it. Read by knobs.idle_hold, applied by cpus.hold_idle.

MSTAR_DECODE_BUCKETS

1,2,4,8,16,32,48,64,96,128

The decode batch sizes a model whose LLM node is a SampledTokenDecoder captures as CUDA graphs, read at import in mstar/model/components/sampled_token_decoder.py; other decoders keep their own ladders. A decode step runs at the smallest bucket that holds its rows, so the ladder bounds both the rows per step and the graph memory; narrow it (for example 1,2,4,8,16,32) on a memory-tight GPU or for an A/B against the former ladder. MSTAR_QWEN35_DECODE_BUCKETS overrides it for Qwen3.5 alone.

MSTAR_LAZY_PUBLISH

1

Publish resource state (KV pages, positions) only for the requests whose frame or completion carries it: finalize_batch keeps a cheap per-request snapshot and the worker finishes it on demand. 0 builds the full publication for every request on every step, the old behaviour; it was a quarter of the GPU thread’s host time at batch 32 in a decode loop that never reads it.

MSTAR_MAX_CONSECUTIVE_SPEC_STEPS

1024

Cap on back-to-back speculative steps before the leader yields to other ready work. 0 turns speculation off. A value that is not an integer is ignored with a warning when the worker starts.

MSTAR_LAUNCH_WAIT_MS

5

Milliseconds the speculative submitter holds off the GIL waiting for the GPU thread to reach the launch before it carries on. A value that is not a number, is nan or inf, or is past threading.TIMEOUT_MAX seconds is ignored with a warning when the worker starts and 5 is used; a negative value is raised to 0 with a warning.

MSTAR_KV_CHAIN_LAZY_STEPS

config’s knobs: kv_chain_lazy_steps, else 1

How many decode steps of sampled ids the KV manager holds back before it extends the requests’ prefix chains (the per-page keys the prefix cache is indexed by), one extend per request per batch of steps instead of one per step. The per-request Python of that extension was 0.3 to 0.4 ms a step at 128 rows. A page key then appears up to that many steps late and a later commit indexes it, as the single-request path already allows; a request that finishes in between may leave its last page unindexed. The default 1 extends on every step: 16 measured as noise at c64/c128 on 0.8B (the extension is a small part of the stop check’s tail).

MSTAR_WORKSPACE_BUFFER_MB

512

The size each FlashInfer attention workspace starts at, per (plan label, slot), read by workspace_bytes (mstar/engine/resources/attn/workspace.py). A floor: a plan at startup that asks for more grows that workspace to what it asked for. Raise it for an eager step that wants more than startup’s plans left it, which raises with the bytes it asked for, since a captured graph holds the workspace’s address by then. A ragged resource that declares workspace_mb starts there instead, and this doesn’t reach it: LTX-2.5’s twelve workspaces hold 1.5 GiB at its 128, and would hold 6 GiB at 512 and 24 GiB at 2048 if this applied.

MSTAR_KV_PINNED_INDPTRS

config’s knobs: kv_pinned_indptrs, else 0

The KV plan writes FlashInfer’s index arrays (qo_indptr, kv_indptr, kv_indices, last_page_len) into a ring of page-locked host buffers and the decode wrapper copies the indices straight into its captured device buffer from there, so the copies FlashInfer’s plan issues are really asynchronous. From pageable memory the driver stages each one and FlashInfer makes the indices copy blocking. No measurable gain at c32-c128 on 0.8B, so it is off by default. The cost is one CUDA event per plan, which a buffer set waits on before it is rewritten.

MSTAR_GC_GEN0_THRESHOLD

config’s knobs: gc_gen0_threshold, else 0

The generation-0 garbage-collection threshold a worker sets after warmup (0 keeps CPython’s 700). A decode step at 128 rows keeps more than 700 young container objects alive at its peak, so at 700 a collection runs almost every step and escalates to a full one (about 10 ms) every few hundred steps; the time lands in whichever span is allocating. 50000 removed them on Qwen3.5 0.8B at c=128 (an 8% shorter decode step), at the cost of holding at most that many young objects between collections. Read by knobs.gc_gen0_threshold.

MSTAR_RESIDENT_PLANNING

config’s knobs: resident_planning, else auto

auto runs a node’s captured decode steps off per-request tables on the device where the submodule names resident walks (NodeSubmodule.resident_walks) and every resource of the node can (Resource.resident_refusal); the worker log says why a node that names walks stays on the host. off plans every step on the host. The tables are sized at startup, not by a setting: twice the walk’s largest capture batch in rows, plus one, since a request that finds no row takes the least recently run one’s. A row has a cell per page of the KV pool’s max_seq_len (the model’s context, or resources: <kv>: max_seq_len; not the config’s top-level key), at most the pool’s pages, plus a few, 8 bytes each. Read by knobs.resident_planning.

MSTAR_KV_PLAN_CACHE

config’s knobs: kv_plan_cache, else 0

The KV manager plans a captured decode step off the previous step’s plan when the batch is the same rows in the same order and nothing but that step’s commit touched the streams: every length is one more, pages change only where a row crossed a page boundary, and the five FlashInfer index arrays come from numpy instead of a per-row Python pass. The first version measured a loss at c64/c128 on 0.8B (batches change every step or two there, and a hit still rebuilt the views); the reworked path keeps the views lazy and is off by default until measured. 0 plans every step from the streams. 2 runs both and logs a difference, for checking a run.

MSTAR_DEVICE_LOOPBACK

config’s knobs: device_loopback, else 1

Whether a decode node may keep its loop-back token on the device: the sampler writes each request’s last token to a slot-addressed master and the next step reads its input ids from it, so the worker routes the signal with no tensor (no per-request store, hold or cleanup per step). Only nodes that declare the read (DeviceLoopback: the decode walks of Qwen3.5, Higgs Audio, Qwen3-ASR and Whisper, which also keeps its timestamp rule state in the sampler’s declared loop-back state) and only when the sampler has graph buffers; the sampler keeps the masters for those nodes alone, so a node that takes its tokens on the host does not pay the eager-step master write. 0 keeps every node on the per-request tensor path.

MSTAR_INGRAPH_DECODE_ROPE

config’s knobs: ingraph_decode_rope, else 0

A captured Qwen3.5 text prefill or mixed step builds its rotary tables inside the graph from the position resource’s planned positions instead of building them before the replay (LLMSubmodule._rope_in_graph); a captured decode step always does. Experiment knob, measured and gated on its own.

MSTAR_SAMPLER_INGRAPH_SCATTER

config’s knobs: sampler_ingraph_scatter, else 1

The captured sampler gathers its RNG offsets from the slot masters and scatters the advanced offsets and the sampled tokens back inside the graph, so a replayed decode step issues none of those launches on the gpu thread afterwards (each one is a GIL release the main thread’s postprocess wins). Padding rows, and the rows a step drops (SamplerStep.kept_rids), write a trash master row. The offset and last-token masters, which the captured graphs index, are allocated once for MSTAR_SAMPLER_SLOTS requests, and a request takes its row when the runner admits it (SamplerResource.reserve): past them it waits, as for a recurrent state slot, until a removed request frees one. Only for a node that reads its tokens back on the device (Qwen3.5, Qwen3-ASR, Whisper) with the device loop-back on, on one rank: above one, where admission doesn’t decide, the masters grow and the scatter runs after the replay, as with 0.

MSTAR_SAMPLER_SLOTS

config’s knobs: sampler_slots, else 4096

Request rows of the sampler masters a captured graph indexes when MSTAR_SAMPLER_INGRAPH_SCATTER is on (at least the node’s largest capture batch): the most requests admitted at once on such a node; the rest wait to be admitted. Read by knobs.sampler_slots.

MSTAR_RUST_SEND_TIMING

unset

1 makes the Rust runtime print, every 500 calls, how long the worker’s per-step send spends in its own work against the whole call (which includes taking the GIL back when the send had to release it). A diagnostic for the host-overhead work; off by default.

MSTAR_SPEC_PEEK_FOR_FAIRNESS

1

Yield the speculation chain only when another (node, walk) is actually ready right now; 0 uses the consecutive-step cap alone.

MSTAR_PHASE_TIMING

0

N > 0: every N iterations log per-phase p50/p95/mean of the worker main loop (speculate, await_gpu, submit_spec, …).

MSTAR_KV_DEBUG_ASSERTS

0

1: after every admit, commit, reset_request, remove_request and change to an admitted request’s pages, check the KV page bookkeeping against the streams holding the pages, the reservations against the free and evictable pages, and each commit against its request’s layout. A recurrent snapshot’s key is checked against the KV chain’s key at its boundary. Logs a label left under its footprint at removal. Tests and debugging only.

Compilation#

Variable

Default

Meaning

MSTAR_RECOMPILE_LIMIT

84

Dynamo’s recompile limit, clamped to [8, 256]. Raise it if the logs show hit config.recompile_limit.

MSTAR_KERNEL_TABLES

unset

Directories of tuned kernel-config tables (os.pathsep-separated), searched before the package’s mstar/kernels/tables (read by mstar.kernels.table.table_dirs()). python -m benchmark.kernels.sweep ... --out DIR writes one, so a sweep can be served and measured before it is committed. Set it the same way on every node, so every rank of a lockstep instance picks the same configs.

MSTAR_COMPILE_LEVEL

1

How much a captured forward’s compile tunes; mstar serve -O sets it (read by mstar.engine.accelerator_graph_runner.compile_level()). 0: torch.compile’s default mode. 1: coordinate descent tuning. 2: max-autotune-no-cudagraphs. A capture config that names its own compile_mode keeps it.

MSTAR_COMPILE_DETERMINISTIC

config’s knobs: compile_deterministic, else 0

1: Qwen3.5’s decode capture compiles in Inductor’s deterministic mode, as its prefill, mixed and vision captures always do, so greedy output reproduces across server processes and the cold start shrinks, at the cost of one-row decode speed. Only Qwen3.5’s text walks read it (knobs.compile_deterministic).

TRITON_PTXAS_BLACKWELL_PATH

torch’s ptxas, when needed

Triton’s own variable: the ptxas it runs for sm_100 and up. ptxas before 13.1.115, including the 13.1.80 that Triton 3.7.1 bundles, miscompiles some integer floor divisions on sm_103 (Inductor’s fused interleaved mRoPE table came out wrong on B300). Unset, the conductor points its workers at torch’s torch/bin/ptxas when a compute capability 10.x GPU is present, Triton’s bundled ptxas predates 13.1.115 and torch’s does not; it then also sets TORCH_COMPILE_CACHE_KEY_TAG, since Inductor’s graph cache would otherwise reuse kernels compiled by the bundled ptxas (decided by mstar.platforms.ptxas.blackwell_ptxas_env()). A value you set is kept, and the cache tag then names the ptxas you chose, so a cache another ptxas wrote is not reused. A TORCH_COMPILE_CACHE_KEY_TAG you set gets -ptxas-blackwell-<version> appended.

TORCHINDUCTOR_CACHE_DIR

$XDG_CACHE_HOME/mstar/torchinductor (~/.cache/mstar/torchinductor when unset)

torch’s own variable; M* sets this default when importing mstar.engine (_default_compile_caches), so compiled graphs and autotune picks survive a restart that keeps the cache directory. An exported value wins, and a home that can’t be written keeps torch’s /tmp/torchinductor_<user>. Point it at an empty directory to measure a cold start (Triton’s kernels follow it unless TRITON_CACHE_DIR is exported; FlashInfer’s JIT cache stays warm). See Restarts and the compile cache.

TRITON_CACHE_DIR

$TORCHINDUCTOR_CACHE_DIR/triton

Triton’s own variable, defaulted by the same function once TORCHINDUCTOR_CACHE_DIR is resolved (exported or M*’s default; left unset when that falls back to torch’s). An exported value wins. Holds Triton’s compiled kernels, Inductor’s and those compiled outside it; without it, where a kernel lands depends on whether Inductor compiled first.