Supported Models

mstar ships the following model families. The table below summarizes the registered families, their registry key (the value of model: in a config YAML), and a representative Hugging Face identifier.

Registry keys live in mstar/model/registry.py (MODEL_REGISTRY / HF_MODELS).

Registered model families

Registry key	Example Hugging Face model ID	Description
bagel	ByteDance-Seed/BAGEL-7B-MoT	Unified multimodal model (text + image understanding and generation).
cosmos3	nvidia/Cosmos3-Nano	Cosmos3 world model: t2i/t2v/i2v/v2v diffusion, robot-action modes, opt-in sound.
cosmos3_super	nvidia/Cosmos3-Super	Cosmos3-Super (64B) variant of the above; TP/SP for multi-GPU serving.
orpheus	canopylabs/orpheus-3b-0.1-ft	TTS: Llama 3.2 3B LLM emitting audio tokens + SNAC 24 kHz decoder.
pi05	lerobot/pi05_base	Pi0.5 vision-language-action robotics model (ViT encoder + LLM + flow action expert).
qwen3_omni	Qwen/Qwen3-Omni-30B-A3B-Instruct	Omni-modal (text/image/audio/video in, text/audio out): Thinker + Talker + codec.
vjepa2	facebook/vjepa2-vitl-fpc64-256	V-JEPA 2 video encoder + masked predictor.
vjepa2_ac	vjepa2-ac-vitg	V-JEPA 2-AC encoder + action-conditioned predictor.
whisper_large (Beta)	openai/whisper-large-v3	Encoder-decoder ASR (audio in, transcript out). Beta / un-optimized.
higgs_audio (Beta)	bosonai/higgs-audio-v3-stt	Audio-tower + Qwen3 LLM speech-to-text. Beta / un-optimized.

Notes

• Models marked (Beta) are functionally supported but not yet performance-optimized; treat their throughput/latency as provisional.

• The IDs above are representative. You may use local paths or compatible variants.

• Some families accept multimodal input (image/audio/video); see the model’s process_prompt for the inputs it expects.

• To add a new family, see Adding a New Model.

Cosmos3 environment requirements

• flashinfer is required: it is the paged KV/attention backend used by the prefill, the captured CUDA graphs, and multi-request batches.

• The default denoise attention backend is dense_gen (Cosmos3Config.attention_backend), which runs bs=1 eager generation attention as one FlashAttention-3 varlen kernel from the fa3-fwd wheel. That wheel is ABI-tied to the installed torch/CUDA build (Hopper builds exist for at least torch 2.9 + cu12.8 and torch 2.11 + cu13.0); install the one matching your environment. When it is not importable, the engine logs a warning at startup and automatically falls back to the paged flashinfer backend — serving still works, only the bs=1 dense fast path is lost. model_kwargs.attention_backend: flashinfer in the config YAML selects the paged backend explicitly.

• Video-input requests (video-to-video, action inverse-dynamics) decode the conditioning clip with torchcodec; environments without it reject those requests at preprocessing (other modes are unaffected).

• Generated video containers are written with torchcodec’s VideoEncoder when available (torchcodec >= 0.9), otherwise with torchvision’s write_video, which needs the PyAV (av) package.

• Sound-enabled video responses mux the AAC track with the ffmpeg and ffprobe binaries, which must be on PATH (system packages, not pip-installable).

• The Wan-VAE decode dtype is gated on the cuDNN build: bf16 needs cuDNN >= 9.16 (fast Hopper bf16 conv3d); older cuDNN serves the decode in fp32/TF32 automatically.