mstar.model.nemotron_duplex.config

Config dataclasses for NVIDIA NemotronLabs VoiceChat-11B (duplex).

The base checkpoint (nvidia/NVIDIA-NemotronLabs-VoiceChat-11B) is a single model.safetensors composite of three stages, whose per-stage architecture dims come from the Spark repo’s per-folder config.json files:

• nano/ — Nemotron-H hybrid Mamba-2 / attention / MLP LLM (9B)

• eartts/ — Gemma3-text “talker” transformer + RVQ codec (audio out)

• (stt) — Fast-Conformer streaming encoder + RNN-T head (audio in)

Only the fields M* actually needs are translated here. Numbers are taken verbatim from the published nano/config.json / eartts/config.json; see the module docstrings for the source keys. Anything still unverified against the NeMo reference (nemotron-labs-voicechat branch of NVIDIA-NeMo/Speech) is flagged with TODO(verify).

Classes

CodecConfig([base_hidden_size, ...])	RVQ audio codec (config.json -> model.speech_generation.model.codec_config).
ConformerSTTConfig([d_model, num_heads, ...])	Audio-input stage (config.json -> model.stt.model.perception).
EarTTSConfig([hidden_size, ...])	Audio-output stage (config.json -> model.speech_generation.model.tts_config).
NanoConfig([num_hidden_layers, hidden_size, ...])	Nemotron-H backbone (nano/config.json, model_type='nemotron_h').
NemotronDuplexConfig([nano, eartts, stt, ...])	Top-level config bundling the stages plus generation defaults.
RnntConfig([pred_hidden, pred_rnn_layers, ...])	RNN-T transcription head (config.json -> _rnnt_merge_info).

class mstar.model.nemotron_duplex.config.CodecConfig(base_hidden_size=384, channel_mult=(1, 2, 4), codebook_size=1024, groups=1, hop_length=4, kernel_size=7, latent_size=512, n_fft=16, num_blocks=3, num_quantizers=31, rates=(7, 7, 9), wav_to_token_ratio=1764, max_magnitude=100.0)

Bases: object

RVQ audio codec (config.json -> model.speech_generation.model.codec_config).

The encoder/decoder conv+ConvNeXt-block stacks are derived from these fields (channel schedule from base_hidden_size * channel_mult, strides from rates, num_blocks blocks per stage) rather than hardcoded — see encoder_layers / decoder_layers.

Parameters:

• base_hidden_size (int)

• channel_mult (tuple[int, ...])

• codebook_size (int)

• groups (int)

• hop_length (int)

• kernel_size (int)

• latent_size (int)

• n_fft (int)

• num_blocks (int)

• num_quantizers (int)

• rates (tuple[int, ...])

• wav_to_token_ratio (int)

• max_magnitude (float)

base_hidden_size: int = 384

channel_mult: tuple[int, ...] = (1, 2, 4)

property channels: list[int]

codebook_size: int = 1024

decoder_layers()

Return type:

list[tuple]

encoder_layers()

(kind, weight_shape, stride, transpose) per encoder layer.

Return type:

list[tuple]

groups: int = 1

hop_length: int = 4

kernel_size: int = 7

latent_size: int = 512

max_magnitude: float = 100.0

n_fft: int = 16

num_blocks: int = 3

num_quantizers: int = 31

rates: tuple[int, ...] = (7, 7, 9)

property spec_channels: int

Encoder input channels = (n_fft/2 + 1) real + imag = 2*(n_fft/2+1).

wav_to_token_ratio: int = 1764

class mstar.model.nemotron_duplex.config.ConformerSTTConfig(d_model=1024, num_heads=8, num_layers=24, feat_in=128, ff_expansion_factor=4, conv_kernel_size=9, subsampling_conv_channels=256, subsampling_factor=8, att_context_size=(70, 0), output_dim=4480, streaming_lookahead_frames=1, sample_rate=16000, n_fft=512, window_size=0.025, window_stride=0.01, n_mels=128, preemph=0.97, mag_power=2.0)

Bases: object

Audio-input stage (config.json -> model.stt.model.perception).

Fast-Conformer streaming encoder + mel preprocessor. Encodes 16 kHz speech into LLM-space embeddings (output_dim); the RNN-T head (RnntConfig) emits the streaming user transcription.

Parameters:

• d_model (int)

• num_heads (int)

• num_layers (int)

• feat_in (int)

• ff_expansion_factor (int)

• conv_kernel_size (int)

• subsampling_conv_channels (int)

• subsampling_factor (int)

• att_context_size (tuple[int, int])

• output_dim (int)

• streaming_lookahead_frames (int)

• sample_rate (int)

• n_fft (int)

• window_size (float)

• window_stride (float)

• n_mels (int)

• preemph (float)

• mag_power (float)

att_context_size: tuple[int, int] = (70, 0)

conv_kernel_size: int = 9

d_model: int = 1024

feat_in: int = 128

property ff_dim: int

ff_expansion_factor: int = 4

property head_dim: int

property hop_length: int

mag_power: float = 2.0

n_fft: int = 512

n_mels: int = 128

num_frames(num_samples)

Encoder frames for num_samples of audio: the mel frame count (get_seq_len), then the subsampling’s calc_length.

Parameters:

num_samples (int)

Return type:

int

num_heads: int = 8

num_layers: int = 24

output_dim: int = 4480

preemph: float = 0.97

sample_rate: int = 16000

streaming_lookahead_frames: int = 1

subsampling_conv_channels: int = 256

subsampling_factor: int = 8

property subsampling_flat: int

property win_length: int

window_size: float = 0.025

window_stride: float = 0.01

class mstar.model.nemotron_duplex.config.EarTTSConfig(hidden_size=1152, intermediate_size=4608, num_hidden_layers=28, num_attention_heads=16, num_key_value_heads=16, head_dim=72, kv_head_dim=128, sliding_window=7500, rms_norm_eps=1e-06, query_pre_attn_scalar=256.0, rope_theta_full=1000000.0, rope_theta_local=10000.0, sliding_window_pattern=6, num_quantizers=31, codebook_size=1024, code_dim=512, mog_num_predictions=1024, mog_low_rank=64, mog_min_log_std=-4.0, mog_num_layers=3, mog_intermediate_size=4608, mog_eps=1e-06, mog_exponent=3.0, inference_num_iter=8, inference_guidance_scale=0.5, inference_noise_scale=0.8, inference_top_p=0.8, inference_seed=0, codec_chunk_frames=5, codec_left_context_frames=3, sample_rate=22050)

Bases: object

Audio-output stage (config.json -> model.speech_generation.model.tts_config).

A Gemma3-text “talker” transformer autoregressively emits RVQ codec codes (num_quantizers codebooks of codebook_size), then a mixture-of- gaussians (MoG) head over the code_dim RVQ latent space.

Parameters:

• hidden_size (int)

• intermediate_size (int)

• num_hidden_layers (int)

• num_attention_heads (int)

• num_key_value_heads (int)

• head_dim (int)

• kv_head_dim (int)

• sliding_window (int)

• rms_norm_eps (float)

• query_pre_attn_scalar (float)

• rope_theta_full (float)

• rope_theta_local (float)

• sliding_window_pattern (int)

• num_quantizers (int)

• codebook_size (int)

• code_dim (int)

• mog_num_predictions (int)

• mog_low_rank (int)

• mog_min_log_std (float)

• mog_num_layers (int)

• mog_intermediate_size (int)

• mog_eps (float)

• mog_exponent (float)

• inference_num_iter (int)

• inference_guidance_scale (float)

• inference_noise_scale (float)

• inference_top_p (float)

• inference_seed (int)

• codec_chunk_frames (int)

• codec_left_context_frames (int)

• sample_rate (int)

code_dim: int = 512

codebook_size: int = 1024

codec_chunk_frames: int = 5

codec_left_context_frames: int = 3

head_dim: int = 72

hidden_size: int = 1152

inference_guidance_scale: float = 0.5

inference_noise_scale: float = 0.8

inference_num_iter: int = 8

inference_seed: int = 0

inference_top_p: float = 0.8

intermediate_size: int = 4608

kv_head_dim: int = 128

mog_eps: float = 1e-06

mog_exponent: float = 3.0

mog_intermediate_size: int = 4608

mog_low_rank: int = 64

mog_min_log_std: float = -4.0

mog_num_layers: int = 3

mog_num_predictions: int = 1024

num_attention_heads: int = 16

num_hidden_layers: int = 28

num_key_value_heads: int = 16

num_quantizers: int = 31

query_pre_attn_scalar: float = 256.0

rms_norm_eps: float = 1e-06

rope_theta_full: float = 1000000.0

rope_theta_local: float = 10000.0

sample_rate: int = 22050

sliding_window: int = 7500

sliding_window_pattern: int = 6

class mstar.model.nemotron_duplex.config.NanoConfig(num_hidden_layers=56, hidden_size=4480, intermediate_size=15680, vocab_size=131072, rms_norm_eps=1e-05, tie_word_embeddings=False, hybrid_override_pattern='M-M-M-MM-M-M-M*-M-M-M*-M-M-M-M*-M-M-M-M*-M-MM-M-M-M-M-M-', num_attention_heads=40, num_key_value_heads=8, head_dim=128, attention_bias=False, max_position_embeddings=131072, mamba_num_heads=128, mamba_head_dim=80, ssm_state_size=128, conv_kernel=4, n_groups=8, mamba_expand=2, mamba_chunk_size=1, time_step_rank=256, time_step_limit=(0.0, inf), use_conv_bias=True, mamba_proj_bias=False, mamba_hidden_act='silu', mlp_bias=False, mlp_hidden_act='relu2', combined_embed_dim=4480, has_function_head=True, bos_token_id=1, eos_token_id=12, pad_token_id=0, pad_pair_token_id=12)

Bases: object

Nemotron-H backbone (nano/config.json, model_type='nemotron_h').

hybrid_override_pattern is the per-layer type string: M = Mamba-2 mixer, * = self-attention (NoPE), - = MLP (squared-ReLU). Its length equals num_hidden_layers.

Parameters:

• num_hidden_layers (int)

• hidden_size (int)

• intermediate_size (int)

• vocab_size (int)

• rms_norm_eps (float)

• tie_word_embeddings (bool)

• hybrid_override_pattern (str)

• num_attention_heads (int)

• num_key_value_heads (int)

• head_dim (int)

• attention_bias (bool)

• max_position_embeddings (int)

• mamba_num_heads (int)

• mamba_head_dim (int)

• ssm_state_size (int)

• conv_kernel (int)

• n_groups (int)

• mamba_expand (int)

• mamba_chunk_size (int)

• time_step_rank (int)

• time_step_limit (tuple[float, float])

• use_conv_bias (bool)

• mamba_proj_bias (bool)

• mamba_hidden_act (str)

• mlp_bias (bool)

• mlp_hidden_act (str)

• combined_embed_dim (int)

• has_function_head (bool)

• bos_token_id (int)

• eos_token_id (int)

• pad_token_id (int)

• pad_pair_token_id (int)

attention_bias: bool = False

bos_token_id: int = 1

combined_embed_dim: int = 4480

property conv_dim: int

conv_kernel: int = 4

property d_inner: int

eos_token_id: int = 12

has_function_head: bool = True

head_dim: int = 128

hidden_size: int = 4480

hybrid_override_pattern: str = 'M-M-M-MM-M-M-M*-M-M-M*-M-M-M-M*-M-M-M-M*-M-MM-M-M-M-M-M-'

intermediate_size: int = 15680

property layer_types: list[str]

"mamba" | "attention" | "mlp".

Type:

Per-layer kind

mamba_chunk_size: int = 1

mamba_expand: int = 2

mamba_head_dim: int = 80

mamba_hidden_act: str = 'silu'

mamba_num_heads: int = 128

mamba_proj_bias: bool = False

max_position_embeddings: int = 131072

mlp_bias: bool = False

mlp_hidden_act: str = 'relu2'

n_groups: int = 8

num_attention_heads: int = 40

property num_attention_layers: int

num_hidden_layers: int = 56

num_key_value_heads: int = 8

property num_mamba_layers: int

pad_pair_token_id: int = 12

pad_token_id: int = 0

rms_norm_eps: float = 1e-05

ssm_state_size: int = 128

tie_word_embeddings: bool = False

time_step_limit: tuple[float, float] = (0.0, inf)

time_step_rank: int = 256

use_conv_bias: bool = True

vocab_size: int = 131072

class mstar.model.nemotron_duplex.config.NemotronDuplexConfig(nano=<factory>, eartts=<factory>, stt=<factory>, codec=<factory>, rnnt=<factory>, temperature=0.7, top_p=0.9, repetition_penalty=1.0, ignore_eos=False, max_new_tokens=2048, agent_text_weight=1.0, user_audio_weight=1.0, function_weight=2.0, use_function_head=True, predict_user_text=False, text_pad_id=12, text_bos_id=1, text_eos_id=2)

Bases: object

Top-level config bundling the stages plus generation defaults.

Use from_pretrained() to populate every sub-config from the checkpoint’s config.json; the dataclass defaults mirror that file so dummy-mode / weightless construction still works.

Parameters:

• nano (NanoConfig)

• eartts (EarTTSConfig)

• stt (ConformerSTTConfig)

• codec (CodecConfig)

• rnnt (RnntConfig)

• temperature (float)

• top_p (float)

• repetition_penalty (float)

• ignore_eos (bool)

• max_new_tokens (int)

• agent_text_weight (float)

• user_audio_weight (float)

• function_weight (float)

• use_function_head (bool)

• predict_user_text (bool)

• text_pad_id (int)

• text_bos_id (int)

• text_eos_id (int)

agent_text_weight: float = 1.0

codec: CodecConfig

eartts: EarTTSConfig

property eos_token_id: int

classmethod from_pretrained(ckpt_dir)

Build from the checkpoint’s config.json (NeMo VoiceChat layout).

Only the stages whose components read from config are populated here; the rest fall back to dataclass defaults. Extend as modules migrate to config-driven construction.

Parameters:

ckpt_dir (str)

Return type:

NemotronDuplexConfig

function_weight: float = 2.0

ignore_eos: bool = False

max_new_tokens: int = 2048

nano: NanoConfig

predict_user_text: bool = False

repetition_penalty: float = 1.0

rnnt: RnntConfig

stt: ConformerSTTConfig

temperature: float = 0.7

text_bos_id: int = 1

text_eos_id: int = 2

text_pad_id: int = 12

top_p: float = 0.9

use_function_head: bool = True

user_audio_weight: float = 1.0

property vocab_size: int

class mstar.model.nemotron_duplex.config.RnntConfig(pred_hidden=640, pred_rnn_layers=2, encoder_hidden=1024, joint_hidden=640, vocab_size=1024, activation='relu')

Bases: object

RNN-T transcription head (config.json -> _rnnt_merge_info).

Parameters:

• pred_hidden (int)

• pred_rnn_layers (int)

• encoder_hidden (int)

• joint_hidden (int)

• vocab_size (int)

• activation (str)

activation: str = 'relu'

encoder_hidden: int = 1024

joint_hidden: int = 640

property num_classes: int

pred_hidden: int = 640

pred_rnn_layers: int = 2

vocab_size: int = 1024