mstar.model.chatterbox.config

Contents

mstar.model.chatterbox.config#

Configuration for Chatterbox and Chatterbox-Turbo.

Resemble publishes no config.json for either checkpoint: every hyper parameter lives in the reference package’s Python sources (T3Config, LLAMA_CONFIGS, the S3Gen constructor arguments) and, for Turbo, in a stale training yaml. The dataclasses below are that specification, written down once, with the two shipped variants as constructors. Every shape they declare is checked against the checkpoint when the weights are loaded.

Sources (read, not copied): chatterbox/models/t3/modules/t3_config.py, chatterbox/models/t3/llama_configs.py, chatterbox/tts.py, chatterbox/tts_turbo.py, chatterbox/models/s3gen/s3gen.py, chatterbox/models/voice_encoder/config.py, chatterbox/models/s3tokenizer/s3tokenizer.py.

Classes

ChatterboxConfig(variant, t3, voice_encoder, ...)

GenerationDefaults([temperature, top_p, ...])

S3GenCFMConfig([sigma_min, t_scheduler, ...])

S3GenConfig([sample_rate, vocab_size, ...])

S3 speech tokens -> mel (flow matching) -> waveform (HiFT).

S3GenEncoderConfig([input_size, ...])

UpsampleConformerEncoder: 6 blocks at 25 Hz, x2 upsample, 4 blocks at 50 Hz.

S3GenEstimatorConfig([in_channels, ...])

ConditionalDecoder: causal 1-D UNet with transformer blocks.

S3GenHiFTConfig([in_channels, ...])

S3GenMelConfig([n_fft, num_mels, ...])

S3GenXVectorConfig([feat_dim, ...])

CAMPPlus speaker encoder (80-bin Kaldi fbank, 16 kHz).

S3TokenizerConfig([n_mels, n_fft, hop_size, ...])

S3TokenizerV2 (25 Hz): Whisper-style log-mel front end, FSMN attention encoder, finite-scalar quantizer.

T3BackboneConfig([kind, hidden_size, ...])

The causal transformer behind T3.

T3Config(backbone, text_vocab_size, ...)

VoiceEncoderConfig([num_mels, sample_rate, ...])

class mstar.model.chatterbox.config.ChatterboxConfig(variant: 'str' = 'chatterbox', t3: 'T3Config' = <factory>, voice_encoder: 'VoiceEncoderConfig' = <factory>, s3_tokenizer: 'S3TokenizerConfig' = <factory>, s3gen: 'S3GenConfig' = <factory>, generation: 'GenerationDefaults' = <factory>, t3_weights: 'str' = 't3_cfg.safetensors', s3gen_weights: 'str' = 's3gen.safetensors', voice_encoder_weights: 'str' = 've.safetensors', text_tokenizer_file: 'str' = 'tokenizer.json', builtin_voice_file: 'str' = 'conds.pt', cangjie_file: 'str' = '', default_language: 'str | None' = None, enc_cond_seconds: 'float' = 6.0, dec_cond_seconds: 'float' = 10.0, normalize_reference_loudness: 'bool' = False, reference_target_lufs: 'float' = -27.0, trailing_silence_tokens: 'int' = 0, max_text_tokens: 'int' = 512, voice_cache_size: 'int' = 64, max_new_tokens_limit: 'int | None' = None, stream_first_chunk_tokens: 'int' = 15, stream_chunk_tokens: 'int' = 50, stream_chunk_growth: 'float' = 2.0, stream_max_chunk_tokens: 'int' = 200, stream_mel_cache_frames: 'int' = 8, stream_context_tokens: 'int' = 25, s3gen_compile: 'bool' = False, s3gen_compile_mode: 'str' = 'default', s3gen_frame_bucket: 'int' = 64, s3gen_graphs: 'bool' = True, s3gen_graph_stages: 'str' = 'solve, encoder, vocoder', s3gen_estimator_dtype: 'str' = 'float16', t3_prefill_graphs: 'bool' = True, s3gen_max_batch_size: 'int' = 8)[source]#

Bases: object

Parameters:

  • variant (str)

  • t3 (T3Config)

  • voice_encoder (VoiceEncoderConfig)

  • s3_tokenizer (S3TokenizerConfig)

  • s3gen (S3GenConfig)

  • generation (GenerationDefaults)

  • t3_weights (str)

  • s3gen_weights (str)

  • voice_encoder_weights (str)

  • text_tokenizer_file (str)

  • builtin_voice_file (str)

  • cangjie_file (str)

  • default_language (str | None)

  • enc_cond_seconds (float)

  • dec_cond_seconds (float)

  • normalize_reference_loudness (bool)

  • reference_target_lufs (float)

  • trailing_silence_tokens (int)

  • max_text_tokens (int)

  • voice_cache_size (int)

  • max_new_tokens_limit (int | None)

  • stream_first_chunk_tokens (int)

  • stream_chunk_tokens (int)

  • stream_chunk_growth (float)

  • stream_max_chunk_tokens (int)

  • stream_mel_cache_frames (int)

  • stream_context_tokens (int)

  • s3gen_compile (bool)

  • s3gen_compile_mode (str)

  • s3gen_frame_bucket (int)

  • s3gen_graphs (bool)

  • s3gen_graph_stages (str)

  • s3gen_estimator_dtype (str)

  • t3_prefill_graphs (bool)

  • s3gen_max_batch_size (int)

builtin_voice_file: str = 'conds.pt'#
cangjie_file: str = ''#
classmethod chatterbox()[source]#
Return type:

ChatterboxConfig

dec_cond_seconds: float = 10.0#
default_language: str | None = None#
enc_cond_seconds: float = 6.0#
classmethod from_model_path(model_path_hf)[source]#

Pick the variant from the repo id (ResembleAI/chatterbox-turbo).

Parameters:

model_path_hf (str)

Return type:

ChatterboxConfig

classmethod from_variant(variant)[source]#
Parameters:

variant (str)

Return type:

ChatterboxConfig

generation: GenerationDefaults#
property is_multilingual: bool#
property is_turbo: bool#
max_new_tokens_limit: int | None = None#
max_text_tokens: int = 512#
classmethod multilingual()[source]#

Chatterbox Multilingual (reference mtl_tts.py): the same S3Gen, voice encoder, built-in voice and sampling as the English model; the T3 weights t3_mtl23ls_v2.safetensors and the grapheme tokenizer with a language token.

Return type:

ChatterboxConfig

normalize_reference_loudness: bool = False#
reference_target_lufs: float = -27.0#
s3_tokenizer: S3TokenizerConfig#
s3gen: S3GenConfig#
s3gen_compile: bool = False#
s3gen_compile_mode: str = 'default'#
s3gen_estimator_dtype: str = 'float16'#
s3gen_frame_bucket: int = 64#
s3gen_graph_stages: str = 'solve,encoder,vocoder'#
s3gen_graphs: bool = True#
s3gen_max_batch_size: int = 8#
s3gen_weights: str = 's3gen.safetensors'#
property sample_rate: int#
stream_chunk_growth: float = 2.0#
stream_chunk_tokens: int = 50#
stream_context_tokens: int = 25#
stream_first_chunk_tokens: int = 15#
stream_max_chunk_tokens: int = 200#
stream_mel_cache_frames: int = 8#
t3: T3Config#
t3_prefill_graphs: bool = True#
t3_weights: str = 't3_cfg.safetensors'#
text_tokenizer_file: str = 'tokenizer.json'#
trailing_silence_tokens: int = 0#
classmethod turbo()[source]#
Return type:

ChatterboxConfig

variant: str = 'chatterbox'#
voice_cache_size: int = 64#
voice_encoder: VoiceEncoderConfig#
voice_encoder_weights: str = 've.safetensors'#
class mstar.model.chatterbox.config.GenerationDefaults(temperature: 'float' = 0.8, top_p: 'float' = 1.0, top_k: 'int' = 0, min_p: 'float' = 0.05, repetition_penalty: 'float' = 1.2, cfg_weight: 'float' = 0.5, exaggeration: 'float' = 0.5, max_new_tokens: 'int' = 1000, n_cfm_timesteps: 'int' = 10, watermark: 'bool' = True)[source]#

Bases: object

Parameters:
cfg_weight: float = 0.5#
exaggeration: float = 0.5#
max_new_tokens: int = 1000#
min_p: float = 0.05#
n_cfm_timesteps: int = 10#
repetition_penalty: float = 1.2#
temperature: float = 0.8#
top_k: int = 0#
top_p: float = 1.0#
watermark: bool = True#
class mstar.model.chatterbox.config.S3GenCFMConfig(sigma_min: 'float' = 1e-06, t_scheduler: 'str' = 'cosine', inference_cfg_rate: 'float' = 0.7, n_timesteps: 'int' = 10)[source]#

Bases: object

Parameters:

  • sigma_min (float)

  • t_scheduler (str)

  • inference_cfg_rate (float)

  • n_timesteps (int)

inference_cfg_rate: float = 0.7#
n_timesteps: int = 10#
sigma_min: float = 1e-06#
t_scheduler: str = 'cosine'#
class mstar.model.chatterbox.config.S3GenConfig(sample_rate=24000, vocab_size=6561, token_mel_ratio=2, output_size=80, spk_embed_dim=192, mel=<factory>, encoder=<factory>, estimator=<factory>, cfm=<factory>, hift=<factory>, xvector=<factory>, meanflow=False, trim_fade_frames=480)[source]#

Bases: object

S3 speech tokens -> mel (flow matching) -> waveform (HiFT).

Parameters:
cfm: S3GenCFMConfig#
encoder: S3GenEncoderConfig#
estimator: S3GenEstimatorConfig#
hift: S3GenHiFTConfig#
meanflow: bool = False#
classmethod meanflow_distilled()[source]#
Return type:

S3GenConfig

mel: S3GenMelConfig#
output_size: int = 80#
sample_rate: int = 24000#
spk_embed_dim: int = 192#
classmethod standard()[source]#
Return type:

S3GenConfig

token_mel_ratio: int = 2#
trim_fade_frames: int = 480#
vocab_size: int = 6561#
xvector: S3GenXVectorConfig#
class mstar.model.chatterbox.config.S3GenEncoderConfig(input_size=512, output_size=512, attention_heads=8, linear_units=2048, num_blocks=6, num_up_blocks=4, pre_lookahead_len=3, up_stride=2, layer_norm_eps=1e-12)[source]#

Bases: object

UpsampleConformerEncoder: 6 blocks at 25 Hz, x2 upsample, 4 blocks at 50 Hz.

Parameters:

  • input_size (int)

  • output_size (int)

  • attention_heads (int)

  • linear_units (int)

  • num_blocks (int)

  • num_up_blocks (int)

  • pre_lookahead_len (int)

  • up_stride (int)

  • layer_norm_eps (float)

attention_heads: int = 8#
input_size: int = 512#
layer_norm_eps: float = 1e-12#
linear_units: int = 2048#
num_blocks: int = 6#
num_up_blocks: int = 4#
output_size: int = 512#
pre_lookahead_len: int = 3#
up_stride: int = 2#
class mstar.model.chatterbox.config.S3GenEstimatorConfig(in_channels=320, out_channels=80, channels=(256,), attention_head_dim=64, n_blocks=4, num_mid_blocks=12, num_heads=8, causal=True)[source]#

Bases: object

ConditionalDecoder: causal 1-D UNet with transformer blocks.

Parameters:

  • in_channels (int)

  • out_channels (int)

  • channels (tuple[int, ...])

  • attention_head_dim (int)

  • n_blocks (int)

  • num_mid_blocks (int)

  • num_heads (int)

  • causal (bool)

attention_head_dim: int = 64#
causal: bool = True#
channels: tuple[int, ...] = (256,)#
in_channels: int = 320#
n_blocks: int = 4#
num_heads: int = 8#
num_mid_blocks: int = 12#
out_channels: int = 80#
class mstar.model.chatterbox.config.S3GenHiFTConfig(in_channels: 'int' = 80, base_channels: 'int' = 512, nb_harmonics: 'int' = 8, sampling_rate: 'int' = 24000, nsf_alpha: 'float' = 0.1, nsf_sigma: 'float' = 0.003, nsf_voiced_threshold: 'float' = 10.0, upsample_rates: 'tuple[int, ...]' = (8, 5, 3), upsample_kernel_sizes: 'tuple[int, ...]' = (16, 11, 7), istft_n_fft: 'int' = 16, istft_hop_len: 'int' = 4, resblock_kernel_sizes: 'tuple[int, ...]' = (3, 7, 11), resblock_dilation_sizes: 'tuple[tuple[int, ...], ...]' = ((1, 3, 5), (1, 3, 5), (1, 3, 5)), source_resblock_kernel_sizes: 'tuple[int, ...]' = (7, 7, 11), source_resblock_dilation_sizes: 'tuple[tuple[int, ...], ...]' = ((1, 3, 5), (1, 3, 5), (1, 3, 5)), lrelu_slope: 'float' = 0.1, audio_limit: 'float' = 0.99, f0_cond_channels: 'int' = 512)[source]#

Bases: object

Parameters:
audio_limit: float = 0.99#
base_channels: int = 512#
f0_cond_channels: int = 512#
in_channels: int = 80#
istft_hop_len: int = 4#
istft_n_fft: int = 16#
lrelu_slope: float = 0.1#
nb_harmonics: int = 8#
nsf_alpha: float = 0.1#
nsf_sigma: float = 0.003#
nsf_voiced_threshold: float = 10.0#
resblock_dilation_sizes: tuple[tuple[int, ...], ...] = ((1, 3, 5), (1, 3, 5), (1, 3, 5))#
resblock_kernel_sizes: tuple[int, ...] = (3, 7, 11)#
sampling_rate: int = 24000#
source_resblock_dilation_sizes: tuple[tuple[int, ...], ...] = ((1, 3, 5), (1, 3, 5), (1, 3, 5))#
source_resblock_kernel_sizes: tuple[int, ...] = (7, 7, 11)#
property upsample_factor: int#
upsample_kernel_sizes: tuple[int, ...] = (16, 11, 7)#
upsample_rates: tuple[int, ...] = (8, 5, 3)#
class mstar.model.chatterbox.config.S3GenMelConfig(n_fft: 'int' = 1920, num_mels: 'int' = 80, sample_rate: 'int' = 24000, hop_size: 'int' = 480, win_size: 'int' = 1920, fmin: 'float' = 0.0, fmax: 'float' = 8000.0)[source]#

Bases: object

Parameters:
fmax: float = 8000.0#
fmin: float = 0.0#
hop_size: int = 480#
n_fft: int = 1920#
num_mels: int = 80#
sample_rate: int = 24000#
win_size: int = 1920#
class mstar.model.chatterbox.config.S3GenXVectorConfig(feat_dim=80, embedding_size=192, growth_rate=32, bn_size=4, init_channels=128, block_num_layers=(12, 24, 16), block_kernel_sizes=(3, 3, 3), block_dilations=(1, 2, 2))[source]#

Bases: object

CAMPPlus speaker encoder (80-bin Kaldi fbank, 16 kHz).

Parameters:

  • feat_dim (int)

  • embedding_size (int)

  • growth_rate (int)

  • bn_size (int)

  • init_channels (int)

  • block_num_layers (tuple[int, ...])

  • block_kernel_sizes (tuple[int, ...])

  • block_dilations (tuple[int, ...])

block_dilations: tuple[int, ...] = (1, 2, 2)#
block_kernel_sizes: tuple[int, ...] = (3, 3, 3)#
block_num_layers: tuple[int, ...] = (12, 24, 16)#
bn_size: int = 4#
embedding_size: int = 192#
feat_dim: int = 80#
growth_rate: int = 32#
init_channels: int = 128#
class mstar.model.chatterbox.config.S3TokenizerConfig(n_mels=128, n_fft=400, hop_size=160, sample_rate=16000, n_state=1280, n_head=20, n_layer=6, fsmn_kernel_size=31, fsq_levels=3, fsq_dim=8, token_rate=25, max_frames_per_pass=3000)[source]#

Bases: object

S3TokenizerV2 (25 Hz): Whisper-style log-mel front end, FSMN attention encoder, finite-scalar quantizer.

Parameters:

  • n_mels (int)

  • n_fft (int)

  • hop_size (int)

  • sample_rate (int)

  • n_state (int)

  • n_head (int)

  • n_layer (int)

  • fsmn_kernel_size (int)

  • fsq_levels (int)

  • fsq_dim (int)

  • token_rate (int)

  • max_frames_per_pass (int)

property codebook_size: int#
fsmn_kernel_size: int = 31#
fsq_dim: int = 8#
fsq_levels: int = 3#
hop_size: int = 160#
max_frames_per_pass: int = 3000#
n_fft: int = 400#
n_head: int = 20#
n_layer: int = 6#
n_mels: int = 128#
n_state: int = 1280#
sample_rate: int = 16000#
token_rate: int = 25#
class mstar.model.chatterbox.config.T3BackboneConfig(kind='llama', hidden_size=1024, num_hidden_layers=30, num_attention_heads=16, num_key_value_heads=16, head_dim=64, intermediate_size=4096, norm_eps=1e-05, max_position_embeddings=131072, rope_theta=500000.0, rope_scaling=<factory>)[source]#

Bases: object

The causal transformer behind T3.

kind selects the block: "llama" (RMSNorm, SwiGLU, RoPE through the position resource) for Chatterbox, "gpt2" (LayerNorm, GELU-tanh MLP, learned absolute positions looked up on the planned position ids) for Turbo.

Parameters:

  • kind (str)

  • hidden_size (int)

  • num_hidden_layers (int)

  • num_attention_heads (int)

  • num_key_value_heads (int)

  • head_dim (int)

  • intermediate_size (int)

  • norm_eps (float)

  • max_position_embeddings (int)

  • rope_theta (float)

  • rope_scaling (dict)

classmethod gpt2_medium()[source]#
Return type:

T3BackboneConfig

head_dim: int = 64#
hidden_size: int = 1024#
intermediate_size: int = 4096#
property is_gpt2: bool#
kind: str = 'llama'#
classmethod llama_520m()[source]#
Return type:

T3BackboneConfig

max_position_embeddings: int = 131072#
norm_eps: float = 1e-05#
num_attention_heads: int = 16#
num_hidden_layers: int = 30#
num_key_value_heads: int = 16#
rope_scaling: dict#
rope_theta: float = 500000.0#
class mstar.model.chatterbox.config.T3Config(backbone: 'T3BackboneConfig' = <factory>, text_vocab_size: 'int' = 704, speech_vocab_size: 'int' = 8194, start_text_token: 'int' = 255, stop_text_token: 'int' = 0, start_speech_token: 'int' = 6561, stop_speech_token: 'int' = 6562, max_text_tokens: 'int' = 2048, max_speech_tokens: 'int' = 4096, learned_pos_emb: 'bool' = True, speaker_embed_size: 'int' = 256, speech_cond_prompt_len: 'int' = 150, use_perceiver_resampler: 'bool' = True, perceiver_num_queries: 'int' = 32, perceiver_num_heads: 'int' = 4, emotion_adv: 'bool' = True, speech_head_bias: 'bool' = False, duplicate_bos_in_prefill: 'bool' = True)[source]#

Bases: object

Parameters:

  • backbone (T3BackboneConfig)

  • text_vocab_size (int)

  • speech_vocab_size (int)

  • start_text_token (int)

  • stop_text_token (int)

  • start_speech_token (int)

  • stop_speech_token (int)

  • max_text_tokens (int)

  • max_speech_tokens (int)

  • learned_pos_emb (bool)

  • speaker_embed_size (int)

  • speech_cond_prompt_len (int)

  • use_perceiver_resampler (bool)

  • perceiver_num_queries (int)

  • perceiver_num_heads (int)

  • emotion_adv (bool)

  • speech_head_bias (bool)

  • duplicate_bos_in_prefill (bool)

backbone: T3BackboneConfig#
property cond_len: int#

speaker, prompt (resampled or raw), emotion.

Type:

Conditioning tokens in front of the text

duplicate_bos_in_prefill: bool = True#
emotion_adv: bool = True#
classmethod english()[source]#
Return type:

T3Config

property hidden_size: int#
learned_pos_emb: bool = True#
max_speech_tokens: int = 4096#
max_text_tokens: int = 2048#
classmethod multilingual()[source]#

The 23-language checkpoint: the English model with a 2454-token grapheme text vocabulary (reference T3Config.multilingual).

Return type:

T3Config

perceiver_num_heads: int = 4#
perceiver_num_queries: int = 32#
speaker_embed_size: int = 256#
speech_cond_prompt_len: int = 150#
speech_head_bias: bool = False#
property speech_pos_table_size: int#
speech_vocab_size: int = 8194#
start_speech_token: int = 6561#
start_text_token: int = 255#
stop_speech_token: int = 6562#
stop_text_token: int = 0#
property text_pos_table_size: int#
text_vocab_size: int = 704#
classmethod turbo()[source]#
Return type:

T3Config

use_perceiver_resampler: bool = True#
class mstar.model.chatterbox.config.VoiceEncoderConfig(num_mels: 'int' = 40, sample_rate: 'int' = 16000, hidden_size: 'int' = 256, num_layers: 'int' = 3, speaker_embed_size: 'int' = 256, n_fft: 'int' = 400, hop_size: 'int' = 160, win_size: 'int' = 400, fmin: 'float' = 0.0, fmax: 'float' = 8000.0, mel_power: 'float' = 2.0, partial_frames: 'int' = 160, partials_rate: 'float' = 1.3, min_coverage: 'float' = 0.8, final_relu: 'bool' = True)[source]#

Bases: object

Parameters:
final_relu: bool = True#
fmax: float = 8000.0#
fmin: float = 0.0#
hidden_size: int = 256#
hop_size: int = 160#
mel_power: float = 2.0#
min_coverage: float = 0.8#
n_fft: int = 400#
num_layers: int = 3#
num_mels: int = 40#
partial_frames: int = 160#
partials_rate: float = 1.3#
sample_rate: int = 16000#
speaker_embed_size: int = 256#
win_size: int = 400#
Contents