mstar.model.omnilingual_asr.audio

mstar.model.omnilingual_asr.audio#

Waveform preprocessing for Omnilingual ASR, in the reference’s order.

The reference pipeline (omnilingual_asr models/inference/pipeline.py and datasets/utils/audio.py) decodes the file at its own rate (libsndfile, float32), resamples every channel to 16 kHz with torchaudio.functional.resample, averages the channels, normalizes the utterance to zero mean and unit variance (layer_norm over the whole waveform, eps 1e-5) and rejects clips over 40 s. load_waveform and preprocess do the same, with the same ops, so an upload reaches the encoder bit-identical to the reference’s input.

Functions

check_length(num_samples)

decode_file(path)

Float32 samples at the file's own rate: (time,) for mono, (time, channels) otherwise, as libsndfile (the reference's decoder) returns them.

load_waveform(path)

A file's 16 kHz mono float32 waveform, not yet normalized.

normalize(waveform)

Zero mean, unit variance over the whole utterance, in float32 (the reference's layer_norm(waveform, waveform.shape)).

preprocess(waveform)

16 kHz mono waveform -> the encoder input: length-checked, normalized.

to_16k_mono(audio, sample_rate)

Resample each channel to 16 kHz, then average the channels, as the reference does (resample, then convert_to_mono's in-order sum and divide).
mstar.model.omnilingual_asr.audio.check_length(num_samples)[source]#
Parameters:

num_samples (int)

Return type:

None

mstar.model.omnilingual_asr.audio.decode_file(path)[source]#

Float32 samples at the file’s own rate: (time,) for mono, (time, channels) otherwise, as libsndfile (the reference’s decoder) returns them.

libsndfile through soundfile first; then, for a WAV file, a reader here that scales samples as libsndfile does, so a server without soundfile still decodes WAV exactly; then FFmpeg through torchcodec for the containers neither reads. The two library decoders stop just past MAX_AUDIO_SECONDS and refuse a longer file with AudioTooLongError, whatever its header declares.

Parameters:

path (str)

Return type:

tuple[Tensor, int]

mstar.model.omnilingual_asr.audio.load_waveform(path)[source]#

A file’s 16 kHz mono float32 waveform, not yet normalized.

Parameters:

path (str)

Return type:

Tensor

mstar.model.omnilingual_asr.audio.normalize(waveform)[source]#

Zero mean, unit variance over the whole utterance, in float32 (the reference’s layer_norm(waveform, waveform.shape)).

Parameters:

waveform (Tensor)

Return type:

Tensor

mstar.model.omnilingual_asr.audio.preprocess(waveform)[source]#

16 kHz mono waveform -> the encoder input: length-checked, normalized.

Parameters:

waveform (Tensor)

Return type:

Tensor

mstar.model.omnilingual_asr.audio.to_16k_mono(audio, sample_rate)[source]#

Resample each channel to 16 kHz, then average the channels, as the reference does (resample, then convert_to_mono’s in-order sum and divide).

Parameters:
Return type:

Tensor