mstar.model.omnilingual_asr.tokenizer

The Omnilingual ASR character vocabulary and CTC decoding.

The tokenizer is a SentencePiece model whose pieces are single characters (the space is a literal ' ' piece, not ▁), plus <s>, <pad>, </s> (control) and <unk>. The reference decodes with fairseq2’s SentencePiece decoder, which concatenates pieces, renders control pieces as nothing and <unk> as `` ⁇ , and adds or strips no whitespace. So a lookup table reproduces it, and the ``.model file is read here directly (its protobuf wire format, pieces and types only) rather than through the sentencepiece package.

CTC decoding is the reference’s _apply_model_wav2vec2asr: argmax per frame, drop each frame equal to the one before it, then decode, which drops the blank (a control piece) along with the other specials.

Functions

ctc_collapse(ids)	Keep the first frame and every frame that differs from the one before it (the blank is dropped later, at decode).
ctc_greedy_ids(logits, num_frames)	Collapsed argmax ids over a request's num_frames valid frames.
read_sentencepiece_pieces(data)	(piece, type) per id from a serialized SentencePiece ModelProto.

Classes

CharTokenizer(pieces)

id -> text for a single-character SentencePiece vocabulary.

class mstar.model.omnilingual_asr.tokenizer.CharTokenizer(pieces)

Bases: object

id -> text for a single-character SentencePiece vocabulary.

Parameters:

pieces (list[tuple[str, int]])

decode(ids)

Return type:

str

classmethod from_file(path)

Parameters:

path (str | Path)

Return type:

CharTokenizer

property vocab_size: int

mstar.model.omnilingual_asr.tokenizer.ctc_collapse(ids)

Keep the first frame and every frame that differs from the one before it (the blank is dropped later, at decode).

Parameters:

ids (Tensor)

Return type:

Tensor

mstar.model.omnilingual_asr.tokenizer.ctc_greedy_ids(logits, num_frames)

Collapsed argmax ids over a request’s num_frames valid frames.

Parameters:

• logits (Tensor)

• num_frames (int)

Return type:

Tensor

mstar.model.omnilingual_asr.tokenizer.read_sentencepiece_pieces(data)

(piece, type) per id from a serialized SentencePiece ModelProto.

Parameters:

data (bytes)

Return type:

list[tuple[str, int]]