mstar.utils.fast_tokenize#
Tokenize a long prompt as parallel chunks, byte-identical to one encode.
An 8k-token prompt costs the fast tokenizer ~15 ms single-threaded, a quarter of the request’s time to first token. The Rust backend encodes a batch in parallel, so a long text is cut into chunks and encoded as one batch; the cuts are chosen so every pre-token stays whole, which keeps the BPE merges, and so the ids, identical to encoding the whole text.
The cut rule assumes a GPT-style pre-tokenizer (the Qwen2 family, GPT-2, Llama 3): a space starts a pre-token unless it sits inside a run of whitespace, so a cut is placed only right before a space whose previous character is not whitespace. Runs of whitespace, newline runs and the punctuation-plus-newline pre-tokens are never cut, and special-token strings contain no space. The identity is checked by test/modular/test_fast_tokenize.py against the real tokenizer over the benchmark prompts.
Functions
|
Indices where |
|
Token ids of |
|
- mstar.utils.fast_tokenize.cut_points(text, chunk_chars=DEFAULT_CHUNK_CHARS, max_chunks=DEFAULT_MAX_CHUNKS)[source]#
Indices where
textmay be cut: each is a space that starts a whitespace run, at or after the next chunk target. Empty when the text is short or no safe cut exists.
- mstar.utils.fast_tokenize.encode_ids(tokenizer, text, chunk_chars=DEFAULT_CHUNK_CHARS, max_chunks=DEFAULT_MAX_CHUNKS)[source]#
Token ids of
textwith the tokenizer’s defaults for special tokens: a long text goes through the Rust backend as a batch of chunks, a short one (or a tokenizer without a Rust backend) throughtokenizeritself.