mstar.utils.utf8_stream

Decode a model’s text chunks without splitting characters.

Models with a byte-level detokenizer (Whisper, Cosmos3) send each token’s raw bytes as its own text chunk, so one character can span two chunks, and decoding each chunk on its own turns both halves into U+FFFD. Stdlib only: the Python SDK imports this too. The Rust frontend’s twin is rust/server/src/utf8.rs.

Functions

decode_joined(parts)

A whole response's text chunks as one string: join, then decode once.

Classes

Utf8Stream()

One text stream's incremental decoder: feed returns the complete characters so far and holds back an incomplete tail, flush releases that tail when the stream ends (as U+FFFD if it never completed).

class mstar.utils.utf8_stream.Utf8Stream

Bases: object

One text stream’s incremental decoder: feed returns the complete characters so far and holds back an incomplete tail, flush releases that tail when the stream ends (as U+FFFD if it never completed).

feed(data)

Parameters:

data (bytes)

Return type:

str

flush()

Return type:

str

mstar.utils.utf8_stream.decode_joined(parts)

A whole response’s text chunks as one string: join, then decode once.

Parameters:

parts (Iterable[bytes])

Return type:

str