mstar.api_server.media_io

Media decode/encode helpers shared by the native and OpenAI-compatible APIs.

Two directions:

• Inbound — turn media referenced by an OpenAI-style request (data: URLs, http(s) URLs, or base64 blobs) into files under the API server’s upload_dir, so model.load_image / load_audio / load_video can read them by path. This is the same contract /generate already uses for multipart uploads.

• Outbound — wrap raw model audio output (16-bit PCM, no container header) into a real audio container (WAV by default) and encode image bytes (PNG) as a data: URL for OpenAI chat image output.

Only stdlib + numpy are required. mp3 / flac / ogg encoding is opt-in and degrades to WAV when the optional soundfile backend is unavailable, so the base install stays slim.

Functions

audio_formats([stream, sample_rate])	The response_format values this frontend can produce.
check_audio_format(fmt[, stream, sample_rate])	fmt lower-cased (wav when unset); ValueError when it cannot be produced.
check_ref_audio(path, decoder)	Open the reference clip at path the way the model's load_audio will; on failure delete it and raise ValueError.
check_resample_rate(rate, target)	Refuse a file rate outside 1..MAX_SAMPLE_RATE Hz or whose resampling to target would cost more than MAX_RESAMPLE_KERNEL (every real rate, pull-down rates included, passes).
decode_audio(path[, sample_rate, max_seconds])	Decode an audio file to a float32 mono numpy waveform at sample_rate, refused with AudioTooLongError past max_seconds.
decode_torchcodec_capped(path, sample_rate)	FFmpeg (torchcodec) decode of path: a float32 mono tensor at sample_rate, the same samples get_all_samples() returns.
modality_from_mime(mime)	Map a MIME type to one of our modality strings (image/audio/video).
mux_mp4_with_pcm16(video_mp4, pcm, sample_rate)	Mux an H.264 mp4 and raw interleaved 16-bit PCM into one mp4 with an AAC audio track (stream-copies the video).
pcm16_to_container(pcm, sample_rate[, fmt])	Encode raw 16-bit PCM into fmt.
pcm16_to_wav_bytes(pcm, sample_rate[, ...])	Wrap raw little-endian 16-bit PCM (the model's audio output) into a WAV blob.
png_to_data_url(png_bytes)	Encode PNG image bytes (the model's image output) as a data URL.
read_soundfile_capped(path[, max_seconds])	libsndfile decode of path: float32 mono at the file's own rate.
remove_uploads(file_paths, upload_dir)	Delete a request's files under upload_dir now.
resample_kernel_size(rate, target)	Floats in torchaudio's default sinc kernel resampling rate to target.
resolve_media_ref(ref, upload_dir, *[, ...])	Resolve a media reference from a request: a data URL, or an http(s) URL when remote fetch is allowed.
save_base64(b64, fmt, modality_hint, upload_dir)	Persist a bare base64 blob with a known fmt (e.g. "wav").
save_base64_audio(b64, upload_dir, decoder)	Persist bare base64 that passes check_ref_audio(); return its path, or None (leaving nothing behind) when it is not base64 or not loadable audio.
save_data_url(data_url, upload_dir)	Persist a data:<mime>;base64,<payload> URL.
save_remote_url(url, upload_dir[, timeout])	Download an http(s) URL into upload_dir.
schedule_upload_cleanup(file_paths, upload_dir)	remove_uploads() after UPLOAD_CLEANUP_DELAY_S, for a request that was submitted and may not have been read yet.
split_windows(audio, window_seconds[, ...])	Cut a waveform into consecutive windows of at most window_seconds; concatenating them gives the input back sample for sample.
torchcodec_samples_capped(path[, max_seconds])	AudioDecoder(path, **decoder_args)'s samples from the stream's first frame, as get_all_samples() returns them, decoding just past max_seconds of play time; refused with AudioTooLongError when the audio plays longer or holds more samples than that.
wav_stream_header(sample_rate[, ...])	A 44-byte WAV header with streaming (unknown-length) size fields.
window_stop(start, total, step, balanced)	Nominal end of the window starting at start: step samples on, or, balanced, an equal share of what is left over the fewest windows of at most step (so the last window is never a sliver).
write_wav(audio, path[, sample_rate])	Float waveform -> 16-bit PCM WAV file at path.

Classes

StreamResampler(rate_in, rate_out[, ...])

Resample a stream that arrives in pieces of any size (push), with the same output however it was split.

Exceptions

AudioRefusedError	Audio a decoder opened and refuses (too long, or an unsupported rate): callers re-raise it rather than fall back to another decoder.
AudioTooLongError(max_seconds)	Audio that decodes past the limit, whatever its header declares.
RefAudioTooLongError	A clip that opens as audio but runs past MAX_REF_AUDIO_SECONDS.

exception mstar.api_server.media_io.AudioRefusedError

Bases: ValueError

Audio a decoder opened and refuses (too long, or an unsupported rate): callers re-raise it rather than fall back to another decoder.

exception mstar.api_server.media_io.AudioTooLongError(max_seconds)

Bases: AudioRefusedError

Audio that decodes past the limit, whatever its header declares.

Parameters:

max_seconds (float)

status_code = 413

exception mstar.api_server.media_io.RefAudioTooLongError

Bases: ValueError

A clip that opens as audio but runs past MAX_REF_AUDIO_SECONDS.

class mstar.api_server.media_io.StreamResampler(rate_in, rate_out, half_width=48, beta=10.0, rolloff=0.95)

Bases: object

Resample a stream that arrives in pieces of any size (push), with the same output however it was split. A Kaiser-windowed sinc lowpass at the lower Nyquist, applied polyphase: output k is centered on input time k * rate_in / rate_out, so timing is kept exactly. The passband is flat to 0.1 dB up to 0.9 of the lower Nyquist and 3 dB down at 0.94 (7.2 and 7.5 kHz at 24 to 16 kHz), close to what a 16 kHz recording holds; anything above that Nyquist, which would fold back, is down 46 dB or more, and 100 dB from 1.06 of it on. Each output waits for half a kernel of input past its center (72 samples, 3 ms, at 24 to 16 kHz); flush emits the rest as if the input went on as silence, ceil(n_in * rate_out / rate_in) samples in all, and starts over. The kernel has 2 * half_width * max(up, down) + 1 taps for the reduced ratio up/down, so callers bound the rates.

Parameters:

• rate_in (int)

• rate_out (int)

• half_width (int)

• beta (float)

• rolloff (float)

flush()

Return type:

ndarray

held(extra=0)

Output samples flush would still emit, after extra more input samples.

Parameters:

extra (int)

Return type:

int

push(x)

Parameters:

x (ndarray)

Return type:

ndarray

reset()

Return type:

None

mstar.api_server.media_io.audio_formats(stream=False, sample_rate=None)

The response_format values this frontend can produce.

A stream carries an open-ended WAV or bare PCM, so only those two; the compressed containers need the optional soundfile (the audio extra), and Opus also needs one of its native sample rates when sample_rate is given.

Parameters:

• stream (bool)

• sample_rate (int | None)

Return type:

tuple[str, …]

mstar.api_server.media_io.check_audio_format(fmt, stream=False, sample_rate=None)

fmt lower-cased (wav when unset); ValueError when it cannot be produced.

Parameters:

• fmt (str | None)

• stream (bool)

• sample_rate (int | None)

Return type:

str

mstar.api_server.media_io.check_ref_audio(path, decoder)

Open the reference clip at path the way the model’s load_audio will; on failure delete it and raise ValueError.

decoder is the library that load_audio uses: "soundfile", "torchcodec", or "soundfile+torchcodec" for a loader that tries libsndfile first and FFmpeg for what it cannot open, as Model.load_audio does. torchcodec decodes the clip, not just its header: some FFmpeg builds parse a container header (MAT4, MAT5) they cannot decode. A clip whose header rate is below MIN_REF_AUDIO_SAMPLE_RATE, or whose header duration is over MAX_REF_AUDIO_SECONDS, is refused before any decode; the decode itself stops just past MAX_REF_AUDIO_SECONDS, so a header that hides or understates the length cannot make it unbounded.

Parameters:

• path (str)

• decoder (str)

Return type:

None

mstar.api_server.media_io.check_resample_rate(rate, target)

Refuse a file rate outside 1..MAX_SAMPLE_RATE Hz or whose resampling to target would cost more than MAX_RESAMPLE_KERNEL (every real rate, pull-down rates included, passes).

Parameters:

• rate (int)

• target (int)

Return type:

None

mstar.api_server.media_io.decode_audio(path, sample_rate=16000, max_seconds=MAX_AUDIO_SECONDS)

Decode an audio file to a float32 mono numpy waveform at sample_rate, refused with AudioTooLongError past max_seconds.

The same decode as a model’s own uploads (load_speech_file): a file neither libsndfile nor FFmpeg reads raises ValueError (HTTP 400), while a missing FFmpeg stays the server’s error.

Parameters:

• path (str)

• sample_rate (int)

• max_seconds (float)

mstar.api_server.media_io.decode_torchcodec_capped(path, sample_rate, max_seconds=MAX_AUDIO_SECONDS)

FFmpeg (torchcodec) decode of path: a float32 mono tensor at sample_rate, the same samples get_all_samples() returns.

The decode stops just past max_seconds of play time, and the audio is refused with AudioTooLongError when it plays longer or holds more samples than that. The decoder resamples every frame to sample_rate, so what is counted is decoded output, not the rate or duration the container declares.

Parameters:

• path (str)

• sample_rate (int)

• max_seconds (float)

mstar.api_server.media_io.modality_from_mime(mime)

Map a MIME type to one of our modality strings (image/audio/video).

Parameters:

mime (str)

Return type:

str

mstar.api_server.media_io.mux_mp4_with_pcm16(video_mp4, pcm, sample_rate, num_channels=2, video_fps=None)

Mux an H.264 mp4 and raw interleaved 16-bit PCM into one mp4 with an AAC audio track (stream-copies the video). video_fps is the true frame rate of the content: when it differs from the container’s rate the video timestamps are rescaled (no re-encode) so playback runs at the real rate and stays in sync with the audio track. Requires the ffmpeg binary; raises on failure — callers fall back to the video-only mp4.

Parameters:

• video_mp4 (bytes)

• pcm (bytes)

• sample_rate (int)

• num_channels (int)

• video_fps (float | None)

Return type:

bytes

mstar.api_server.media_io.pcm16_to_container(pcm, sample_rate, fmt='wav')

Encode raw 16-bit PCM into fmt. Returns (bytes, mime_type).

wav and pcm use the stdlib (the bytes are already PCM_16); the compressed formats use the optional soundfile. A format this server cannot produce raises ValueError (see check_audio_format()) rather than coming back as WAV under another name.

Parameters:

• pcm (bytes)

• sample_rate (int)

• fmt (str)

Return type:

tuple[bytes, str]

mstar.api_server.media_io.pcm16_to_wav_bytes(pcm, sample_rate, num_channels=1)

Wrap raw little-endian 16-bit PCM (the model’s audio output) into a WAV blob.

Parameters:

• pcm (bytes)

• sample_rate (int)

• num_channels (int)

Return type:

bytes

mstar.api_server.media_io.png_to_data_url(png_bytes)

Encode PNG image bytes (the model’s image output) as a data URL.

Parameters:

png_bytes (bytes)

Return type:

str

mstar.api_server.media_io.read_soundfile_capped(path, max_seconds=MAX_AUDIO_SECONDS)

libsndfile decode of path: float32 mono at the file’s own rate.

Reads in blocks and counts the frames it decodes, so a header that hides or understates the length cannot make the read unbounded; past max_seconds it raises AudioTooLongError. A rate outside 1..MAX_SAMPLE_RATE is refused (AudioRefusedError) before any read. A container libsndfile cannot read raises soundfile’s own error.

Parameters:

• path (str)

• max_seconds (float)

Return type:

tuple[ndarray, int]

mstar.api_server.media_io.remove_uploads(file_paths, upload_dir)

Delete a request’s files under upload_dir now. A path elsewhere is left alone: no route puts one in file_paths today, and a file the server did not save is never the request’s to delete.

Parameters:

• file_paths (dict[str, list[str]] | None)

• upload_dir (Path)

Return type:

None

mstar.api_server.media_io.resample_kernel_size(rate, target)

Floats in torchaudio’s default sinc kernel resampling rate to target.

Parameters:

• rate (int)

• target (int)

Return type:

int

mstar.api_server.media_io.resolve_media_ref(ref, upload_dir, *, allow_remote=False)

Resolve a media reference from a request: a data URL, or an http(s) URL when remote fetch is allowed. Returns (modality, path).

A request cannot name a file on the server, not even one under upload_dir: plain paths and other URL schemes are refused without accessing client-named server files, so a caller can neither read files nor probe for them.

Parameters:

• ref (str)

• upload_dir (Path)

• allow_remote (bool)

Return type:

tuple[str, str]

mstar.api_server.media_io.save_base64(b64, fmt, modality_hint, upload_dir)

Persist a bare base64 blob with a known fmt (e.g. "wav").

Parameters:

• b64 (str)

• fmt (str)

• modality_hint (str)

• upload_dir (Path)

Return type:

tuple[str, str]

mstar.api_server.media_io.save_base64_audio(b64, upload_dir, decoder)

Persist bare base64 that passes check_ref_audio(); return its path, or None (leaving nothing behind) when it is not base64 or not loadable audio.

A path such as /etc/passwd is valid base64 too, so the alphabet cannot tell them apart; the decoder can. A clip that is audio but too long raises RefAudioTooLongError.

Parameters:

• b64 (str)

• upload_dir (Path)

• decoder (str)

Return type:

str | None

mstar.api_server.media_io.save_data_url(data_url, upload_dir)

Persist a data:<mime>;base64,<payload> URL. Returns (modality, path).

Parameters:

• data_url (str)

• upload_dir (Path)

Return type:

tuple[str, str]

mstar.api_server.media_io.save_remote_url(url, upload_dir, timeout=30.0)

Download an http(s) URL into upload_dir. Returns (modality, path).

Note: fetching arbitrary URLs has SSRF surface. Callers exposing this publicly should allowlist hosts or disable remote fetch (data-URL only).

Parameters:

• url (str)

• upload_dir (Path)

• timeout (float)

Return type:

tuple[str, str]

mstar.api_server.media_io.schedule_upload_cleanup(file_paths, upload_dir)

remove_uploads() after UPLOAD_CLEANUP_DELAY_S, for a request that was submitted and may not have been read yet.

Parameters:

• file_paths (dict[str, list[str]] | None)

• upload_dir (Path)

Return type:

None

mstar.api_server.media_io.split_windows(audio, window_seconds, sample_rate=16000, search_seconds=0.0, frame_seconds=0.02, balanced=False)

Cut a waveform into consecutive windows of at most window_seconds; concatenating them gives the input back sample for sample.

With search_seconds a cut is moved back from its nominal boundary to the end of the quietest frame_seconds frame within that span (the last such frame on a tie, so silence cuts at the boundary): the cut lands in a pause rather than on a word when there is one to find. balanced places the nominal boundaries as window_stop() does.

Parameters:

• window_seconds (float)

• sample_rate (int)

• search_seconds (float)

• frame_seconds (float)

• balanced (bool)

Return type:

list

mstar.api_server.media_io.torchcodec_samples_capped(path, max_seconds=MAX_AUDIO_SECONDS, **decoder_args)

AudioDecoder(path, **decoder_args)’s samples from the stream’s first frame, as get_all_samples() returns them, decoding just past max_seconds of play time; refused with AudioTooLongError when the audio plays longer or holds more samples than that.

Parameters:

• path (str)

• max_seconds (float)

mstar.api_server.media_io.wav_stream_header(sample_rate, num_channels=1, bits=16)

A 44-byte WAV header with streaming (unknown-length) size fields.

Used to stream TTS audio over a single HTTP response: emit this header, then 16-bit PCM frames as they arrive. The 0xFFFFFFFF placeholders signal an open-ended stream, which players and the OpenAI client’s stream_to_file handle.

Parameters:

• sample_rate (int)

• num_channels (int)

• bits (int)

Return type:

bytes

mstar.api_server.media_io.window_stop(start, total, step, balanced)

Nominal end of the window starting at start: step samples on, or, balanced, an equal share of what is left over the fewest windows of at most step (so the last window is never a sliver).

Parameters:

• start (int)

• total (int)

• step (int)

• balanced (bool)

Return type:

int

mstar.api_server.media_io.write_wav(audio, path, sample_rate=16000)

Float waveform -> 16-bit PCM WAV file at path. The inverse of reading PCM16 as int16 / 32768, so 16-bit audio comes back sample for sample.

Parameters:

• path (str)

• sample_rate (int)

Return type:

str