mstar.model.nemotron_duplex.realtime_api

mstar.model.nemotron_duplex.realtime_api#

OpenAI Realtime-API-compatible WebSocket server for the Nemotron VoiceChat duplex model (Phase C).

This is a thin protocol layer over the model’s streaming duplex inference (Phase B / NemotronDuplexModel.create_stream). It speaks a subset of the OpenAI Realtime API events so existing Realtime clients can talk to it:

client -> server:

  • session.update (config: voice, output/ input formats)

  • input_audio_buffer.append (base64 PCM16 mono @ input_sample_rate)

  • input_audio_buffer.commit / clear

  • response.create / response.cancel

server -> client:

  • session.created / session.updated

  • response.created

  • response.audio.delta (base64 PCM16 @ 24k -> here 22.05k)

  • response.audio_transcript.delta (agent text, streamed)

  • response.audio.done / response.done

  • error

The model is full-duplex: it consumes user audio continuously and emits agent audio+text as it generates, so we run in a server-VAD-like continuous mode — every appended audio chunk is fed to the stream and any generated agent audio/transcript is pushed back as deltas. Turn boundaries (barge-in / end of utterance) come from the model’s own RNN-T turn signals when available.

Run: python -m mstar.model.nemotron_duplex.realtime_api --host 0.0.0.0 --port 8000 (requires fastapi + uvicorn). The heavy model streaming lives in the model; this file only does event framing.

Functions

build_app(model[, api_key])

Build a FastAPI app exposing /v1/realtime for the given loaded model.

main()

Classes

RealtimeConnection(model, send_json[, ...])

Per-WebSocket session: bridges Realtime events to a model duplex stream.
class mstar.model.nemotron_duplex.realtime_api.RealtimeConnection(model, send_json, send_defaults=None)[source]#

Bases: object

Per-WebSocket session: bridges Realtime events to a model duplex stream.

stream is a DuplexStream from NemotronDuplexModel.create_stream (Phase B), exposing:

  • append_audio(pcm_int16_bytes) -> list[Output] where each Output has .audio (int16 bytes @ output_sr) and/or .text (str delta),

  • reset().

Parameters:

send_defaults (dict | None)

async handle(event)[source]#
Parameters:

event (dict)

async start()[source]#
mstar.model.nemotron_duplex.realtime_api.build_app(model, api_key=None)[source]#

Build a FastAPI app exposing /v1/realtime for the given loaded model. With api_key, the handshake needs Authorization: Bearer <key>.

Parameters:

api_key (str | None)

mstar.model.nemotron_duplex.realtime_api.main()[source]#