mstar.api_server.ws_frames#

Parsing large /generate/ws frames without holding the event loop.

json.loads, msgpack.unpackb and base64 decoding hold the GIL for the whole call, so a thread doesn’t free the loop; nor does pickling the frame across to a process, one more whole-frame call. A large frame is therefore handed to a worker process through a file, written from a thread in slices short enough that the loop gets the GIL between them; the worker parses it, saves each file’s data under the upload dir (decoding base64 as it goes) and sends back only the message, each saved file’s data a SavedUpload. A small frame stays on the loop, where the round trip costs more than the parse.

Taking the frame in stays on the loop: uvicorn and websockets assemble, unmask, decode and, with deflate on, inflate it there.

The pool’s processes re-import the main module, as every spawned process does: under mstar serve that is light, under the mstar-serve script it is the server’s entrypoint (torch included, a few hundred MB each).

Functions

parse(payload)

A frame's message: msgpack for bytes, JSON for text.

parse_frame(payload, upload_dir)

A frame's message.

save_entry(name, data, upload_dir[, prefix])

Save a file's data: raw bytes, or base64 text, by the decoder the data-URL paths use (unpadded or wrapped is fine; a bad alphabet is the client's error).

start()

Start the pool with the server, so its processes are starting before the first frame (they take a few seconds, spawned off the server's main module).

stop()

Classes

SavedUpload(path)

A file's data the pool already saved to path.

class mstar.api_server.ws_frames.SavedUpload(path)[source]#

Bases: object

A file’s data the pool already saved to path. The file is removed with this object unless claim takes it first, so a message refused before its files are taken leaves nothing behind.

Parameters:

path (str)

claim()[source]#
Return type:

str

mstar.api_server.ws_frames.parse(payload)[source]#

A frame’s message: msgpack for bytes, JSON for text.

Parameters:

payload (bytes | str)

async mstar.api_server.ws_frames.parse_frame(payload, upload_dir)[source]#

A frame’s message. A large frame parses in the pool, its files’ data already saved under upload_dir (SavedUpload). Raises what the parse raises, and OSError when it could not be handed off, or a pool process died with the frame in the pool: that fails every frame the pool held, on any socket, and a new pool takes the next. A pool found dead at the hand-off held nothing of this frame’s, which goes to a new pool.

Parameters:
mstar.api_server.ws_frames.save_entry(name, data, upload_dir, prefix=None)[source]#

Save a file’s data: raw bytes, or base64 text, by the decoder the data-URL paths use (unpadded or wrapped is fine; a bad alphabet is the client’s error). A failed write leaves nothing.

Parameters:
  • name (str)

  • upload_dir (Path)

  • prefix (str | None)

Return type:

Path

mstar.api_server.ws_frames.start()[source]#

Start the pool with the server, so its processes are starting before the first frame (they take a few seconds, spawned off the server’s main module).

Return type:

None

mstar.api_server.ws_frames.stop()[source]#
Return type:

None