mstar.model.multimodal

Order-preserving multimodal prompt adapter.

Intake hands models an ordered list of PromptPart — the user’s text and attachments as written — instead of a bag of per-modality lists. prefill_plan() turns that into the ordered sequence of text spans and attachments to prefill.

A model renders its prompt once with the tokenizer’s own placeholders where the attachments belong, tokenizes it once, and reads the placement back with find_media_spans() / split_around_spans(). One tokenizer call, so no BPE merge is cut at a span boundary, and a model says only how its prompt renders — never how to take one apart.

Prompt building and schedule building both call prefill_plan(), so they cannot disagree about the spans; check_plan() fails the request if a rendered prompt ever does.

Functions

chat_turns(parts)	Group parts into turns, starting a new one at every role change.
check_attachments(parts, counts)	Fail loudly when the layout and the attachments that arrived disagree.
check_plan(plan, spans, n_text)	Fail loudly when the rendered prompt disagrees with the plan.
find_media_spans(input_ids, specs)	Locate every placeholder run in input_ids, in token order.
messages_from_parts(parts[, default_system])	One OpenAI-style message per turn of parts, for a chat template to take as is.
parts_from_modalities(input_modalities[, ...])	Rebuild the parts for a layout, filling the text slots from texts.
prefill_plan(parts, *[, leading_text])	Order the prefill segments for parts.
split_around_spans(input_ids, spans)	Return the text between the spans, in order, dropping empty pieces.

Classes

MediaSpan(modality, index, start, stop)	A <|x_start|> pad* <|x_end|> run located in a tokenized prompt.
PromptPart(modality[, text, index, role])	One element of a prompt, in request order.
Turn(role, parts)	The parts of consecutive messages of one role.

class mstar.model.multimodal.MediaSpan(modality, index, start, stop)

Bases: object

A <|x_start|> pad* <|x_end|> run located in a tokenized prompt.

Bounds are sentinel-inclusive: the modality walk emits those embeddings itself, alongside the encoder output.

Parameters:

• modality (str)

• index (int)

• start (int)

• stop (int)

index: int

modality: str

start: int

stop: int

class mstar.model.multimodal.PromptPart(modality, text=None, index=0, role=None)

Bases: object

One element of a prompt, in request order.

index is the position within its own modality’s list, so a media part addresses tensors[f"{modality}_inputs"][index]. role is the chat role of the message the part came from, None from entrypoints with no messages.

Parameters:

• modality (str)

• text (str | None)

• index (int)

• role (str | None)

index: int = 0

modality: str

role: str | None = None

text: str | None = None

class mstar.model.multimodal.Turn(role, parts)

Bases: NamedTuple

The parts of consecutive messages of one role.

Parameters:

• role (str)

• parts (list[PromptPart])

parts: list[PromptPart]

Alias for field number 1

role: str

Alias for field number 0

mstar.model.multimodal.chat_turns(parts)

Group parts into turns, starting a new one at every role change.

A part with no role is the user’s, so a request from an entrypoint with no messages renders as the one user turn it always did.

Parameters:

parts (list[PromptPart])

Return type:

list[Turn]

mstar.model.multimodal.check_attachments(parts, counts)

Fail loudly when the layout and the attachments that arrived disagree.

check_plan() compares the plan against the prompt it rendered, but both sides come from the same parts — a layout claiming two images renders two placeholders and scans two spans whatever was uploaded. Only the caller knows how many attachments the request actually carried, so it checks here, at intake, where the failure can still be answered with a 400.

Parameters:

• parts (list[PromptPart])

• counts (dict[str, int])

Return type:

None

mstar.model.multimodal.check_plan(plan, spans, n_text)

Fail loudly when the rendered prompt disagrees with the plan.

A mismatch means an attachment was dropped or reordered between intake and tokenization — the failure this adapter exists to prevent.

Parameters:

• plan (list[PromptPart])

• spans (list[MediaSpan])

• n_text (int)

Return type:

None

mstar.model.multimodal.find_media_spans(input_ids, specs)

Locate every placeholder run in input_ids, in token order.

specs maps a modality to its (start_id, pad_id, end_id) triple. Indices run per modality, so the nth image span addresses the nth image.

Parameters:

• input_ids (Tensor)

• specs (dict[str, tuple[int, int, int]])

Return type:

list[MediaSpan]

mstar.model.multimodal.messages_from_parts(parts, default_system=None)

One OpenAI-style message per turn of parts, for a chat template to take as is.

An attachment is an empty {"type": modality, modality: ""} item: the template writes its placeholder there, in the turn it belongs to, and the tokenized prompt is scanned for it. default_system leads only when no part is a system message’s, wherever that message sits: a template writes every system message it is given, so both would reach the model.

Parameters:

• parts (list[PromptPart])

• default_system (str | None)

Return type:

list[dict]

mstar.model.multimodal.parts_from_modalities(input_modalities, texts=None, roles=None)

Rebuild the parts for a layout, filling the text slots from texts.

input_modalities is one entry per part, in order, and is the layout every consumer plans from. Rebuilding from it beats carrying a second copy of the ordering that could drift. Unfilled text slots carry None, which is enough to plan with. roles is one per part, in the same order.

Parameters:

• input_modalities (list[str])

• texts (Iterable[str] | str | None)

• roles (Iterable[str | None] | None)

Return type:

list[PromptPart]

mstar.model.multimodal.prefill_plan(parts, *, leading_text=True)

Order the prefill segments for parts.

Adjacent text collapses into one segment: the rendered prompt puts it in one contiguous run. A trailing segment always exists — the template’s turn close — even with no user text there. leading_text says whether the template opens with one too; BAGEL’s generation prompt does not.

Text segments are numbered in order, so index addresses the model’s own list of spans, and each carries its text (None when only the template contributes it).

Parameters:

• parts (list[PromptPart])

• leading_text (bool)

Return type:

list[PromptPart]

mstar.model.multimodal.split_around_spans(input_ids, spans)

Return the text between the spans, in order, dropping empty pieces.

Two adjacent attachments leave nothing between them, and a zero-length walk has nothing to embed. prefill_plan() drops the same pieces.

Parameters:

• input_ids (Tensor)

• spans (list[MediaSpan])

Return type:

list[Tensor]