mstar.model.multimodal

Order-preserving multimodal prompt adapter.

Intake hands models an ordered list of PromptPart — the user’s text and attachments as written — instead of a bag of per-modality lists. prefill_plan() turns that into the ordered sequence of text spans and attachments to prefill.

A model renders its prompt once with the tokenizer’s own placeholders where the attachments belong, tokenizes it once, and reads the placement back with find_media_spans() / split_around_spans(). One tokenizer call, so no BPE merge is cut at a span boundary, and a model says only how its prompt renders — never how to take one apart.

Prompt building and schedule building both call prefill_plan(), so they cannot disagree about the spans; check_plan() fails the request if a rendered prompt ever does.

Functions

check_attachments(parts, counts)	Fail loudly when the layout and the attachments that arrived disagree.
check_plan(plan, spans, n_text)	Fail loudly when the rendered prompt disagrees with the plan.
find_media_spans(input_ids, specs)	Locate every placeholder run in input_ids, in token order.
parts_from_modalities(input_modalities[, texts])	Rebuild the parts for a layout, filling the text slots from texts.
prefill_plan(parts, *[, leading_text])	Order the prefill segments for parts.
split_around_spans(input_ids, spans)	Return the text between the spans, in order, dropping empty pieces.

Classes

MediaSpan(modality, index, start, stop)	A <|x_start|> pad* <|x_end|> run located in a tokenized prompt.
PromptPart(modality[, text, index])	One element of a prompt, in request order.

class mstar.model.multimodal.MediaSpan(modality, index, start, stop)

Bases: object

A <|x_start|> pad* <|x_end|> run located in a tokenized prompt.

Bounds are sentinel-inclusive: the modality walk emits those embeddings itself, alongside the encoder output.

Parameters:

• modality (str)

• index (int)

• start (int)

• stop (int)

index: int

modality: str

start: int

stop: int

class mstar.model.multimodal.PromptPart(modality, text=None, index=0)

Bases: object

One element of a prompt, in request order.

index is the position within its own modality’s list, so a media part addresses tensors[f"{modality}_inputs"][index].

Parameters:

• modality (str)

• text (str | None)

• index (int)

index: int = 0

modality: str

text: str | None = None

mstar.model.multimodal.check_attachments(parts, counts)

Fail loudly when the layout and the attachments that arrived disagree.

check_plan() compares the plan against the prompt it rendered, but both sides come from the same parts — a layout claiming two images renders two placeholders and scans two spans whatever was uploaded. Only the caller knows how many attachments the request actually carried, so it checks here, at intake, where the failure can still be answered with a 400.

Parameters:

• parts (list[PromptPart])

• counts (dict[str, int])

Return type:

None

mstar.model.multimodal.check_plan(plan, spans, n_text)

Fail loudly when the rendered prompt disagrees with the plan.

A mismatch means an attachment was dropped or reordered between intake and tokenization — the failure this adapter exists to prevent.

Parameters:

• plan (list[PromptPart])

• spans (list[MediaSpan])

• n_text (int)

Return type:

None

mstar.model.multimodal.find_media_spans(input_ids, specs)

Locate every placeholder run in input_ids, in token order.

specs maps a modality to its (start_id, pad_id, end_id) triple. Indices run per modality, so the nth image span addresses the nth image.

Parameters:

• input_ids (Tensor)

• specs (dict[str, tuple[int, int, int]])

Return type:

list[MediaSpan]

mstar.model.multimodal.parts_from_modalities(input_modalities, texts=None)

Rebuild the parts for a layout, filling the text slots from texts.

input_modalities is one entry per part, in order, and is the layout every consumer plans from. Rebuilding from it beats carrying a second copy of the ordering that could drift. Unfilled text slots carry None, which is enough to plan with.

Parameters:

• input_modalities (list[str])

• texts (Iterable[str] | str | None)

Return type:

list[PromptPart]

mstar.model.multimodal.prefill_plan(parts, *, leading_text=True)

Order the prefill segments for parts.

Adjacent text collapses into one segment: the rendered prompt puts it in one contiguous run. A trailing segment always exists — the template’s turn close — even with no user text there. leading_text says whether the template opens with one too; BAGEL’s generation prompt does not.

Text segments are numbered in order, so index addresses the model’s own list of spans, and each carries its text (None when only the template contributes it).

Parameters:

• parts (list[PromptPart])

• leading_text (bool)

Return type:

list[PromptPart]

mstar.model.multimodal.split_around_spans(input_ids, spans)

Return the text between the spans, in order, dropping empty pieces.

Two adjacent attachments leave nothing between them, and a zero-length walk has nothing to embed. prefill_plan() drops the same pieces.

Parameters:

• input_ids (Tensor)

• spans (list[MediaSpan])

Return type:

list[Tensor]