mstar.worker.scheduling_policy#

Which (node, walk) a worker’s next step goes to.

The micro-scheduler asks a SchedulingPolicy two things: the order in which to try the (node, walk) keys that have ready rows (order), and whether a speculation chain gives the step after its own to other ready work (chain_yields). A key is the scheduler’s: a real walk, or a combined walk its rows share.

RoundRobin is the default: least recently scheduled first, and a chain yields only after it has had a step of its own. Priority serves keys a model or a deployment declares latency-sensitive (WalkPriority) first. Its cost falls on every other key’s steps, which wait while a higher one is ready, so a key’s max_wait bounds that wait: once a ready key has gone that many steps since it last ran, it goes first.

Functions

resolve_walk_priorities(entries, walk_nodes, ...)

Each (node, walk) the entries reach (walk_nodes: walk -> the nodes it runs), as the most specific entry that names it: a pair, then a walk, then a node.

Classes

Priority(priorities, max_waits)

Higher priorities first, round robin among equals and for an unlisted key (0); ahead of both, the keys that have waited their max_wait, longest waiting first.

ReadyPeek(*args, **kwargs)

What a policy reads off the scheduler.

RoundRobin()

SchedulingPolicy()

WalkPriority([node, walk, priority, max_wait])

One entry of a model's or a deployment's priorities (Model.get_walk_priorities, the serving config's scheduling.walk_priorities).

class mstar.worker.scheduling_policy.Priority(priorities, max_waits)[source]#

Bases: RoundRobin

Higher priorities first, round robin among equals and for an unlisted key (0); ahead of both, the keys that have waited their max_wait, longest waiting first.

A chain also gives up its first step to a ready key above its own or one that has waited its bound. Where round robin would ask anyway, its peek covers those keys too, so a pass makes at most one. A key’s wait is the chain steps this worker has taken since that key last ran, so one that comes ready after an idle spell may go first once.

Parameters:
  • priorities (Mapping[Key, int])

  • max_waits (Mapping[Key, int])

chain_yields(chain_key, consecutive, mid_prompt, peek)[source]#

Whether a speculation chain into chain_key gives the next step to other ready work. consecutive: the chain’s steps since it last yielded; mid_prompt: the chain is cutting a prompt into chunks.

order(ready, peek)[source]#

The keys of ready (each key’s ready rows, in scan order) in the order the scheduler tries them.

priority(key)[source]#
Parameters:

key (tuple[str, str])

Return type:

int

class mstar.worker.scheduling_policy.ReadyPeek(*args, **kwargs)[source]#

Bases: Protocol

What a policy reads off the scheduler.

batch_number: int#
chain_step_of: Mapping[tuple[str, str], int]#
chain_steps: int#
has_ready_excluding(exclude_target, among=None)[source]#

Whether a key other than exclude_target (and, given among, one of those) has a row the engine would take. Its readiness checks are real ones, so a policy asks only when it would yield on the answer.

Parameters:
Return type:

bool

node_and_walk_to_last_batch_num: Mapping[tuple[str, str], int]#
class mstar.worker.scheduling_policy.RoundRobin[source]#

Bases: SchedulingPolicy

chain_yields(chain_key, consecutive, mid_prompt, peek)[source]#

Whether a speculation chain into chain_key gives the next step to other ready work. consecutive: the chain’s steps since it last yielded; mid_prompt: the chain is cutting a prompt into chunks.

order(ready, peek)[source]#

The keys of ready (each key’s ready rows, in scan order) in the order the scheduler tries them.

class mstar.worker.scheduling_policy.SchedulingPolicy[source]#

Bases: ABC

abstractmethod chain_yields(chain_key, consecutive, mid_prompt, peek)[source]#

Whether a speculation chain into chain_key gives the next step to other ready work. consecutive: the chain’s steps since it last yielded; mid_prompt: the chain is cutting a prompt into chunks.

Parameters:
Return type:

bool

abstractmethod order(ready, peek)[source]#

The keys of ready (each key’s ready rows, in scan order) in the order the scheduler tries them.

Parameters:
Return type:

list[tuple[str, str]]

class mstar.worker.scheduling_policy.WalkPriority(node=None, walk=None, priority=0, max_wait=None)[source]#

Bases: NamedTuple

One entry of a model’s or a deployment’s priorities (Model.get_walk_priorities, the serving config’s scheduling.walk_priorities). It applies to every (node, walk) it names: a node’s walks, a walk’s nodes, or one pair. Where entries overlap, the most specific decides: a pair over a walk over a node.

priority: higher first, unlisted 0. max_wait: the most of the worker’s chain steps the key may go without running while it has ready rows, before it goes first; None falls back to the policy’s default, which may itself be none (no bound).

Parameters:
  • node (str | None)

  • walk (str | None)

  • priority (int)

  • max_wait (int | None)

max_wait: int | None#

Alias for field number 3

node: str | None#

Alias for field number 0

priority: int#

Alias for field number 2

walk: str | None#

Alias for field number 1

mstar.worker.scheduling_policy.resolve_walk_priorities(entries, walk_nodes, owner)[source]#

Each (node, walk) the entries reach (walk_nodes: walk -> the nodes it runs), as the most specific entry that names it: a pair, then a walk, then a node. Refuses an entry naming neither, a node or walk no walk runs, a pair the walk doesn’t run, a negative priority, a max_wait under 1, and two entries for the same target. owner names where the entries came from, in the error.

Parameters:
Return type:

dict[tuple[str, str], WalkPriority]