mstar.worker.scheduling_policy#
Which (node, walk) a worker’s next step goes to.
The micro-scheduler asks a SchedulingPolicy two things: the order in which to try
the (node, walk) keys that have ready rows (order), and whether a speculation
chain gives the step after its own to other ready work (chain_yields). A key is
the scheduler’s: a real walk, or a combined walk its rows share.
RoundRobin is the default: least recently scheduled first, and a chain yields
only after it has had a step of its own. Priority serves keys a model or a
deployment declares latency-sensitive (WalkPriority) first. Its cost falls on
every other key’s steps, which wait while a higher one is ready, so a key’s
max_wait bounds that wait: once a ready key has gone that many steps since it
last ran, it goes first.
Functions
|
Each (node, walk) the entries reach ( |
Classes
|
Higher |
|
What a policy reads off the scheduler. |
|
One entry of a model's or a deployment's priorities ( |
- class mstar.worker.scheduling_policy.Priority(priorities, max_waits)[source]#
Bases:
RoundRobinHigher
prioritiesfirst, round robin among equals and for an unlisted key (0); ahead of both, the keys that have waited theirmax_wait, longest waiting first.A chain also gives up its first step to a ready key above its own or one that has waited its bound. Where round robin would ask anyway, its peek covers those keys too, so a pass makes at most one. A key’s wait is the chain steps this worker has taken since that key last ran, so one that comes ready after an idle spell may go first once.
- chain_yields(chain_key, consecutive, mid_prompt, peek)[source]#
Whether a speculation chain into
chain_keygives the next step to other ready work.consecutive: the chain’s steps since it last yielded;mid_prompt: the chain is cutting a prompt into chunks.
- class mstar.worker.scheduling_policy.ReadyPeek(*args, **kwargs)[source]#
Bases:
ProtocolWhat a policy reads off the scheduler.
- class mstar.worker.scheduling_policy.RoundRobin[source]#
Bases:
SchedulingPolicy
- class mstar.worker.scheduling_policy.SchedulingPolicy[source]#
Bases:
ABC
- class mstar.worker.scheduling_policy.WalkPriority(node=None, walk=None, priority=0, max_wait=None)[source]#
Bases:
NamedTupleOne entry of a model’s or a deployment’s priorities (
Model.get_walk_priorities, the serving config’sscheduling.walk_priorities). It applies to every (node, walk) it names: a node’s walks, a walk’s nodes, or one pair. Where entries overlap, the most specific decides: a pair over a walk over a node.priority: higher first, unlisted 0.max_wait: the most of the worker’s chain steps the key may go without running while it has ready rows, before it goes first; None falls back to the policy’s default, which may itself be none (no bound).
- mstar.worker.scheduling_policy.resolve_walk_priorities(entries, walk_nodes, owner)[source]#
Each (node, walk) the entries reach (
walk_nodes: walk -> the nodes it runs), as the most specific entry that names it: a pair, then a walk, then a node. Refuses an entry naming neither, a node or walk no walk runs, a pair the walk doesn’t run, a negative priority, amax_waitunder 1, and two entries for the same target.ownernames where the entries came from, in the error.- Parameters:
entries (Sequence[WalkPriority])
walk_nodes (Mapping[str, Collection[str]])
owner (str)
- Return type:
dict[tuple[str, str], WalkPriority]