mstar.worker.preemption#

The worker’s half of preemption: the moves a paged KV pool chooses when a request runs short or a waiting request doesn’t fit, made through the engine, and the counts the metrics read. The pool keeps every victim, so nothing here outlives a call.

Classes

Preempted([times, sampled_at])

How often a request was preempted, and how many samples it had kept when it was last preempted.

Preemption(engine_manager, rebuilds, ...)

class mstar.worker.preemption.Preempted(times=0, sampled_at=-1)[source]#

Bases: object

How often a request was preempted, and how many samples it had kept when it was last preempted.

Parameters:
  • times (int)

  • sampled_at (int)

sampled_at: int#
times: int#
class mstar.worker.preemption.Preemption(engine_manager, rebuilds, scheduler, request_state, log, rid_str)[source]#

Bases: object

Parameters:
  • engine_manager (EngineManager)

  • rebuilds (Rebuilds)

  • scheduler (MicroScheduler)

  • request_state (RequestStateManager)

  • log (Callable[..., None])

  • rid_str (Callable[[int], str])

counts: dict[str, PreemptionCounts]#
make_room()[source]#

Move the newer requests that wait holding pages for the oldest waiting request a paged KV pool refused that such moves can let in, as KVManager.make_room chooses them. No step of theirs runs short to move them, since none of them runs. An older request stuck on something no move frees, another resource or a pool that doesn’t move, holds up no younger one, and one the room now fits moves nothing, so no younger one takes that room.

A walk that moved nothing runs again only once the refusals or what the pools can move have changed. A refusal’s mark changes once its room grows, but a request becomes droppable when its ids land, which grows nothing.

Return type:

None

preempt(node_batch)[source]#

KVManager.preempt chooses victims for the request that hit KV OOM. Move each victim to the host or drop it, or, when no move covers the shortage, move the request itself to the host or queue it for its shortage, and return True. Return False, so the offload and the hold run, when the failed resource isn’t a paged KV cache, when the node rebuilds no request here and the cache has no host pool or isn’t the node’s only one, or when KVManager.preempt returns None.

Parameters:

node_batch (ExecutingBatch)

Return type:

bool

rebuilt(rid, node_name, tokens)[source]#

Count the tokens a row of rid’s rebuild on node_name ran.

Parameters:
Return type:

None

refuse_drops(rids, node_name)[source]#

Tell each paged KV cache of node_name that no drop can free rids, whose ids the node forgot.

Parameters:
Return type:

None

remove_request(rid)[source]#
Parameters:

rid (int)

Return type:

None