mstar.worker.preemption#
The worker’s half of preemption: the moves a paged KV pool chooses when a request runs short or a waiting request doesn’t fit, made through the engine, and the counts the metrics read. The pool keeps every victim, so nothing here outlives a call.
Classes
|
How often a request was preempted, and how many samples it had kept when it was last preempted. |
|
- class mstar.worker.preemption.Preempted(times=0, sampled_at=-1)[source]#
Bases:
objectHow often a request was preempted, and how many samples it had kept when it was last preempted.
- class mstar.worker.preemption.Preemption(engine_manager, rebuilds, scheduler, request_state, log, rid_str)[source]#
Bases:
object- Parameters:
engine_manager (EngineManager)
rebuilds (Rebuilds)
scheduler (MicroScheduler)
request_state (RequestStateManager)
log (Callable[..., None])
- counts: dict[str, PreemptionCounts]#
- make_room()[source]#
Move the newer requests that wait holding pages for the oldest waiting request a paged KV pool refused that such moves can let in, as KVManager.make_room chooses them. No step of theirs runs short to move them, since none of them runs. An older request stuck on something no move frees, another resource or a pool that doesn’t move, holds up no younger one, and one the room now fits moves nothing, so no younger one takes that room.
A walk that moved nothing runs again only once the refusals or what the pools can move have changed. A refusal’s mark changes once its room grows, but a request becomes droppable when its ids land, which grows nothing.
- Return type:
None
- preempt(node_batch)[source]#
KVManager.preempt chooses victims for the request that hit KV OOM. Move each victim to the host or drop it, or, when no move covers the shortage, move the request itself to the host or queue it for its shortage, and return True. Return False, so the offload and the hold run, when the failed resource isn’t a paged KV cache, when the node rebuilds no request here and the cache has no host pool or isn’t the node’s only one, or when KVManager.preempt returns None.
- Parameters:
node_batch (ExecutingBatch)
- Return type:
- refuse_drops(rids, node_name)[source]#
Tell each paged KV cache of
node_namethat no drop can freerids, whose ids the node forgot.- Parameters:
rids (Collection[int])
node_name (str)
- Return type:
None