mstar.conductor.lora_ops#
Loading and unloading LoRA adapters at runtime, as the conductor sees it.
The API server asks; the conductor forwards the request to every worker hosting a node
with the LoRA pool and answers once all of them have acknowledged. A load is answered as
done only when every worker loaded it; if any failed, the others are told to unload it
again. Which names are served is the API server’s to track (ServedModels, which a
request’s adapter is selected from): the conductor keeps no registry of its own, only
the operations in flight, during which it refuses requests on their adapters with a 409:
each worker finishes an operation in its own time, so the ranks of one lockstep instance
could open such a request differently.
A worker that never answers (hung, dead) would leave the name refused with a 409 for good.
Past runtime_update_timeout_s an operation is given up: a load is undone on every
worker and answered with a 504, an unload (the name already out of the API server’s
selection) answered as done.
Classes
|
The conductor's half of |
- class mstar.conductor.lora_ops.LoraAdapterOps(spec, workers, send, clock=time.perf_counter)[source]#
Bases:
objectThe conductor’s half of
/v1/load_lora_adapterand/v1/unload_lora_adapter.specis a LoRA pool’s spec, whose runtime-update settings every pool shares (None for a deployment without LoRA);workersare the workers hosting the pools’ nodes;send(peer, message)is the conductor’s communicator;clockits clock (time.perf_counter, as its other timed checks).- Parameters:
- ack(body)[source]#
- Parameters:
body (LoraAdapterAck)
- Return type:
None
- expire()[source]#
Give up on the operations some worker has not answered within
runtime_update_timeout_s. A late ack for one is ignored.- Return type:
None
- load(body)[source]#
- Parameters:
body (LoadLoraAdapter)
- Return type:
None
- next_deadline()[source]#
When
expirenext has something to give up on, onclock; None with nothing pending.- Return type:
float | None
- refuse_request(configs)[source]#
Refuse (409) a request whose adapter a load or unload is in flight for.
- Parameters:
configs (Iterable[ResourceReqConfig])
- Return type:
None
- unload(body)[source]#
Running requests on the name finish on its weights; the API server took it out of selection before asking.
- Parameters:
body (UnloadLoraAdapter)
- Return type:
None