mstar.worker.worker_metrics

mstar.worker.worker_metrics#

The engine worker’s metrics, built only while metrics are on.

Steps are recorded on the main thread as they finish (two histogram bumps, no per-request work), and the tensor manager records each edge it reads (EdgeTransfers). Everything else is sampled at most once per status interval, when the worker sends the conductor one WorkerStatus with cumulative snapshots; the conductor forwards them to the API server.

The gauges are live from worker start. The histograms and counters, and the transfers, start at start_counting(), which the API server triggers after its warmup requests, as it does for its own families.

Engine families cover only the nodes this rank leads (components): the other ranks of a lockstep instance run the same steps over the same page counts. Transfers are every rank’s own reads (each reads its slice). Recording never changes what any rank runs.

Classes

WorkerMetrics(worker_id, interval_s, ...)

class mstar.worker.worker_metrics.WorkerMetrics(worker_id, interval_s, components, engine, scheduler, send, *, enable_transfers)[source]#

Bases: object

Parameters:
maybe_report()[source]#

Send the conductor a WorkerStatus once per status interval.

Return type:

None

snapshot()[source]#

Every worker family in catalog.WORKER_FAMILIES order.

Return type:

list[MetricSnapshot]

start_counting()[source]#

Count the steps whose time starts from now on (see step_done), the prefix-cache lookups after this, and the tensors this rank reads from now on.

Return type:

None

step_done(node, graph_walk, inputs, submitted_at)[source]#

A step finished postprocessing. Its time counts from submitted_at (the hand-off to the GPU thread) or from the end of the worker’s previous step if that was later, so pipelined steps are not counted twice.

Parameters:
Return type:

None