Metrics reference#

Every Prometheus metric family M* serves at GET /metrics, what moves it, and which process measures it. How to turn metrics on is in Metrics; a ready-to-run Prometheus and Grafana setup is in Prometheus and Grafana.

mstar serve kokoro --enable-metrics
curl -s localhost:8000/metrics | grep '^mstar_'

Metrics are off by default. With --enable-metrics off, /metrics answers 404, the conductor sends no status message and the workers record nothing. Like /health, /metrics needs no API key; it exposes counts, latencies and the names of the model, its graph nodes and routes, never prompts or outputs.

Where the numbers come from#

Only the API server process renders /metrics, from one registry. The other processes send it plain numbers on messages they already exchange, so the time a value reflects depends on where it was measured:

API server. Request counts, request latencies, output volume and HTTP refusals are recorded as requests pass through it. The data worker is a thread of the API server, so its families are recorded in the same process. These are current at every scrape.
Conductor. While metrics are on, the conductor sends the API server a status message every second. Its gauges show the latest message, so they can be up to a second old; mstar_conductor_status_age_seconds says how old.
Engine workers. Each worker sends the conductor its cumulative numbers every second, and the conductor adds them to its next status message, so worker families can be up to two seconds old. A lost or reordered message cannot corrupt a counter: every message carries totals, not increments, and the API server keeps the latest.
Rust frontend. With --rust-frontend, mstar-server serves /metrics by asking the API server for the rendered text over the bridge, so both frontends show the same families. It answers 503 if the bridge does not reply within two seconds. mstar-server counts its own HTTP refusals and sends the totals with each /metrics request, and the API server exports them in mstar_http_refusals_total.

The registry holds only mstar_ families: no process_ or python_ metrics. Every value starts from zero when the server starts.

Conventions#

Names use Prometheus base units (_seconds, _bytes), and counters end in _total. Read counters and histograms with rate() or increase().
Every family has a model_name label: the config’s model key (the registry name, e.g. kokoro or qwen3_omni).
Label values come from fixed sets or from the model and its config, never from requests, so the number of series is bounded:
- outcome: success, error, abort, timeout.
- output_modalities and input_modalities: the request’s modalities, sorted and comma-joined (audio,text), from text, image, audio, video, video_frame, action, scalar, tensor. In practice a model has a few combinations.
- modality: one of those modality names.
- state: waiting, running, draining.
- component: a graph node name from the config’s node_groups.
- graph_walk: a graph walk name of the model.
- resource: the key of a paged KV cache in the model.
- src and dst: a graph node name, or api_server.
- transport: SHM, TCP, RDMA.
- route: a route path as declared (/v1/chat/completions), or unmatched.
- status: an HTTP status code the frontend sent (a 4xx, or 503).
A labeled series appears the first time it is used. For a new set of output modalities, mstar_requests_total exports all four outcomes at zero from that request on, and the data worker’s two families start together, so a later first error shows in rate(). rate() and increase() miss what was counted before the first scrape that saw a series: if a set’s first request fails, its error series starts at 1.

What is counted from when#

API server families count from the first request after warmup. The server enables metrics after its warmup requests, so first-shape compile times stay out of the request latency histograms.
Engine worker histograms and counters, and the transfers workers read, count from the same point. When the API server enables its metrics it has the conductor tell every worker, so a worker counts the steps whose time starts after that, and the prefix-cache lookups and tensor reads after it. The gauges (KV pages, pending steps) show the current state from worker start.
Requests refused before submission (a bad form, a missing API key, a body over the limit, a request_id already in flight) are counted only in mstar_http_refusals_total. A request that was submitted and then failed is counted once, as outcome="error" in mstar_requests_total.

Requests#

Exported by the API server, recorded where a request leaves it, the same way for /generate, /generate/ws, the OpenAI routes and both frontends.

mstar_requests_total{model_name, output_modalities, outcome}: Counter. Requests that left the API server. success: the result went to the HTTP layer. error: the request failed in the engine, preprocessing or result delivery, or the conductor went away. abort: the client went away, or its finished result was dropped unread. timeout: --timeout expired.
mstar_requests_in_flight{model_name}: Gauge, read at scrape time. Requests the API server holds: submitted and not yet returned, failed or aborted.
mstar_e2e_request_latency_seconds{model_name, output_modalities}: Histogram, seconds, successful requests. From submission (after the request body was read) to the result leaving the API server. Buckets: E2E.
mstar_time_to_first_output_seconds{model_name, output_modalities}: Histogram, seconds, successful requests. From submission to the first result chunk of any modality reaching the API server. Measured on the server: it leaves out the network to the client, and a non-streaming client sees nothing until the end. Buckets: first output.

Request stages#

Exported by the API server, for successful requests, from the timestamps every request carries. The conductor’s timestamps are compared with the API server’s, which assumes both run on one host.

mstar_request_preprocess_seconds{model_name, output_modalities}: Histogram, seconds. From submission to the data worker handing the preprocessed request to the conductor, including the wait in the data worker’s queue. Buckets: stage.
mstar_request_queue_seconds{model_name, output_modalities}: Histogram, seconds. From the conductor taking the request in to admitting it for dispatch: zero unless the request queued at max_concurrent_requests. The conductor sends it with the request’s completion. Buckets: queue.
mstar_request_engine_seconds{model_name, output_modalities}: Histogram, seconds. From the conductor dispatching the request to the workers (after any wait in mstar_request_queue_seconds) to the conductor finishing it. Buckets: E2E.
mstar_request_delivery_seconds{model_name, output_modalities}: Histogram, seconds. From the conductor finishing the request to the result leaving the API server: the last chunks’ postprocessing and transfer, and the client handler picking the result up. Buckets: stage.

Output#

Exported by the API server, counted per result chunk as it reaches the API server.

mstar_output_chunks_total{model_name, modality}: Counter. Result chunks, for every request whatever its outcome. Error chunks are not counted.
mstar_output_bytes_total{model_name, modality}: Counter. Bytes of those chunks.
mstar_output_audio_seconds_total{model_name}: Counter, seconds. Audio in those chunks, from their byte length as 16-bit PCM at the chunk’s sample rate and channel count. Audio chunks that carry no sample rate are not counted.
mstar_time_to_first_chunk_seconds{model_name, modality}: Histogram, seconds, successful requests. From submission to the first chunk of each output modality reaching the API server: time to first audio, first image, first video frame. Buckets: first output.

Data worker#

Exported by the API server. The data worker is the API server thread that loads each request’s media and runs the model’s prompt processing, one request at a time.

mstar_data_worker_preprocess_seconds{model_name, input_modalities}: Histogram, seconds. From the data worker taking a request off its queue to handing it to the conductor, for every request whose preprocessing finished, whatever its outcome later. Unlike mstar_request_preprocess_seconds, it leaves out the wait in the queue. Buckets: stage.
mstar_data_worker_preprocess_failures_total{model_name, input_modalities}: Counter. Requests whose preprocessing failed in the data worker, each once: a file count or media length over its limit, a file that cannot be decoded, the model’s prompt processing, or handing the inputs on. Such a request also ends as outcome="error" in mstar_requests_total unless its client had already gone.
mstar_upload_size_bytes{model_name, modality}: Histogram, bytes. Size on disk of each image, audio or video file a request carried to the data worker, whether the request succeeded or failed: an upload as received, inline media (a data: URL, base64 files on /generate/ws) after decoding. The realtime route and long transcriptions count the WAV files they write from the decoded audio. Buckets: upload size.
mstar_data_worker_queue_depth{model_name}: Gauge, read at scrape time. Requests submitted and not yet taken up by the data worker; the one it is working on is not counted. A request whose client went away counts until the data worker takes it up.

HTTP refusals#

Exported by the API server, counted by whichever frontend serves the deployment.

mstar_http_refusals_total{model_name, route, status}: Counter. HTTP requests the frontend answered with an error without submitting them to the API server: 401 (missing or wrong API key), 413 (body over MSTAR_MAX_BODY_MB), 422 (body that fails validation), 400 (a bad form field or request), 409 (a request_id in flight), 404 (unknown route, or a model without the endpoint), 405 (wrong method), and 503 when the Rust frontend’s MSTAR_MAX_CONCURRENT_REQUESTS cap turns a request away. WebSocket handshakes are not counted. With --rust-frontend, the values are the totals mstar-server sent with the latest /metrics request.

Conductor#

From the conductor’s status message, sent every second while metrics are on.

mstar_conductor_requests{model_name, state}: Gauge, up to a second old. Requests the conductor holds. waiting: queued at max_concurrent_requests, not dispatched yet. running: dispatched to workers, not finished. draining: completed, aborted or failed, and waiting for every reader to release its outputs before the workers free its state; it still counts toward max_concurrent_requests.
mstar_component_in_flight{model_name, component}: Gauge, up to a second old. Running requests with a pass on the graph node that the conductor dispatched and has not heard back from. The conductor sees whole passes, so a node running a request’s decode loop counts it once for the loop.
mstar_conductor_status_age_seconds{model_name}: Gauge, seconds, computed at scrape time. Time since the API server received the conductor’s last status message, or since metrics were enabled if none arrived yet. More than a few seconds means the conductor stopped reporting, and the conductor, worker and transport values its message carries are stale.

Engine workers#

From the workers’ numbers, forwarded in the conductor’s status message, so up to two seconds old. A node with replicas on several workers shows their sum. The ranks of a tensor or sequence parallel instance count once: the instance’s first rank reports.

mstar_kv_pages_used{model_name, component, resource}: Gauge. Pages of a paged KV cache not on its free list: held by requests, or kept by the prefix cache. A node can have several caches (e.g. a decoder’s self and cross attention), told apart by resource. A cache two nodes of one worker share shows under each node, so summing over component counts it twice.
mstar_kv_pages_total{model_name, component, resource}: Gauge. Pages of the cache that requests can hold.
mstar_kv_pages_cached{model_name, component, resource}: Gauge. Pages the prefix cache names. Running requests may also hold some of them; the cache frees the others when a request needs the room.
mstar_step_seconds{model_name, component, graph_walk}: Histogram, seconds, engine steps that ran a forward. From when the worker hands the step to its GPU thread, or from the end of the previous step if that was later, to the end of the step’s postprocess (outputs on the host, stop check, routing). Overlapping steps are not counted twice: in steady state this is the interval between step completions, and rate(mstar_step_seconds_sum[1m]) is the share of time the worker had a step running. Buckets: step.
mstar_batch_size{model_name, component, graph_walk}: Histogram, rows the step’s forward ran. Buckets: batch.
mstar_prefix_cache_query_tokens_total{model_name, component}: Counter. Prompt tokens looked up in the prefix cache, once per request (its whole keyed prompt). Only models that key their prompts for prefix reuse count (BAGEL and Orpheus); the cache is off under tensor parallelism.
mstar_prefix_cache_hit_tokens_total{model_name, component}: Counter. Tokens the prefix cache already held at that lookup. The hit rate is rate(mstar_prefix_cache_hit_tokens_total[5m]) divided by rate(mstar_prefix_cache_query_tokens_total[5m]).
mstar_component_pending_steps{model_name, component}: Gauge. Requests whose next step on the node is ready (its inputs arrived) and not running: waiting for room in a step, or for the worker to pick the node.

Tensor transport#

Tensor transfers between graph nodes (Tensor transport), counted by the reader, once per edge read: the tensors of one graph edge that one worker read from one producer. Workers report through the conductor (up to two seconds old); the API server adds what its data worker read since metrics were enabled, which the data worker publishes once a second. src is the node that produced the tensors, dst the node that reads them, and api_server stands for the request’s inputs (as src) and its results (as dst). The Python and Rust runtimes report the same series.

mstar_transfer_bytes_total{model_name, src, dst, transport}: Counter, bytes. Tensor bytes read over the edge.
mstar_transfers_total{model_name, src, dst, transport}: Counter. Tensors read over the edge.
mstar_transfer_seconds{model_name, src, dst, transport}: Histogram, seconds, one observation per edge read. The reader’s time to read the edge’s tensors: the SHM file read and copy, the copy out of the SHM arena, or the RDMA or TCP read. A GPU reader of the arena only issues its host-to-device copies in that time. The producer’s own work (serializing, staging, registering memory) is not included; it is the tx time of Request profiling. Buckets: transfer.

A tensor already on the reader moves nothing and is not counted. An edge the conductor joined from several producers is one read per producing node, with its read time split between them by bytes. Under tensor parallelism every rank counts its own read, so an edge two ranks read is two reads.

Histogram buckets#

Upper bounds, without +Inf:

E2E (mstar_e2e_request_latency_seconds, mstar_request_engine_seconds): 0.05, 0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 3, 5, 7.5, 10, 15, 20, 30, 60, 120, 300, 600, 1200 s.
First output (mstar_time_to_first_output_seconds, mstar_time_to_first_chunk_seconds): 0.01, 0.025, 0.05, 0.075, 0.1, 0.15, 0.2, 0.3, 0.5, 0.75, 1, 1.5, 2, 3, 5, 7.5, 10, 20, 30, 60, 120, 300, 600 s.
Stage (mstar_request_preprocess_seconds, mstar_request_delivery_seconds, mstar_data_worker_preprocess_seconds): 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300, 600 s.
Queue (mstar_request_queue_seconds): 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300, 600 s.
Upload size (mstar_upload_size_bytes): 1, 4, 16, 32, 64, 128, 256, 512 KiB, then 1, 2, 4, 8, 16, 32, 64, 128, 256, 1024 MiB.
Step (mstar_step_seconds): 0.0005, 0.001, 0.002, 0.003, 0.005, 0.0075, 0.01, 0.015, 0.02, 0.03, 0.05, 0.075, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30 s.
Batch (mstar_batch_size): 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024 rows.
Transfer (mstar_transfer_seconds): 10, 25, 50, 100, 250, 500 µs, then 1, 2.5, 5, 10, 25, 50, 100, 250, 500 ms, then 1, 2.5, 5, 10 s.

Example queries#

# Requests per second by outcome
sum by (outcome) (rate(mstar_requests_total[1m]))

# Share of requests that did not succeed
sum(rate(mstar_requests_total{outcome!="success"}[5m]))
  / sum(rate(mstar_requests_total[5m]))

# p95 end-to-end latency and time to first output
histogram_quantile(0.95, sum by (le) (rate(mstar_e2e_request_latency_seconds_bucket[5m])))
histogram_quantile(0.95, sum by (le) (rate(mstar_time_to_first_output_seconds_bucket[5m])))

# KV cache use per node and cache
mstar_kv_pages_used / mstar_kv_pages_total

# The conductor stopped reporting (alert)
mstar_conductor_status_age_seconds > 10

Prometheus and Grafana#

examples/observability/ runs Prometheus and Grafana in Docker Compose against an M* server on the host:

mstar serve kokoro --enable-metrics          # on the host, port 8000
docker compose -f examples/observability/docker-compose.yml up -d

Prometheus (localhost:9090) scrapes /metrics every 5 seconds; Grafana (localhost:3000) has Prometheus as its default data source and loads any dashboard JSON placed in examples/observability/grafana/dashboards/. Its README covers how the containers reach the host on Linux, a server on another port, and the optional anonymous viewer.

The M* serving dashboard there (mstar.json) has a panel for every family on this page, in one row per component: requests, request stages and output, data worker, conductor, engine workers, tensor transport and HTTP refusals. Its model variable picks the model_name values to show.

To scrape from an existing Prometheus, add a job for the server’s address:

scrape_configs:
  - job_name: mstar
    metrics_path: /metrics
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:8000"]

The conductor and workers report once a second, so a scrape interval under a second only repeats values. Keep the scrape timeout above two seconds with --rust-frontend, which waits that long for the bridge before answering 503.

Metrics reference

Contents