Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Telemetry and benchmarking

MiRMiR publishes the same runtime state to the TUI, web dashboard, CLI status, and private gRPC clients.

Metrics

MetricMeaning
TTFTTime from request start to the first generated token.
PrefillPrompt tokens processed per second.
DecodeGenerated tokens processed per second after prefill.
E2E throughputCompletion tokens divided by complete request time.
MemoryActive and cached accelerator or unified-memory occupancy.
K/V occupancyUsed blocks, free capacity, and prefix-cache effectiveness.

The server samples runtime metrics every second, keeps 900 samples, and persists telemetry atomically under ~/.local/state/mirmir/telemetry.toml. The web dashboard renders the latest 60 samples and restores them after a restart. Shutdown waits up to two seconds for the final history flush. If a system read or filesystem write stalls, the server logs a warning and continues shutdown; the last unflushed samples may be lost. The history worker holds no model or inference-runtime handles. History memory samples report host system memory; a busy logical K/V cache retains its previous reading until the next sample.

Inspect current state

mirmir status

The command prints server health, ready models, and the latest telemetry snapshot as JSON.

Benchmark a prompt

mirmir prompt \
  --model qwen \
  --prompt "Explain continuous batching." \
  --warmup 1 \
  --samples 5 \
  --seed 7 \
  --json \
  --csv benchmark.csv

The report contains individual samples and mean, median, p50, p95, population standard deviation, minimum, and maximum for E2E, TTFT, prefill, decode, and their durations.

Logging

mirmir serve emits structured lifecycle and request logs at INFO. Narrow or expand application targets explicitly:

RUST_LOG=mirmir=trace,libmir=debug mirmir serve
RUST_LOG=mirmir=warn,libmir=warn mirmir serve