Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Native inference, from app to library

MiRMiR 0.6.2 is a native local model runtime with a terminal dashboard, an optional browser dashboard, a command-line interface, and an OpenAI-compatible HTTP server. libmir is the embeddable Rust inference library underneath it.

Use these docs to operate MiRMiR or build your own native application on the same model discovery, loading, generation, K/V cache, Metal, and CUDA layers.

See what changes in 0.6.2: faster Metal and more models on it. In 0.6.1 each model’s context is chosen before it loads, so several models fit side by side. 0.6.0 added remote access, Mirmir for Mac and images for Nemotron Omni on CUDA.

Choose a surface

SurfaceUse it when
TUIYou want an interactive terminal workflow with no browser.
Web dashboardYou want the same local runtime controls and telemetry in a browser.
CLIYou need scripts, machine-readable output, or repeatable benchmarks.
HTTP APIAn existing application already understands OpenAI-style requests.
libmirA Rust application needs direct ownership of models and sessions.

MiRMiR and libmir are pre-1.0. These pages describe version 0.6.2. Support claims require complete checkpoint and backend validation; planned formats are not described as available.

The stack

your application
      |
      +-- HTTP / SSE --> mirmir server --+
      +-- TUI / CLI --> private gRPC ----+--> libmir --> Metal / CUDA
      +-- Rust API -----------------------+

MiRMiR owns the application, configuration, transport, dashboards, model catalog, and telemetry. Interactive and scripted clients attach to the same local runtime or start a temporary one when needed. libmir owns checkpoint discovery, tokenization, generation semantics, sessions, scheduling, cache policy, and backend adapters. Laya decisions use a separate typed API backed by Metal, CUDA, or the native CPU module mircup.

Install MiRMiR

MiRMiR ships as one native binary. Install version 0.6.2 from crates.io:

cargo install mirmir --version 0.6.2 --locked
mirmir --version

The final command should print mirmir 0.6.2.

Platform requirements

MiRMiR currently targets accelerator-backed inference:

  • Apple Silicon macOS uses the Metal backend and requires the native MLX 0.32.4 framework;
  • Linux uses CUDA and requires a supported NVIDIA driver, CUDA Toolkit 13, and git;
  • no Python, PyTorch, or Transformers runtime is used in the request path.

On Linux, the build compiles native CUDA kernels. It uses the toolkit in /usr/local/cuda, or the one named by CUDA_HOME (and NVCC for a specific compiler); nvcc does not need to be on PATH. The first build fetches the pinned CUTLASS and FlashAttention sources into ~/.cache/mircuda; set MIRCUDA_CUTLASS_DIR and MIRCUDA_FLASH_ATTN_DIR to reuse existing checkouts or build offline. Kernels target sm_120f (Blackwell) by default; set MIRCUDA_CUDA_ARCH for another architecture.

For a source build, clone the application repository and build it using its pinned Rust toolchain:

git clone https://github.com/mirmir-tech/mirmir.git
cd mirmir
cargo build --release --locked

Initialize configuration

mirmir config init
mirmir config validate

The persistent files are stored under ~/.config/mirmir. Runtime state and the fallback Unix socket live under ~/.local/state/mirmir.

Open the terminal dashboard

Run MiRMiR without a subcommand:

mirmir

The process attaches to an existing local runtime when one is available. If it starts a temporary runtime itself, it also shuts that runtime down when the TUI exits.

Continue with Run your first model.

Upgrading an existing installation? Read Upgrade to 0.6.2, Upgrade to 0.6.1, Upgrade from 0.3, or Upgrade from 0.2 for older installations.

Run your first model

MiRMiR can search Hugging Face, download a snapshot into its managed cache, inspect compatibility, and load it into accelerator memory.

Search and download

mirmir model search "Qwen 2.5" --limit 10
mirmir model pull Qwen/Qwen2.5-0.5B-Instruct

Search results are ranked by the execution contract discovered by libmir and by a conservative estimate of whether the checkpoint fits this machine. MiRMiR evaluates architecture and physical encoding separately, including dense, integer, block-floating-point, and codebook-quantized SafeTensors.

For gated repositories, configure a Hugging Face token without placing the secret directly in shell history:

printf '%s' "$HF_TOKEN" | mirmir config set hugging_face.token
mirmir config test hugging_face.token

Inspect and load

mirmir model inspect Qwen/Qwen2.5-0.5B-Instruct
mirmir model load Qwen/Qwen2.5-0.5B-Instruct

Inspection identifies the task, tokenizer, tensor layout, memory requirement, configured K/V cache, and safe context capacity before loading weights. --force bypasses only the conservative memory-fit rejection; it cannot make an unsupported execution contract valid.

See Model compatibility for the 0.6.2 format and backend matrix.

Generate

Open the TUI and select Chat, or use the non-interactive path:

mirmir prompt \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --prompt "Explain paged K/V cache in three sentences."

For a reproducible measurement:

mirmir prompt \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --prompt "Explain paged K/V cache." \
  --seed 7 --temperature 0 --warmup 1 --samples 5 --json

Unload or remove

mirmir model unload Qwen/Qwen2.5-0.5B-Instruct
mirmir model remove Qwen/Qwen2.5-0.5B-Instruct

Removing is destructive and requires the model to be unloaded first.

Mirmir for Mac

Mirmir for Mac is a native SwiftUI app for macOS 26.2 or later on Apple silicon. It lives in the menu bar and has a window for throughput and memory charts, the model library, chat, sessions, logs and the servers you use. It includes the MiRMiR runtime (mirmir 0.6.2 and MLX), so nothing else needs to be installed.

Install

  1. Download Mirmir for Mac, open the disk image and drag Mirmir to Applications. The app is signed and notarized.
  2. Start Mirmir. It starts the mirmir it includes as its own server.

Its own server

Mirmir runs sandboxed, like an App Store app. Its server lives in the app’s container, ~/Library/Containers/tech.mirmir.app/Data, with its own configuration, models and logs, and is separate from the mirmir CLI:

  • the app does not attach to a server the CLI or TUI started, and the CLI does not see the app’s server;
  • models the app downloads stay in its container; the CLI’s models are not visible until you grant their folder (see below);
  • the app’s OpenAI-compatible API listens on http://127.0.0.1:18080/v1, so it does not collide with a CLI server on port 8080. Settings → Server shows the address for Zed, Raycast or curl.

Use the app or the CLI on a Mac; running both means two servers, two model sets and twice the memory for a loaded model. Quitting Mirmir stops its server.

Model folders

The sandbox lets Mirmir read only the folders you choose. In Settings → Model folders, Add Folder… grants one:

  • a Hugging Face cache, such as ~/.cache/huggingface or its hub folder: the server lists its models next to its own (through HF_HUB_CACHE) and restarts once to pick it up;
  • any other folder with checkpoints: load a model from it by path.

Models → Load from Folder… grants a checkpoint folder and loads it in one step. Grants persist across launches; Remove revokes one.

The app speaks the runtime protocol 2 of MiRMiR 0.6.2 and shows Update needed for an older remote server.

The menu bar and the window

Mirmir is a menu bar app: it has no Dock icon. Its icon shows the decode rate while a model generates; click it for the loaded model, live throughput, memory, active sessions and shortcuts to the window (Open Dashboard, Models…, Instances…, Download a Model…).

Closing the window keeps Mirmir running in the menu bar. To quit, choose Quit Mirmir (⌘Q) in the panel. Quitting stops the app’s server.

Servers

Mirmir shows one server at a time: This Mac, its own server, or a remote one. The Server card at the bottom of the sidebar names it, with its address, status, MiRMiR version and device; every screen’s subtitle starts with its name. Click the card to switch to another server or to manage the list. Switching changes the whole window and the menu bar, including the chat list: each conversation belongs to the server it was started on, because a model name means nothing on another server.

This Mac is always listed. Instances → Add Server adds a remote server over TLS: enter host or host:port (the default port is 7443) and an API key created on that server (a pasted grpcs://host:port works too). A CLI server on the same Mac is reachable this way too, once its remote listener is on. The app shows the server’s certificate fingerprint before it sends the key; compare it with mirmir remote status on the server, trust it, then test and add. Later certificate changes are refused. See Remote access.

Edit Server… in Instances changes a server’s name, address, API key or trusted certificate and keeps its chats; a changed connection must pass a test before it is saved. Remove Server asks first and keeps the server’s chats: they come back when a server at the same address is added again.

Create the key on the server and name it after the computer that will use it:

mirmir remote key create --name my-mac --scope control

--scope use (the default) allows chat, model lists and monitoring, and a chat may load a model the server offers; control also loads, unloads, downloads and removes models and changes model and runtime settings. No key can change the server’s own settings or stored secrets. Add --system on a machine that runs the system server; remote servers keep using the CLI.

The first time you add a server on your network, macOS asks whether Mirmir may find devices on your local network: allow it. If you declined, turn Mirmir on in System Settings → Privacy & Security → Local Network; until then the app reports the server’s address as having no route.

API keys are stored in your login Keychain, never in the app’s files. macOS may ask once whether Mirmir may read its key; choose Always Allow.

Share this Mac

Open Instances → This Mac → Remote Access… to let other computers use the app’s server: turn on the listener, copy its certificate fingerprint, create keys with the use or control scope (shown once), revoke them and see which clients connected recently.

Upgrade to 0.6.2

MiRMiR 0.6.2 makes Metal faster and broader, and Mirmir for Mac smaller. All native crates and Mirmir for Mac move to 0.6.2 together; configuration, protocol 2 and the HTTP API are unchanged.

cargo install mirmir --version 0.6.2 --locked
mirmir --version

Restart the runtime after upgrading.

What changes

  • Faster Metal. Prefill keeps long contexts in full chunks, decode attends over a persistent K/V history shared by concurrent requests, and K/V admission is bounded by pending prefill, so deep, concurrent requests fit in memory. On an Apple M5 Pro, mirmir is ahead of MLX-LM on Qwen3.6-35B-A3B and Gemma 4 26B-A4B; see the benchmarks.
  • More models on Metal. Dense linear-attention models, dense GELU Gemma models, Nemotron-H state-space hybrids and the pixel-shuffle vision encoder of Nemotron Omni run on Metal.
  • Mirmir for Mac is about 76 MB instead of 320 MB and runs on macOS 26.2 or later. Its Metal kernels compile on first use: the first request after installing or updating the app, or after a macOS update, takes a moment longer.

Building from source

MLX 0.32.4 remains the native requirement. Workmir’s scripts/mlx/setup.sh builds it with Metal JIT for macOS 26.2 and applies a fix for its JIT d-split attention kernel; a prebuilt MLX 0.32.4 works as before.

Upgrade to 0.6.1

MiRMiR 0.6.1 lets you choose each model’s context before it loads, so several models fit in memory side by side. All native crates and Mirmir for Mac move to 0.6.1 together.

cargo install mirmir --version 0.6.1 --locked
mirmir --version

Restart the runtime after upgrading.

What changes

  • Context per model. The TUI’s load dialog, the web dashboard and Mirmir for Mac show a Context control before Load: automatic, or a token count (the K/V cache shared by the model’s concurrent requests), with the memory it needs. The choice is saved for the model and used by every load. From the CLI: mirmir model load <model> --context 32768. See Configuration.
  • Loaded models report real memory. Inspecting a loaded model no longer shows a 16-token cache; it estimates a reload with the model’s own memory counted as free.
  • Bounded generation settings. Max tokens, temperature, top P, top K and repetition penalty are sliders with fixed ranges in every client; max tokens is limited by the chosen context.
  • Saved context survives a new download. Pulling a newer snapshot of a model keeps its saved generation settings and context.
  • Mirmir for Mac. A download in progress shows its progress instead of an error. Removing a model from disk and selecting a model on a remote server no longer crash the app.

gRPC clients

The protocol stays version 2. ModelContextChoice and ModelContext are new; InspectModelRequest, InspectModelResponse, LoadModelRequest and UpdateModelGenerationRequest gain an optional context. A 0.6.0 server sends no context in its inspection; clients hide the control then.

Rust consumers

libmir::Library::preview_load resolves what a load would use without changing the library, and libmir::LoadPreview holds the result. Library::model_config_with_options no longer replaces the library’s cached configuration.

Upgrade to 0.6.0

MiRMiR 0.6.0 adds remote access to the runtime, the Mirmir for Mac app, and image input for Nemotron Omni on CUDA. mirtal, mircuda, mircup, libmir, their support crates and mirmir move to 0.6.0 together.

cargo install mirmir --version 0.6.0 --locked
mirmir --version
mirmir config validate

Restart the runtime after upgrading: attaching a new client does not replace a running server.

What changes

  • Remote access. mirmir remote enable serves the runtime over TLS with scoped API keys; clients use --remote or Mirmir for Mac. Nothing listens remotely until you enable it. See Remote access.
  • Mirmir for Mac. A native menu bar and window app; see Mirmir for Mac.
  • Nemotron Omni images on CUDA. Checkpoints with the RADIO vision encoder accept images through /v1/chat/completions. The vision weights now count toward the model’s resident memory; on a shared GB10, lower the model’s kv_blocks if smaller models no longer fit beside it. Metal reports image input for these checkpoints as unavailable. Images are sized by the checkpoint processor’s rule; for the few aspect ratios where that rule would exceed the patch budget (very long, thin images), MiRMiR fits the grid to the budget instead, so those images use fewer image tokens than the Hugging Face processor reports.
  • Telemetry memory. Host RAM and the accelerator budget are reported separately with a typed memory pool (runtime protocol 2). The HTTP dashboard JSON keeps its shape.
  • Local model paths. ListLocalModels reports canonical paths, so models reached through a symlinked cache show the right state.

gRPC clients

The private runtime protocol is now version 2. A client built for protocol 1 is refused with a request to update. The removed telemetry fields are listed in mirmir/docs/protocol.md.

Configuration

New optional keys: server.remote_bind, server.remote_tls_cert and server.remote_tls_key. Existing configuration files stay valid.

Rust consumers

Update the libmir requirement to 0.6.0. Source changes from 0.5.5:

Public APIRequired adaptation
libmir_models::layout::VisionPipelineNew variant PixelShuffleEncoder; add an arm or a wildcard to exhaustive matches.
libmir_models::layout::VisionConfigNew variant PixelShuffleEncoder(PixelShuffleVisionConfig).
libmir_models::layout::ImageProcessorConfigNew variant PixelShuffle(PixelShuffleImageProcessorConfig).
libmir::PreparedVisionPromptNew variant PixelShuffle; add an arm or a wildcard to exhaustive matches.
libmir_metal::engine::hybrid_moeMoved to libmir_metal::engine::hybrid::moe.
libmir_metal::engine::hybrid_linear_moeMoved to libmir_metal::engine::hybrid::linear_moe.

Upgrade to 0.5.5

The 0.5.5 release aligns mirtal, mircuda, mircup, libmir, their support crates, and mirmir at one version. The website and bundled dashboard carry the same version. Existing 0.4.x and 0.5.2 installations should review these changes before replacing their binary.

cargo install mirmir --version 0.5.5 --locked
mirmir --version
mirmir config validate

Stop and restart the runtime after upgrading; attaching a new client does not replace an already running server. Keep your configuration and model files. For service installations, retain the service account and socket permissions.

Available capabilities

  • Nemotron-H on CUDA: text generation for the validated Nemotron-3.5-Super NVFP4 checkpoint, with Mamba-2, attention without RoPE, latent MoE, and optional multi-token prediction (MTP). This does not add Nemotron-H execution to Metal or enable the checkpoint’s vision tower.
  • Gemma 4 on CUDA: sliding-window attention and 512-wide global heads use accelerated prefill paths. This is separate from the existing Metal Gemma path; benchmark results remain tied to their recorded builds.
  • Structured content: response_format accepts JSON objects and strict JSON schemas on Metal and CUDA. Reasoning is separate from JSON content; literal protocol markers inside JSON strings remain ordinary text.
  • Laya decisions: libmir::decision runs on CPU, Metal, or CUDA. The manual-testing /v1/decide endpoint exposes CPU and macOS Metal only.
  • Memory and scheduling: per-model K/V block limits, opt-in MTP, and the existing explicit cached-refill and short-prompt admission policies.
  • Metal foundation: the native bridge targets MLX 0.32.3. Build against the matching native installation; the Python mlx package is not required.

Request compatibility

JSON content cannot be combined with schema-constrained tools or a required or named tool_choice. Such requests return 400 before generation. Use JSON content for a final structured answer, or a named schema-constrained tool for a tool invocation. strict: false is not supported.

thinking_token_budget now applies to both schema tools and JSON content, with enabled reasoning and an explicit total completion limit. It reserves space inside that limit; it does not increase the limit. Incomplete or invalid structured output remains an error, including an SSE error after streaming has started.

Rust consumers

Update your libmir requirement to 0.5.5 and select metal, cuda, or cpu as needed. The default feature set enables no backend. CPU currently provides Laya decisions, not the general generation/embedding runtime. Use GenerationRequest { ..Default::default() } to retain defaults for new request fields. Cross-repository dependencies remain crates.io versions.

This release changes Rust source compatibility with 0.5.2:

Public APIRequired adaptation
Error::InvalidResponseFormatUpdate exhaustive error matches for invalid or incomplete structured content.
ArchitectureCapabilityHandle StateSpaceMixer, LatentRoutedExperts, ReluSquared, and SigmoidBiasedTopKRouter in exhaustive matches.
GenerationRequestNew reasoning_effort, reasoning_history, and response_format fields; prefer ..Default::default() where appropriate.
Runtime DecodeRequest / DecodeSequenceInitialize speculation (None for ordinary decoding).
Runtime DecodeOutputInitialize accepted and drafts (empty for ordinary decoding).
DecoderConfig / Metal MixerLowering::SoftmaxInitialize the new typed Q/K normalization field when constructing these structs directly. Prefer parsing checkpoint configuration and lowering the resolved spec.
New MTP configurationConstruct the bounded window with DraftTokens::try_from(count); values must be 1–8.

These additions are not hidden by adding #[non_exhaustive] retroactively, which would itself change construction and matching rules. Rebuild and test Rust consumers when upgrading; binary configuration compatibility is separate.

See JSON output, Laya decisions, and configuration for complete examples.

Upgrade from 0.3

MiRMiR 0.4.0 keeps the 0.3 control surfaces, CLI, configuration keys, and private protocol. It changes how the runtime is organized internally, makes the OpenAI-compatible API stricter and more explicit, adds tool calling and per-request reasoning control, and narrows the libmir facade. Version 0.3.1 was never published; upgrade directly from 0.3.0.

HTTP clients

Review clients that relied on silently ignored fields or on the 0.3 status codes:

  • A non-zero presence_penalty or frequency_penalty, and any non-null reasoning_budget or reasoning_effort, now return 400. Omit them, or use repetition_penalty and the MiRMiR reasoning extension instead.
  • tool_choice must be auto, none, required, or a named function.
  • A prompt that exceeds the model context, or can never fit the K/V cache, is now 400 instead of 500. A request that could fit later remains 429.
  • A cancelled operation returns 408 cancelled.
  • A tool call that violates tools or tool_choice, or is malformed or truncated, returns 422 invalid_model_output.
  • Inference endpoints return 503 until startup model restoration completes. Point readiness probes at the new GET /ready; keep GET /health for liveness.
  • Closing an SSE connection now cancels the generation on the server.

New request capabilities are opt-in: tools with validated tool_calls responses, tool_constraints: "schema" on Metal and CUDA, reasoning and chat_template_kwargs.enable_thinking, thinking_token_budget, return_token_ids for streams, and top_n for reranking. See Chat completions and Embeddings and reranking.

Configuration

Existing config.toml files remain valid; no key was renamed. New optional runtime keys tune scheduling: prefill_batch_wait_us, cached_prefill_policy, prefill_decode_policy, prefill_refill_policy, and, on macOS, metal_decode_reservation. Omitting them keeps the backend defaults. See Configuration.

Models

Decoder rerankers that ship a sentence-transformers LogitScore module are now admitted as causal rerankers on Metal and CUDA. Qwen XML tool calls are parsed natively. Re-run mirmir model inspect after upgrading, because admission reports can change between releases.

libmir embedders

  • Depend on libmir = "0.4.0". The facade is now the only public entry point: the foundation, models, and runtime modules and the backend-specific CUDA and Metal types are no longer re-exported.
  • ChatCompletionRequest and the Chat* message types are replaced by GenerationRequest, Conversation, Message, Tool, ToolCall, and the ToolChoice enum. Model::generate* takes &GenerationRequest, and Model::prepare takes &Conversation.
  • ProgressEvent is an enum with stage(), unit(), count(), and detail() accessors.
  • SchedulerConfig and GenerationToken gained fields; construct them with ..Default::default() or update struct literals.

See Generate and stream for the new request shape.

Upgrade checklist

  1. Install 0.4.1 with cargo install mirmir --version 0.4.1 --locked and run mirmir --version.
  2. Run mirmir config validate against the existing configuration.
  3. Remove unsupported penalty and reasoning-effort fields from API clients and handle 408, 422, and 503 responses.
  4. Switch readiness checks to GET /ready.
  5. Inspect each checkpoint on its target Metal or CUDA machine.

Upgrade from 0.2

MiRMiR 0.3.0 changes the application from a collection of separate command paths into several control surfaces for one local runtime. The TUI, CLI, web dashboard, and HTTP API now observe and operate the same loaded models, configuration, request activity, and telemetry.

Choose who owns the runtime

  • Run mirmir serve for a persistent runtime shared by local clients and, when enabled, the HTTP API and web dashboard.
  • Run mirmir to open the TUI. It attaches to an existing runtime or starts a temporary one when none is available.
  • Model and prompt commands follow the same rule: attach when possible, otherwise start a temporary runtime for the operation.

A client only shuts down a temporary runtime that it started. Exiting a TUI or CLI command attached to mirmir serve leaves the persistent runtime running.

Manage models as a lifecycle

Model discovery, storage, compatibility, and accelerator residency are separate operations in 0.3.0:

mirmir model search "Qwen 3" --limit 20
mirmir model pull Qwen/Qwen3-4B
mirmir model inspect Qwen/Qwen3-4B
mirmir model load Qwen/Qwen3-4B
mirmir model unload Qwen/Qwen3-4B

Search and pull perform bounded remote inspection. Load performs structural, backend, and memory preflight before allocating accelerator weights. --force can override only the conservative memory-fit decision; it cannot admit an unsupported checkpoint.

Move settings into the shared configuration

Use mirmir config instead of maintaining per-command runtime flags:

mirmir config init
mirmir config show
mirmir config validate
mirmir config set default_model Qwen/Qwen3-4B

Configuration records effective values and their source. Secrets live in a separate protected file and remain redacted in every control surface. When a persistent server is running, configuration mutations go through it so there is one writer.

Review model compatibility

Version 0.3.0 admits a checkpoint from its task, architecture, tokenizer, tensor schema, weight encoding, and selected backend. Family names and Hub tags are not support guarantees. The release adds typed SafeTensors paths for dense, integer, block-floating-point, and codebook quantization, including bitsandbytes NF4 and FP4 checkpoints without a Python runtime.

See Model compatibility for the complete 0.3.0 matrix.

Check the public API boundary

The OpenAI-compatible surface remains deliberately smaller than the complete OpenAI platform. Existing clients should target the documented chat, embedding, reranking, model-list, and health endpoints. Management operations belong to the TUI, CLI, web dashboard, or private local control protocol.

Upgrade checklist

  1. Install the 0.3.0 binary and run mirmir --version.
  2. Run mirmir config validate against the existing configuration.
  3. Inspect each checkpoint on its target Metal or CUDA machine.
  4. Start mirmir serve if the runtime must outlive an interactive client.
  5. Confirm API authentication, CORS, and dashboard settings before binding beyond a local development workflow.

Terminal dashboard

mirmir opens the terminal dashboard. All views operate on the same local runtime used by CLI and HTTP clients.

Views

KeyViewPurpose
F1DashboardLive throughput, latency, memory, K/V cache, models, and activity.
F2ModelsSearch, download, inspect, load, unload, and remove checkpoints.
F3ChatStream multi-turn generation, reasoning, metrics, and image input.
F4SettingsInspect and edit effective configuration and secrets.

Use Tab and Shift+Tab, a function key, or the mouse to switch views. Press ? for context-sensitive help.

The Dashboard aggregates current and mean request rates, memory and K/V-cache history, recent operations, and the active runtime stage without forcing a host synchronization in the generation path.

Models

  • / starts a Hugging Face search;
  • d, l, and u download, load, and unload;
  • i includes incompatible or over-budget remote results;
  • r opens a destructive removal confirmation;
  • X cancels the selected cancellable operation;
  • f explicitly allows an over-budget load from the load dialog.

Download progress is streamed from the server. Additional downloads wait in a FIFO queue and can be cancelled before or during transfer.

Download progress

The transfer panel names the repository and current file, and reports both percentage and transferred bytes. Press X with the active transfer selected to request cancellation; queued downloads can be cancelled the same way.

Memory preflight

Before loading, MiRMiR combines weight, K/V-cache, and workspace estimates and compares the result with the accelerator budget. The dialog also reports a safe context limit. An over-budget result is blocked unless you deliberately press f; forcing a load does not bypass compatibility checks.

Chat

The input remains locked while a request is active. Streamed reasoning is kept separate from final content, and the view reports TTFT, prefill, decode, and end-to-end throughput.

  • Ctrl+Left / Ctrl+Right selects a loaded model;
  • Ctrl+P edits one-off generation parameters;
  • Ctrl+K clears the conversation;
  • drop one PNG, JPEG, WebP, or GIF to attach it;
  • Ctrl+D removes the attachment;
  • X requests cooperative cancellation.

During streaming, the active assistant message keeps reasoning collapsible and the composer reports TTFT, prefill rate, decode rate, and token counts. Input is locked until the request finishes or cooperative cancellation completes.

Image attachment is available only when the selected model advertises image input. The attachment name appears above the prompt; Ctrl+D removes it before submission.

Activity and cancellation

The Dashboard records long-running operations and their stage. Select a cancellable operation and press X. MiRMiR marks it as cancelling immediately, but the operation stops cooperatively at a safe boundary rather than in the middle of an accelerator kernel.

Settings

The settings view presents the effective value, its source, and whether a server restart is required. Press Enter to edit the selected setting. The Hugging Face token can be tested with t, and stored secrets remain redacted.

Exit behavior

Esc opens confirmation. q exits outside the chat input, and Ctrl+C exits immediately. Only a TUI that owns its temporary runtime shuts it down.

Web dashboard

The browser dashboard is disabled by default. Enable it and restart the server:

mirmir config set server.web_enabled true
mirmir serve

Open http://127.0.0.1:8080/ui/.

What it exposes

  • Dashboard shows live and historical throughput, prefill, decode, TTFT, memory, K/V occupancy, runtime counters, and recent activity;
  • Models lists local snapshots, searches Hugging Face, and performs download, inspect, load, unload, and removal actions;
  • Chat streams content and reasoning, accepts one image, exposes generation parameters, reports request metrics, and keeps recent conversations;
  • Settings shows effective values, their source, restart requirements, paths, raw TOML, and redacted secrets.

One WebSocket carries telemetry, model snapshots, configuration, startup readiness, and activity. The browser does not poll each view independently.

The header of every view shows whether the runtime is generating, the active model, and the accelerator.

Models

Search Hugging Face by owner or model name and manage snapshots already present on the machine. Each row shows the weight encoding, the backends that can run it, and its capabilities (tools, thinking, vision). Model actions use the same runtime queue and memory checks as the terminal and CLI clients.

Chat

Select a loaded model, optionally attach one supported image, adjust one-off generation parameters, and stream reasoning and final content independently. The latest reply reports TTFT, prefill, decode, and end-to-end throughput.

Recent conversations appear beside the chat. Start another with New chat, reopen one from the list, or delete it. The history stays in this browser’s local storage, not on the server, and keeps the 30 most recent conversations.

Settings

Settings are grouped by section. Each row shows the effective value, its source, and whether a change applies live or after a restart. After you change a restart-bound setting, a banner lists it until you restart mirmir serve or dismiss it. Secrets are redacted and use dedicated mutation paths.

Local security boundary

The dashboard can only be enabled on a loopback HTTP bind. Opening it creates an eight-hour local session in an HttpOnly, SameSite=Strict cookie. Mutations also require a CSRF token held in page memory. The OpenAI bearer token is not accepted as dashboard administration authority.

For remote programmatic access, keep the dashboard local and configure the HTTP API security boundary separately.

Appearance and runtime details

The theme follows your system until you choose another appearance with the sun/moon button in the sidebar. Your choice is remembered. The Dashboard view includes prefill and decode charts, device telemetry, and server platform, backend support, device and host-memory details. Use Open chat on a resident model to start chatting with it. Screenshots use demonstration data.

Mac app

MiRMiR for Mac shows the same runtime as the terminal and web dashboards, natively and from the menu bar: live throughput and memory, the model library, chat, sessions, logs and every server you use. To install and connect it, see Mirmir for Mac.

The menu bar icon shows the decode rate while a model generates. Its panel has the loaded model, decode rate with a sparkline of the last minute, prefill, active sessions, prefix cache hits and memory, and shortcuts to the window. Switch loads another local model.

Window

While the window is open, Mirmir is also in the Dock and the app switcher, so it is easy to get back to; closing it leaves the menu bar icon alone. The sidebar lists the screens; the Server card at its bottom names the server they show and switches to another one.

  • Overview has decode, prefill, time to first token and memory, a throughput chart over 5 minutes, 1 hour or 24 hours, memory against its limit, and the latest sessions. Rates appear only while the server handles a request.
  • Chat streams replies with the model’s reasoning folded above them and shows tokens, decode rate and time to first token under each reply. Each conversation belongs to the server it was started on. With a vision model, the paperclip attaches a PNG, JPEG, GIF, WebP, HEIC or TIFF image (HEIC and TIFF are converted to JPEG). The image stays with the conversation, so follow-up questions and a reopened chat still see it; attaching another replaces it, and the × on the composer’s image removes it before sending. A message with an image is not sent to a model without image input: the app says so and keeps the message.
  • Models lists local checkpoints and running downloads with their features (tools, think, vision), size and state; All, Ready and Downloading filter the list. The search field filters it at once and, in All and Hugging Face, searches the Hub when you pause typing: results join the list with the memory each needs on this server, and Download returns to your models with the download’s progress. The inspector loads the model and chooses its Context before Load (Automatic, or a token count, with the memory it needs; give each model a context to keep several loaded at once) and edits generation defaults. The context is saved for the model and used by every load, also from the menu bar.
  • Instances shows each server’s status, loaded models, decode rate and memory, adds, edits and removes remote servers and opens Remote Access for this Mac.
  • Sessions lists generations, loads and downloads as they run; Logs follows the app’s server.
  • Settings grants model folders and shows the app’s server: its runtime, its OpenAI-compatible API address and its socket.

How it connects

The app bundles the MiRMiR runtime: the same mirmir server as the CLI, with libmir and MLX, runs as a separate process inside it. The app’s interface is a client of that runtime’s gRPC service, like the TUI.

The app is sandboxed, and its server inherits the sandbox. On this Mac the app starts its bundled mirmir with the app’s container as its home, so its configuration, models, socket and logs stay there, apart from the CLI’s, and it stops that server when you quit. Its OpenAI-compatible API listens on 127.0.0.1:18080. The server reads only folders you grant in Settings → Model folders; see Mirmir for Mac.

Remote servers are reached over TLS with a scoped API key from your Keychain and a pinned certificate; see Remote access.

Configuration

Initialize, inspect, and validate configuration with:

mirmir config init
mirmir config show
mirmir config validate

The main file is ~/.config/mirmir/config.toml. Secrets are stored separately with protected permissions and are never returned in clear text by the TUI, web dashboard, CLI, or private gRPC API.

Runtime settings

KeyDefaultMeaning
runtime.kv_block_sizeautomaticTokens stored in one K/V block.
runtime.kv_blocksautomaticTotal block capacity.
runtime.speculative_draft_tokensdisabledDraft count from 1 to 8 for a compatible CUDA MTP checkpoint. auto removes the override.
runtime.kv_cache_dtypeautoBackend-resolved K/V storage type.
runtime.max_batch_requestsautomaticConcurrent decode batch limit.
runtime.max_batch_tokensautomaticScheduler token budget.
runtime.prefill_batch_wait_us200Short window used to form a prefill batch.
runtime.decode_batch_wait_usautomaticShort window used to form a decode batch.
runtime.decode_priority_burstautomaticDecode fairness before other runnable work.
runtime.cached_prefill_policybackend_defaultinterleave_one_block admits one cached continuation during resident decode, lowering refill first-token latency at some decode latency cost.
runtime.prefill_decode_policybackend_defaultcomplete_cohort finishes an admitted prefill cohort before its first tokens, improving decode occupancy at some first-token latency cost.
runtime.prefill_refill_policyclosedshort_prompt lets prompts of at most 128 tokens join a running prefill on CUDA mixed-attention models.
runtime.metal_decode_reservationone_pagemacOS only. generation_budget reserves K/V pages for the request’s whole generation budget from reclaimable spare pages.
runtime.memory_reserve_percentautomaticPercentage of visible memory kept outside model admission.
runtime.memory_reserve_bytesautomaticAdditional fixed memory reserve used by admission.
runtime.vision_max_pixelsautomaticHard cap on resized image area.
runtime.vision_attention_budget_bytesautomaticVision attention workspace cap.
runtime.vision_memory_percent80Memory available to automatic vision budgeting.

Server settings

KeyDefaultMeaning
server.http_bind127.0.0.1:8080HTTP listen address.
server.allow_remotefalsePermits a non-loopback bind.
server.web_enabledfalseServes the embedded dashboard on /ui/.
server.cors_origins[]Exact allowed browser origins.
server.body_limit_bytes29360128Maximum request body.
server.request_timeout_seconds300HTTP request timeout.
server.max_concurrency16In-flight HTTP request limit.

Most runtime and server settings require a restart. The UI identifies this per row. Set any optional runtime key to auto to remove the override and return to the backend default, for example:

mirmir config set runtime.cached_prefill_policy interleave_one_block
mirmir config set runtime.cached_prefill_policy auto

runtime.prefill_refill_policy = "short_prompt" requires the default prefill_decode_policy; validation rejects it with complete_cohort. Other platforms reject runtime.metal_decode_reservation.

Speculative decoding and model memory

For the supported Nemotron-H CUDA checkpoint with a prediction head:

mirmir config set runtime.speculative_draft_tokens 3

Restart the runtime after changing the setting. auto disables the override. MTP uses extra weights and recurrent snapshots. It can improve single-stream decode, but does not promise a gain for every prompt or concurrent batch. It is not a generic speculative decoder for every model or a Metal feature.

Context per model

A model’s context is the K/V capacity it loads with, in tokens. By default it is automatic: the server gives a model as much cache as free memory allows, which leaves no room for a second model. To run several models side by side, give each one a context. The choice is saved per model and used by every load, including restoration after a restart:

mirmir model inspect Qwen/Qwen3-8B --context 32768   # memory at 32k tokens
mirmir model load Qwen/Qwen3-8B --context 32768      # save it and load
mirmir model load Qwen/Qwen3-8B --context automatic  # back to automatic

The TUI’s load dialog, the web dashboard and Mirmir for Mac show the same choice as a Context control before Load, with the memory it needs. A context is rounded up to whole K/V blocks. The cache is shared by the requests a model serves at once, so it can be larger than the model’s context length, up to that length for each of them (runtime.max_batch_requests); one request still uses at most the model’s context length.

The choice is stored in the model’s TOML file in ~/.config/mirmir/models/ as a block count, which can also be edited by hand:

[runtime]
kv_blocks = 2048   # 2048 blocks × 16 tokens = 32,768 tokens

It takes effect when that model is loaded again, and it is separate from the global runtime.kv_blocks in config.toml.

Set and remove secrets

printf '%s' "$HF_TOKEN" | mirmir config set hugging_face.token
printf '%s' "$MIRMIR_HTTP_API_KEY" | mirmir config set server.api_key
mirmir config remove server.api_key

Environment values take precedence when present. MiRMiR recognizes HF_TOKEN and MIRMIR_HTTP_API_KEY for these two secrets.

CUDA compiler includes

On Linux, MIRMIR_CUDA_INCLUDE_PATH overrides the include directories passed to NVRTC when MiRMiR compiles CUDA kernels at runtime. It is a PATH-style, colon-separated list, useful when the CUDA headers are not under /usr/local/cuda.

Remote access

A MiRMiR server can offer its runtime to other computers: Mirmir for Mac, the CLI and the TUI connect over TLS with an API key. The local Unix socket is unchanged and needs no key. This is separate from the OpenAI-compatible HTTP API, which has its own key.

Turn it on

On the server:

mirmir remote enable                 # listens on 0.0.0.0:7443
mirmir remote enable --bind 10.0.0.4:7443

For a system service, add --system and run as a member of the service’s group. enable starts the listener at once and keeps it enabled across restarts. Only mirmir serve (and the system service) opens the listener; a server that the TUI or a command starts for itself never does. The listener address is changed only with mirmir remote enable --bind and mirmir remote disable, not with mirmir config set. When no key exists yet, it creates a control key and prints it once, together with the certificate fingerprint. Open the port in the firewall only for networks that should reach it.

mirmir remote status                 # listener, fingerprint, keys, recent clients
mirmir remote disable                # stop listening; keys are kept

Keys and scopes

mirmir remote key create --name studio-mac --scope use
mirmir remote key create --name ops-laptop --scope control
mirmir remote key list
mirmir remote key revoke 3f9c0d6e1a2b4c5d
ScopeAllows
useChat, embeddings, reranking, model lists and inspection, catalog search, telemetry, activity and the redacted configuration. A chat, embedding or rerank request may load a model the server offers, and unload another to make room
controlEverything in use, plus explicitly loading, unloading, downloading and removing models, model and runtime settings, and cancelling operations

No key, whatever its scope, can change server.* settings (listener, TLS certificate, HTTP binding, CORS) or the stored HTTP API key and Hugging Face token; those change only on the server itself. A remote key names only models the server offers (configured keys, loaded models and the paths it lists itself); any other name or path is refused as an unknown model. The configuration a remote key reads omits the raw TOML and file paths.

Keys and the listener are managed only on the server itself, over its local socket: no remote key can create or revoke keys. A key reads mmk_<id>_<secret>; the server stores only a digest of the secret. A revoked key is refused on its next request; a stream it already opened, such as a running generation or an activity watch, continues until it ends.

Certificates

By default the server generates a self-signed certificate once and keeps it. Clients trust it by its SHA-256 fingerprint; compare the value they show with mirmir remote status on the server.

  • Mirmir for Mac shows the fingerprint when you add the server, pins it once you trust it and refuses a later change.
  • The CLI and TUI pin nothing on their own: pass the fingerprint with --remote-fingerprint or MIRMIR_REMOTE_FINGERPRINT on every run (read it with mirmir remote fingerprint HOST[:PORT]). Without it they verify the certificate against the system’s CA roots, which refuses the default self-signed certificate.

To use a certificate issued by a CA, which clients verify without pinning:

mirmir config set server.remote_tls_cert /etc/mirmir/remote/fullchain.pem
mirmir config set server.remote_tls_key /etc/mirmir/remote/privkey.pem
mirmir remote enable

Each command saves one key; the listener needs both. While only one is set, mirmir config show reports the other as missing and mirmir remote enable names it; a server that starts with half a pair keeps the listener off and shows why in mirmir remote status. mirmir remote enable restarts a running listener with the new certificate. To return to the self-signed certificate:

mirmir config set server.remote_tls_cert auto
mirmir config set server.remote_tls_key auto
mirmir remote enable

Connect

From Mirmir for Mac: Instances → Add Server, enter host:7443 (a pasted grpcs://host:7443 works too) and the key, check the fingerprint, then Test Connection and Add Server. The key is stored in the Keychain. Edit Server… changes the address, key or trusted certificate and keeps the server’s chats; a removed server’s chats come back when a server at the same address is added again.

From the CLI or TUI on another computer:

mirmir remote fingerprint gpu-server.local    # what the server presents
export MIRMIR_REMOTE_KEY=mmk_…
mirmir --remote gpu-server.local --remote-fingerprint 'AB:CD:…' status
mirmir --remote gpu-server.local --remote-fingerprint 'AB:CD:…' model list
mirmir --remote gpu-server.local --remote-fingerprint 'AB:CD:…'   # TUI

--remote also accepts the grpcs://host:port form that mirmir remote status shows.

The port defaults to 7443. Like --system, a remote client only attaches: it never starts or configures a local runtime.

What is enforced

  • TLS 1.2 or 1.3; a pinned client accepts only the server holding that certificate’s private key.
  • No key or an unknown key: UNAUTHENTICATED. A key whose scope is too narrow, or a local-only method: PERMISSION_DENIED.
  • More than 20 failed attempts within a minute from one IPv4 address or IPv6 /64 network refuse further failed attempts from it for the rest of the minute; a valid key from that network keeps working. A connection without a peer address is refused.
  • Rejections are logged with the address and method, never the key.
  • mirmir remote status lists clients seen in the last 15 minutes with the key name, scope, address and request count.

CLI reference

Running mirmir without a subcommand opens the TUI. Commands expose the same runtime for servers, scripts, model management, configuration, and benchmarks.

Every interactive or one-shot client first tries the local private protocol. If no runtime exists, it starts a temporary one and shuts down only that owned instance when the command finishes. Use mirmir serve when models and runtime state must persist across clients.

CommandPurpose
mirmir servePersistent private gRPC runtime and optional HTTP/Web server.
mirmir statusHealth, loaded models, and latest telemetry as JSON.
mirmir promptStream or benchmark one prompt.
mirmir model …Search, inspect, download, load, unload, and remove models.
mirmir config …Initialize, show, edit, validate, set, remove, and test config.

Serve

mirmir serve
mirmir serve --no-http
mirmir serve --http-bind 127.0.0.1:9090

MIRMIR_HTTP_BIND supplies the same per-process override as --http-bind.

Prompt and benchmarks

Input comes from exactly one of --prompt, --prompt-file, or --stdin. Generation parameters include --max-tokens, --temperature, --top-p, --top-k, --repetition-penalty, and --seed.

printf 'Explain continuous batching.' | mirmir prompt \
  --stdin --model qwen --warmup 1 --samples 5 --json --csv result.csv

--no-stream buffers the final completion. JSON benchmark output contains raw samples and aggregate E2E, TTFT, prefill, and decode distributions.

Model lifecycle

mirmir model list
mirmir model inspect MODEL
mirmir model search QUERY --limit 20
mirmir model pull OWNER/REPO --revision REVISION
mirmir model load MODEL [--force]
mirmir model unload MODEL
mirmir model remove OWNER/REPO

Selectors may be a configured alias, repository ID, local model key, or checkpoint path where the command permits it.

Configuration

mirmir config init
mirmir config show
mirmir config edit
mirmir config validate
mirmir config set KEY [VALUE]
mirmir config remove hugging_face.token
mirmir config test hugging_face.token

When a server is running, commands that change configuration use it as the sole writer.

Remote access

mirmir remote enable [--bind 0.0.0.0:7443]
mirmir remote status
mirmir remote disable
mirmir remote key create --name NAME [--scope use|control]   # default: use
mirmir remote key list
mirmir remote key revoke ID
mirmir remote fingerprint HOST[:PORT]

Global options --remote HOST[:PORT], --remote-key (or MIRMIR_REMOTE_KEY) and --remote-fingerprint (or MIRMIR_REMOTE_FINGERPRINT) attach any command or the TUI to another computer’s server. See Remote access.

Runtime ownership

InvocationRuntime behavior
mirmir serveOwns a persistent runtime until interrupted.
mirmirAttaches to the persistent runtime or owns a temporary TUI runtime.
mirmir model …Uses the shared runtime or owns one for the model operation.
mirmir prompt …Uses the shared runtime or owns one for generation and benchmarks.
mirmir statusReads an already running server and does not start one.

OpenAI-compatible API

mirmir serve listens on http://127.0.0.1:8080 by default. The 0.6.2 public HTTP contract contains:

MethodPathPurpose
GET/healthLiveness, server and private protocol versions.
GET/readySame body as /health, or 503 until startup model restoration completes.
GET/v1/modelsModels ready in the shared runtime.
POST/v1/chat/completionsText or single-image chat, JSON or SSE.
POST/v1/embeddingsSingle or batched text embeddings.
POST/v1/rerankTEI-style document relevance scoring.
POST/v1/decideManual Laya decisions on CPU or macOS Metal; a MiRMiR extension.

The dashboard management API under /api/mirmir/v1 is private to the embedded web application. It is versioned separately and is not an OpenAI client API.

Quick start

mirmir serve
curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/v1/models

Bearer authentication is optional on loopback unless an API key has been configured. When configured, send it on every /v1 request:

curl http://127.0.0.1:8080/v1/models \
  -H "Authorization: Bearer $MIRMIR_HTTP_API_KEY"

Errors

Errors use an OpenAI-style envelope:

{
  "error": {
    "message": "model is not loaded",
    "type": "invalid_request_error",
    "param": null,
    "code": "not_found"
  }
}
Statustype / codeMeaning
400invalid_request_error / invalid_requestInvalid or unsupported request, including a prompt that exceeds the model context or can never fit the K/V cache.
401authentication_error / invalid_api_keyMissing or invalid bearer token.
404invalid_request_error / not_foundUnknown or unloaded model.
408request_error / cancelledThe operation was cancelled.
409invalid_request_error / conflictModel lifecycle conflict, such as a model that is loading or in use.
422model_output_error / invalid_model_outputThe model produced output that violates the request contract, such as a missing, undeclared, malformed, or truncated tool call.
429rate_limit_error / rate_limit_exceededThe request fits the model but not the currently available memory or vision resources; retry later.
503server_error / service_unavailableThe runtime is not ready, for example while restoring models at startup.
500server_error / runtime_errorUnexpected runtime failure.

/v1/chat/completions, /v1/embeddings, and /v1/rerank return 503 until startup restoration finishes. Use GET /ready as a readiness probe and GET /health as a liveness probe.

Chat completions

Use POST /v1/chat/completions with a model already loaded into the shared runtime.

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $MIRMIR_HTTP_API_KEY" \
  -d '{
    "model": "qwen",
    "messages": [{"role": "user", "content": "Hello"}],
    "temperature": 0.2,
    "max_tokens": 256
  }'
import os
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8080/v1",
    api_key=os.environ["MIRMIR_HTTP_API_KEY"],
)

reply = client.chat.completions.create(
    model="qwen",
    messages=[{"role": "user", "content": "Hello"}],
    temperature=0.2,
    max_tokens=256,
)
print(reply.choices[0].message.content)
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://127.0.0.1:8080/v1",
  apiKey: process.env.MIRMIR_HTTP_API_KEY,
});

const reply = await client.chat.completions.create({
  model: "qwen",
  messages: [{ role: "user", content: "Hello" }],
  temperature: 0.2,
  max_tokens: 256,
});
console.log(reply.choices[0].message.content);

Parameters

FieldTypeNotes
modelstringRequired loaded model selector.
messagesarrayRequired non-empty chat history. Assistant tool_calls arguments may be a JSON object or a JSON-encoded string.
streambooleanReturns server-sent events when true.
max_tokensintegerTotal generated-token limit, including reasoning and final content.
max_completion_tokensintegerAlias; must agree with max_tokens if both appear.
min_tokensintegerSuppress end-of-sequence before this many tokens.
ignore_eosbooleanContinue past end-of-sequence until max_tokens.
temperaturenumberSampling temperature.
top_pnumberNucleus cutoff from 0 to 1.
top_kintegerCandidate count; zero leaves it unbounded.
repetition_penaltynumber1 disables the penalty.
seedintegerReproducible sampler seed.
nintegerOnly 1 is supported.
toolsarrayOpenAI function tool declarations.
tool_choicestring or objectauto (default), none, required, or {"type":"function","function":{"name":…}}.
tool_constraintsstringMiRMiR extension: none (default) or schema. See Tool calling.
response_formatobjecttext, json_object, or strict json_schema. See Structured JSON output.
reasoningstringMiRMiR extension: model_default (default), enabled, or disabled.
chat_template_kwargs.enable_thinkingbooleanvLLM-compatible alias for reasoning; must agree with it when both appear.
chat_template_kwargs.medium_effortbooleanUses shorter reasoning when true and the template declares this control.
chat_template_kwargs.truncate_history_thinkingbooleanKeeps or drops earlier reasoning when the template declares this control.
thinking_token_budgetintegerReasoning allowance inside the total limit for schema tools or JSON content.
return_token_idsbooleanStreaming only: adds generated token ids to each chunk.
stream_options.include_usagebooleanAdds usage to the final SSE chunk.

MiRMiR rejects controls it cannot honor instead of silently ignoring them. A non-zero presence_penalty or frequency_penalty and any non-null reasoning_budget or reasoning_effort return 400. Unknown chat_template_kwargs keys are rejected. Explicit effort/history controls are validated against the selected template; they are not universal switches for every model.

Streaming

curl -N http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

Each event contains a chat.completion.chunk. The stream ends with data: [DONE]. Reasoning-capable models emit delta.reasoning_content separately from delta.content. Tool calls arrive as one delta.tool_calls chunk before the final chunk, whose finish_reason is tool_calls. With return_token_ids, each chunk carries choices[0].token_ids, and the final chunk carries any remaining ids.

Closing the connection cancels the generation on the server.

Reasoning

reasoning selects the thinking switch for one text request without changing other requests:

{"model": "qwen", "messages": [{"role": "user", "content": "Return 2+2."}],
 "reasoning": "disabled", "max_completion_tokens": 256}

Explicit enabled and disabled modes require a model chat template that reads enable_thinking, or a supported built-in Qwen or Gemma template. Other templates and image requests reject explicit modes. model_default preserves the template’s own behavior. The max_tokens limit covers reasoning and final content together; reaching it does not guarantee a final answer.

Tool calling

Declare OpenAI function tools and select how the model may use them:

{
  "model": "qwen",
  "messages": [{"role": "user", "content": "What's the weather in Warsaw?"}],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "parameters": {
        "type": "object",
        "properties": {"city": {"type": "string"}},
        "required": ["city"]
      }
    }
  }],
  "tool_choice": "auto"
}

Native Qwen XML (<tool_call><function=…>) and Mistral [TOOL_CALLS] output is normalized into OpenAI message.tool_calls with finish_reason: "tool_calls". MiRMiR validates every call against the request and returns 422 invalid_model_output when the model omits a call that required or a named function demands, calls a tool while tool_choice is none, calls a different or undeclared tool, or emits malformed or truncated arguments.

Schema-constrained tools

On Metal and CUDA, "tool_constraints": "schema" constrains decoding to the named tool’s parameter schema, so the model can only produce a well-formed call. It requires a named tool_choice, repetition_penalty: 1, min_tokens: 0, and ignore_eos: false; other combinations return 400.

  • Root schemas must be objects with declared properties, at most 64 top-level properties and 512 KB. Remote and file $ref retrieval is disabled.
  • Types, required fields, enums, and array cardinality are enforced while decoding; uniqueItems is checked on the completed call (422 on violation). Parameter order follows the declaration.
  • With reasoning: "enabled", reasoning stays unconstrained until the model’s </think> marker, then the tool grammar takes over. thinking_token_budget caps reasoning inside max_tokens and must leave at least two tokens for the delimiter and tool. When omitted, MiRMiR reserves a quarter of max_tokens (at least two tokens) for the tool.
  • A limit that truncates the tool call still returns 422.

One image

The OpenAI-style image_url content part accepts a data URL. Version 0.6.2 supports one PNG, JPEG, WebP, or GIF image per request; an animated GIF contributes its first frame. Image requests cannot set an explicit reasoning mode, effort or history; leave chat_template_kwargs.enable_thinking unset.

IMAGE_DATA=$(base64 < image.jpg | tr -d '\n')
curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"vision-model\",\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/jpeg;base64,$IMAGE_DATA\"}},{\"type\":\"text\",\"text\":\"Describe this image\"}]}]}"

The application limit is 20 MiB per decoded image. The server body limit must also accommodate the larger base64 request.

Structured JSON output

POST /v1/chat/completions accepts response_format on Metal and CUDA:

ValueResult
{"type":"text"} or omittedOrdinary generation.
{"type":"json_object"}A JSON object in message.content.
{"type":"json_schema","json_schema":{...}}JSON matching the supplied schema.
curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer $MIRMIR_HTTP_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen",
    "messages": [{"role":"user","content":"Classify: the parcel arrived damaged."}],
    "reasoning": "disabled",
    "repetition_penalty": 1,
    "max_completion_tokens": 256,
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "classification",
        "strict": true,
        "schema": {
          "type": "object",
          "properties": {"category":{"enum":["damage","delay","other"]}},
          "required": ["category"],
          "additionalProperties": false
        }
      }
    }
  }'

Use a model whose template supports the requested reasoning switch, or omit the switch to retain model defaults. A JSON schema needs a nonblank name; omit strict or set it to true. strict: false returns 400.

Validation and limits

  • Sampling requires repetition_penalty: 1, min_tokens: 0, and ignore_eos: false. Omitted controls use these neutral values, overriding checkpoint defaults.
  • JSON content and tool_constraints: "schema" are exclusive. Required or named tool choices are also rejected. auto/none remain accepted; this mode constrains the final content rather than producing tool calls.
  • Unsupported schemas are rejected before generation. Remote/file schema reference retrieval is disabled. The complete output is validated against the original schema; constraints such as uniqueItems are checked at that final stage, not guaranteed by the token mask.
  • Truncated, invalid, or incomplete JSON produces 422 invalid_model_output for a non-streaming request. This intentionally differs from OpenAI’s finish_reason: "length" with truncated content: MiRMiR does not return an incomplete structured document as a successful completion. Streaming clients receive an error after any partial chunks and must discard that document. Schema validity does not verify factual truth.

JSON response schemas and explicit incompatible sampling settings are rejected before generation or SSE starts. When sampling controls are omitted in schema mode, repetition penalty is 1, minimum tokens is 0, and EOS is honored, even if the checkpoint has different defaults. Valid schemas continue to constrain token selection during decode; completed content is also checked against the original schema. Model-specific tokenizer/backend failures can still occur during setup.

Reasoning and streaming

With supported native reasoning delimiters, reasoning can precede the JSON answer and is returned separately in reasoning_content. Once JSON begins, strings containing text such as <think> remain literal content.

With reasoning: "enabled" and an explicit completion limit, thinking_token_budget bounds reasoning within the total. It must leave at least two tokens for the delimiter and answer. If omitted, the HTTP adapter reserves one quarter of the total (at least two tokens) for structured output. Reserve more for large schemas; this does not guarantee completion.

For stream: true, accumulate delta.content before parsing JSON. Individual chunks need not be valid JSON. The final validation can fail after bytes have already been emitted; handle the stream’s error event rather than treating a partial response as a validated document.

Laya decisions

POST /v1/decide is a manual-testing extension for Laya decision checkpoints. It evaluates one typed question per property of a flat JSON schema and returns decided values plus option probabilities. It is separate from chat completions and the managed model catalog.

curl http://127.0.0.1:8080/v1/decide \
  -H "Authorization: Bearer $MIRMIR_HTTP_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "checkpoint": "/models/laya-multilingual",
    "backend": "cpu",
    "state": {"text": "We were billed twice for March."},
    "schema": {
      "type": "object",
      "properties": {
        "refund": {"type": "boolean"},
        "urgency": {"type": "integer", "minimum": 1, "maximum": 3}
      }
    }
  }'

backend is cpu by default, or metal on macOS. The Rust API also supports CUDA; the HTTP endpoint does not expose it. state accepts either text or a conversation array of turns. Text truncation keeps the beginning; conversation truncation keeps the newest turns.

The response contains values and a fields array with each field’s name, options, probabilities, confidence, concentration, and, for scales, expected_level. Supported questions include booleans, finite enums, and bounded integer scales of at most ten levels. Unsupported schemas (such as free strings or multipleOf), unknown backends, and unreadable paths return 400.

The endpoint uses the server’s bearer authentication and accepts a local checkpoint path. One checkpoint/backend stays resident at a time, on a dedicated worker. Four requests may wait behind the running job; excess work returns 503. Abandoned queued jobs are skipped. A running decision finishes even when its HTTP handler disappears, and its unused result is dropped.

For embedded applications, see the decision API.

Embeddings and reranking

MiRMiR discovers the task from the checkpoint structure. A generation model cannot be used as an embedding model, and it serves reranking only when the checkpoint declares a causal scoring head (see below).

Embeddings

curl http://127.0.0.1:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "embedding",
    "input": ["first text", "second text"],
    "encoding_format": "float"
  }'
FieldTypeNotes
modelstringLoaded embedding model.
inputstring or string[]One input or an ordered batch.
dimensionsintegerOptional prefix dimension, capped by the native size.
prompt_namestringOptional prompt preset declared by the checkpoint.
encoding_formatstringOnly float is supported.

The output follows the OpenAI list shape and returns normalized f32 vectors plus prompt token usage.

Reranking

curl http://127.0.0.1:8080/v1/rerank \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "reranker",
    "query": "native inference",
    "documents": ["Rust and Metal", "a hosted Python service"],
    "return_documents": true
  }'
FieldTypeNotes
modelstringLoaded sequence-scoring or causal-scoring model.
querystringRequired non-empty query.
documentsstring[]Required non-empty candidate list.
max_lengthintegerPair token limit, capped by model context.
top_nintegerReturn only the best n results after sorting.
raw_scoresbooleanReturn logits instead of logistic relevance scores.
return_documentsbooleanInclude each original document in results.

Results are sorted from highest to lowest relevance while retaining the original document index.

Causal rerankers

Since 0.4.0, a decoder checkpoint that ships a sentence-transformers LogitScore module is admitted as a causal reranker on Metal and CUDA. The module declares the “true” and “false” answer tokens and an optional instruction, which is rendered as a system message. MiRMiR renders each query and document through the model’s chat template and scores the pair as the difference between the two answer-token logits at the final position. raw_scores: true returns that difference; otherwise it passes through the logistic function like a cross-encoder score.

Security and remote access

Loopback is the default and safest mode:

[server]
http_bind = "127.0.0.1:8080"
allow_remote = false
web_enabled = false
cors_origins = []

Require an API key

printf '%s' "$MIRMIR_HTTP_API_KEY" | mirmir config set server.api_key

Once configured, clients send:

Authorization: Bearer YOUR_KEY

Bind beyond loopback

A non-loopback bind requires both explicit remote access and an API key:

[server]
http_bind = "0.0.0.0:8080"
allow_remote = true
web_enabled = false

The embedded dashboard deliberately cannot be exposed on this bind. Put TLS, network policy, and any public rate limiting in a trusted reverse proxy or private network boundary.

Browser origins

CORS is disabled until exact origins are listed:

[server]
cors_origins = ["https://app.example.com"]

MiRMiR accepts GET and POST with Authorization and Content-Type headers from configured origins. Do not use * as a substitute for deciding which applications may reach a model server.

Runtime access from other computers

The HTTP API key does not grant control of the runtime. To manage models and chat from Mirmir for Mac or mirmir --remote, enable the separate TLS listener with its own scoped keys; see Remote access.

Embed libmir

libmir is the model-neutral Rust inference library used by MiRMiR. It discovers Hugging Face-style checkpoints, loads a native accelerator backend, creates independent inference sessions, and exposes generation, tool calling, embeddings, reranking, progress, cancellation, metrics, and memory information.

It deliberately does not contain:

  • CLI argument or environment parsing;
  • an HTTP, gRPC, or WebSocket server;
  • a terminal or browser interface;
  • Hugging Face search and download policy;
  • application configuration files.

Those responsibilities belong to an embedding application such as MiRMiR.

Public lifecycle

ModelDescriptor::inspect(path)
        |
        v
Library::new(RuntimeConfig)
        |
        v
Library::load(path) -> Model
        |
        +-- Model::generate(&GenerationRequest, ...)
        +-- Model::embed(...)
        +-- Model::rerank(...)
        +-- Model::session() -> prefill / decode
        +-- Model::unload()

Applications should depend on the top-level libmir package and import from its root. The internal workspace crates, including the CUDA and Metal adapters and their configuration types, are implementation details and are not re-exported; backend settings are adjusted through RuntimeConfig fields (see Metal and CUDA).

Choose an entry point

NeedAPI
Inspect without allocating accelerator weightsModelDescriptor::inspect
Load a checkpointLibrary::load
High-level chat generationModel::generate
Tool calls and schema-constrained argumentsConversation::tools, GenerationRequest::tool_constraints
JSON contentGenerationRequest::response_format
Laya decisionsdecision::DecisionModel on CPU, Metal, or CUDA
Cooperative cancellationModel::generate_cancellable
Single-image generationModel::generate_image_cancellable
Dense vectorsModel::embed
Document scoring (cross-encoder or causal)Model::rerank
Manual token loopModel::prepare, Model::session, Session::prefill, Session::decode
Capacity planningModelDescriptor::memory_estimate, Library::memory_snapshot

Start with Add libmir to a project.

Add libmir to a project

Version 0.6.2 enables no accelerator by default. Select the backend explicitly.

Apple Metal

[dependencies]
libmir = { version = "0.6.2", default-features = false, features = ["metal"] }

The Metal feature currently requires Apple Silicon, a full Xcode installation with the separately installed Metal toolchain, and native MLX.

NVIDIA CUDA

[dependencies]
libmir = { version = "0.6.2", default-features = false, features = ["cuda"] }

CUDA currently targets Linux with the NVIDIA driver, CUDA Toolkit 13.x, NVRTC, CCCL/CUB headers, and CUTLASS 4.4.2 headers.

Minimal generation

use std::io::{self, Write};

use libmir::{
    Conversation, GenerationOverrides, GenerationRequest, Library, Message,
    RuntimeConfig,
};

fn main() -> libmir::Result<()> {
    let model_path = std::env::args_os().nth(1).expect("model path");
    let library = Library::new(RuntimeConfig::default());
    let model = library.load(
        model_path,
        GenerationOverrides::default(),
        &mut |progress| eprintln!("{}: {}", progress.stage(), progress.detail()),
    )?;

    let request = GenerationRequest {
        conversation: Conversation {
            messages: vec![Message {
                role: "user".into(),
                content: "Explain paged K/V cache briefly.".into(),
                reasoning_content: None,
                tool_calls: None,
                tool_call_id: None,
            }],
            ..Conversation::default()
        },
        options: GenerationOverrides {
            max_tokens: Some(256),
            temperature: Some(0.0),
            top_p: Some(1.0),
            top_k: Some(0),
            repetition_penalty: Some(1.0),
            ..GenerationOverrides::default()
        },
        seed: Some(7),
        ..GenerationRequest::default()
    };

    let output = model.generate(&request, &mut |_| {}, &mut |token| {
        print!("{}", token.text);
        let _ = io::stdout().flush();
    })?;

    println!("\nfinish_reason={}", output.finish_reason);
    Ok(())
}

GenerationRequest separates the conversation (messages, tools, tool_choice) from request-scoped sampling overrides. Unset overrides keep the generation settings resolved when the model was loaded.

Library initializes its native backend lazily. Loading inspects the checkpoint first, resolves the execution contract, then reports real loading progress through the callback.

The same lifecycle is available as the generate example in the libmir repository.

Inspect and load models

Inspection is separate from accelerator allocation. Use it to reject an unsupported checkpoint or present its capabilities before committing device memory.

use libmir::{GenerationOverrides, ModelDescriptor};

let descriptor = ModelDescriptor::inspect(
    "/models/qwen",
    GenerationOverrides::default(),
)?;
let manifest = descriptor.manifest()?;

println!("id={}", manifest.id);
println!("context={}", manifest.context_len);
println!("task={:?}", descriptor.task());
println!("quantization={:?}", manifest.quantization);
println!("generation={:?}", descriptor.generation());
Ok::<(), libmir::Error>(())

Discovery is based on configuration values, tokenizer assets, tensor names and shapes, and the resulting execution contract. A recognizable repository or architecture name alone does not admit a checkpoint.

Estimate memory

use libmir::{GenerationOverrides, ModelDescriptor, RuntimeConfig};

let descriptor = ModelDescriptor::inspect(
    "/models/qwen",
    GenerationOverrides::default(),
)?;
let estimate = descriptor.memory_estimate(&RuntimeConfig::default());

println!("weights={} bytes", estimate.weight_bytes);
println!("kv={} bytes", estimate.kv_cache_bytes);
println!("workspace={} bytes", estimate.workspace_bytes);
println!("required={} bytes", estimate.required_bytes);
Ok::<(), libmir::Error>(())

This is a conservative planning value, not a reservation. Compare it with Library::memory_snapshot() on the backend that will load the model.

Load and unload

use libmir::{GenerationOverrides, Library, RuntimeConfig};

let library = Library::new(RuntimeConfig::default());
let model = library.load(
    "/models/qwen",
    GenerationOverrides::default(),
    &mut |event| {
        let count = event.count();
        eprintln!("{} {}/{} {}", event.stage(), count.current(), count.total(), event.unit());
    },
)?;

println!("backend={:?}", model.engine().target());
model.unload()?;
Ok::<(), libmir::Error>(())

Model is cheaply cloneable. Model::unload succeeds only when the value is the sole remaining owner; active clones or sessions produce Error::ModelInUse. Dropping a Session releases its backend and K/V resources automatically.

Several resident models

Library::load admits a model only when its estimate fits the accelerator budget, counting K/V memory that already loaded models could give back; otherwise it returns Error::MemoryAdmission. Library::load_with_options with ModelLoadOptions { allow_memory_overcommit: true } skips that check, although backend allocation may still fail. Library::reclaimable_kv_bytes reports that amount. After loading or unloading, Library::rebalance_kv_caches resizes idle models’ measured K/V caches to the memory the others leave and returns how many changed.

Generate and stream

Model::generate is the preferred high-level text API. It renders the checkpoint chat template, tokenizes, creates an independent session, performs prefill, samples and decodes tokens, normalizes model-specific channels, and returns metrics.

Content and reasoning

The token callback receives GenerationToken values with a semantic channel. Native think markers and channel headers are removed from the public contract. Each token carries its id and, in preceding_ids, any generated identifiers that were folded into it without producing visible text of their own.

The final GenerationOutput contains separate text, reasoning, and tool_calls fields, every generated token_ids entry (including a terminal stop token when emitted), prompt_tokens, a finish_reason such as "stop", "tool_calls", or "max_tokens", and metrics.

let output = model.generate(
    &request,
    &mut |progress| eprintln!("{}: {}", progress.stage(), progress.detail()),
    &mut |token| println!("{:?}: {}", token.channel, token.text),
)?;

println!("answer={}", output.text);
println!("reasoning={}", output.reasoning);
println!("metrics={:#?}", output.metrics);
Ok::<(), libmir::Error>(())

ProgressEvent is an enum with one variant per stage: LoadWeights, InitializeRuntime, Warmup, PrefillTokens, and DecodeTokens. Use stage(), unit() (Byte, Item, or Token), count() for the current()/total() pair, and detail() rather than parsing display text.

Reasoning controls

GenerationRequest::reasoning selects the template’s thinking switch: ReasoningMode::ModelDefault (the default) preserves the model’s rendering, while Enabled and Disabled require a Jinja enable_thinking input or a supported built-in Qwen or Gemma template. Unsupported templates and image requests reject explicit modes.

GenerationOverrides::max_tokens counts every generated token, including reasoning. GenerationRequest::reasoning_cycle decides what happens when a reasoning loop is detected:

PolicyBehavior
ReasoningCyclePolicy::ExitReasoning { min_tokens: 64 }Default. Closes a supported reasoning channel after three consecutive repeated token cycles past the token gate.
ReasoningCyclePolicy::RecoverResumes greedy decoding with a repetition penalty.

Scattered repetitions, such as quoting the same source twice, do not force an exit. The completion limit still bounds reasoning that never reaches an answer.

Tool calls

Declare functions in Conversation::tools and select a ToolChoice: Auto (the default), None, Required, or Function(name).

use libmir::{FunctionDefinition, Tool, ToolChoice};

request.conversation.tools = vec![Tool {
    kind: "function".into(),
    function: FunctionDefinition {
        name: "weather".into(),
        description: Some("Current weather for a city".into()),
        parameters: serde_json::json!({
            "type": "object",
            "properties": { "city": { "type": "string" } },
            "required": ["city"]
        }),
    },
}];
request.conversation.tool_choice = ToolChoice::Function("weather".into());

Native JSON (Mistral-style) and Qwen XML tool calls are parsed and normalized into GenerationOutput::tool_calls as a JSON array of {id, type, function: {name, arguments}} objects, and finish_reason becomes "tool_calls". Malformed output, an undeclared tool, a call that contradicts tool_choice, or a missing required call returns Error::InvalidToolCall. Feed prior calls back through Message::tool_calls and results through Message::tool_call_id.

Schema-constrained tools

Set GenerationRequest::tool_constraints to ToolConstraints::Schema to mask sampling with the named tool’s JSON Schema. The mode is opt-in and currently requires:

  • the Metal or CUDA backend with device sampling;
  • ToolChoice::Function(name) naming a declared tool;
  • a native XML <tool_call>/<function=...> template, such as Qwen’s;
  • repetition_penalty = 1, min_tokens = 0, and ignore_eos = false.

Schemas the grammar compiler cannot express are rejected before generation. Completed arguments are validated against the original schema, including uniqueItems, and violations return Error::InvalidToolCall.

With reasoning enabled, the model may think before the constrained call. GenerationRequest::reasoning_token_budget optionally caps that reasoning; at the cap only the native reasoning delimiter can be sampled. The budget requires ReasoningMode::Enabled and ToolConstraints::Schema, and max_tokens must leave room for the delimiter and the tool output.

JSON content

Set request.response_format = libmir::ResponseFormat::JsonSchema(schema) to constrain final content instead of a tool call. Use {"type":"object"} for an arbitrary JSON object. The default is ResponseFormat::Text.

The Metal/CUDA sampling restrictions are the same as schema tools. JSON content rejects ToolConstraints::Schema and required/named ToolChoice. Native reasoning remains separate; markers inside JSON strings stay literal. The complete content is checked against the original schema. A partial token callback is not a validated document. Invalid final content returns Error::InvalidResponseFormat, mapped to invalid_model_output by the application.

reasoning_token_budget also supports JSON content with enabled reasoning and compatible atomic reasoning delimiters. Direct Rust callers set this allowance explicitly; the HTTP adapter’s default quarter-budget reservation is not a library default.

Cooperative cancellation

CancellationToken is thread-safe and all clones observe the same state. Text prefill observes it at cache waits and scheduler boundaries, and decode checks it between steps.

use libmir::CancellationToken;

let cancellation = CancellationToken::new();
let signal = cancellation.clone();

std::thread::spawn(move || {
    std::thread::sleep(std::time::Duration::from_secs(2));
    signal.cancel();
});

let result = model.generate_cancellable(
    &request,
    &mut |_| {},
    &mut |token| print!("{}", token.text),
    &cancellation,
);
drop(result);

Cancellation is cooperative and returns Error::Cancelled. It does not interrupt an individual accelerator kernel in the middle of execution.

One image

Insert IMAGE_PLACEHOLDER exactly once in the message content and provide an encoded PNG, JPEG, WebP, or GIF:

use libmir::{CancellationToken, IMAGE_PLACEHOLDER};

let mut request = request;
request.conversation.messages[0].content =
    format!("{IMAGE_PLACEHOLDER}\nDescribe this image.");
let image = std::fs::read("image.jpg")?;

let output = model.generate_image_cancellable(
    &request,
    &image,
    &mut |_| {},
    &mut |token| print!("{}", token.text),
    &CancellationToken::new(),
)?;
drop(output);
Ok::<(), Box<dyn std::error::Error>>(())

The checkpoint must expose a complete supported vision contract. Resource limits come from RuntimeConfig::vision. Image prefill observes cancellation before and after the complete prefill, then between decode steps.

Embeddings and reranking

The same loaded Model type exposes task-specific methods. Calling the wrong method for a checkpoint returns Error::TaskMismatch.

Embeddings

use libmir::{EmbeddingRequest, GenerationOverrides, Library, RuntimeConfig};

let model = Library::new(RuntimeConfig::default()).load(
    "/models/embedding",
    GenerationOverrides::default(),
    &mut |_| {},
)?;

let output = model.embed(EmbeddingRequest {
    inputs: vec!["first text".into(), "second text".into()],
    dimensions: None,
    prompt_name: None,
})?;

println!("vectors={}", output.embeddings.len());
println!("tokens={}", output.prompt_tokens);
Ok::<(), libmir::Error>(())

dimensions may retain a prefix no larger than the checkpoint’s native dimension. prompt_name must name a preset declared by the checkpoint.

Reranking

use libmir::{GenerationOverrides, Library, RerankRequest, RuntimeConfig};

let model = Library::new(RuntimeConfig::default()).load(
    "/models/reranker",
    GenerationOverrides::default(),
    &mut |_| {},
)?;

let output = model.rerank(RerankRequest {
    query: "native inference".into(),
    documents: vec!["Rust and Metal".into(), "a hosted service".into()],
    max_length: None,
    raw_scores: false,
})?;

for result in output.results {
    println!("{} {:.4} {}", result.index, result.score, result.document);
}
Ok::<(), libmir::Error>(())

Results are relevance-sorted but preserve each candidate’s original index. Set raw_scores to return the raw score instead of its logistic transform.

Two checkpoint contracts are served by Model::rerank, reported by ModelDescriptor::task_plan():

PlanCheckpointRaw score
TaskExecutionPlan::SequenceScoringSequence-classification cross-encoderClassifier logit for the query/document pair.
TaskExecutionPlan::CausalScoringCausal decoder with a sentence-transformers LogitScore moduleFinal-position logit of the true token minus that of the false token.

Causal scoring runs on Metal and CUDA. It renders the checkpoint chat template with query and document turns, preceded by the checkpoint’s default prompt as a system instruction when one is declared. The rendered prompt is never truncated; a candidate longer than max_length returns Error::Context.

Typed decisions

libmir::decision loads Laya checkpoints independently of Library and the generation scheduler. Its ModernBERT encoder answers option, boolean, and scale questions about text or conversation state.

libmir = { version = "0.6.2", default-features = false, features = ["cpu"] }
serde_json = "1"
use libmir::decision::{DecisionBackend, DecisionModel, DecisionSchema, DecisionState};

let model = DecisionModel::load("/models/laya-multilingual", DecisionBackend::Cpu)?;
let schema = DecisionSchema::from_json_schema(&serde_json::json!({
    "type": "object",
    "properties": {"refund": {"type": "boolean"}}
}))?;
let values = model.decide_schema(&DecisionState::text("We were billed twice."), &schema)?;
println!("{}", serde_json::Value::Object(values));
Ok::<(), libmir::Error>(())

Select DecisionBackend::Metal with the metal feature for Apple GPUs, or DecisionBackend::Cuda(DecisionPrecision::F32) / Bf16 with cuda for NVIDIA GPUs. CPU and Metal evaluate in FP32; CUDA offers FP32 or BF16 matrix products with FP32 normalization, softmax, and residuals. These precisions have separate numerical tolerances in the reference tests.

decide returns typed Answer values with option probabilities and verdicts; decide_schema projects them into a JSON object. rows and logits expose tokenized questions and raw option logits for validation.

The CPU backend uses mircup: Apple Accelerate on macOS and multithreaded GEMM elsewhere. Enabling cpu does not provide general-purpose text generation, embeddings, or reranking through Library.

Sessions and K/V cache

Use Model::generate unless an application needs direct control over the token loop. The low-level Session API owns independent K/V state while sharing the loaded model and its cache arena.

use libmir::{RuntimeError, SamplingLogits};

let prepared = model.prepare(&request.conversation)?;
let mut session = model.session();
let prefill = session.prefill(
    &prepared.tokens.token_ids,
    SamplingLogits::None,
    &mut |_| {},
)?;
let mut next = prefill.next_token.ok_or_else(|| {
    RuntimeError::Backend("device sampling returned no token".into())
})?;

for _ in 0..32 {
    print!("{}", model.descriptor().tokenizer().decode(&[next])?);
    if model.descriptor().tokenizer().stop_token_ids().contains(&next) {
        break;
    }
    next = session
        .decode(next, SamplingLogits::None)?
        .event
        .token_id
        .ok_or_else(|| RuntimeError::Backend("no token".into()))?;
}

println!("{:#?}", session.cache_stats());
Ok::<(), libmir::Error>(())

Model::prepare renders and tokenizes a Conversation and validates it against the context window. SamplingLogits::None requests device sampling when supported. Other variants can expose full or candidate logits for an application-owned sampler.

Session::prefill_cancellable accepts a CancellationToken and observes it between scheduled accelerator steps. Drop a session whose prefill was cancelled instead of reusing it.

Cache behavior

The library uses page-backed K/V storage. Full ready blocks can become shared prefix entries, and sessions use copy-on-write when a shared tail must diverge. Session::cache_stats reports:

  • resolved storage dtype and quantization mode;
  • total, free, and used blocks;
  • cached prefix count;
  • prefix probes, hits, misses, and matching token counts.

Dropping the session releases its block table. A multimodal prefill deliberately does not publish reusable prefix-cache entries.

Runtime configuration

RuntimeConfig is an explicit Rust value. libmir never reads command-line arguments, dotenv files, or environment variables.

use libmir::{CachedPrefillPolicy, KvCacheDType, Library, RuntimeConfig};

let mut config = RuntimeConfig::default();
config.kv_cache.block_size = 32;
config.kv_cache.block_count = 2_048;
config.kv_cache.dtype = KvCacheDType::Auto;
config.automatic_kv_cache = false;
config.scheduler.max_batch_requests = 8;
config.scheduler.max_batch_tokens = 4_096;
config.scheduler.decode_batch_wait_us = 200;
config.scheduler.decode_priority_burst = 8;
config.scheduler.cached_prefill_policy = CachedPrefillPolicy::BackendDefault;
config.memory.reserve_percent = Some(20);
config.vision.max_pixels = Some(1_048_576);
config.vision.memory_percent = 70;

let library = Library::new(config);
drop(library);

Shared configuration

FieldDefaultPurpose
kv_cache.block_size16Tokens per page-backed cache block.
kv_cache.block_count4096Physical block count.
kv_cache.dtypeAutoBackend-resolved cache storage.
automatic_kv_cachetrueDerive block count from model shape and available memory.
scheduler.max_batch_requests16Maximum requests in a decode batch.
scheduler.max_batch_tokens8192Prompt plus output token budget.
scheduler.prefill_batch_wait_us200Prefill collection window.
scheduler.decode_batch_wait_us200Short batching window.
scheduler.decode_priority_burst8Decode fairness control.
scheduler.cached_prefill_policyBackendDefaultAdmission of cached continuations during decode.
scheduler.prefill_decode_policyBackendDefaultWhen newly admitted prompts begin decoding.
scheduler.prefill_refill_policyClosedAdmission of short prompts into a running long prefill.
scheduler.speculationNoneOptional Speculation { draft_tokens: NonZeroUsize } for a compatible CUDA MTP checkpoint.
memory.reserve_percentbackend-awarePercentage of total accelerator memory retained as headroom.
memory.reserve_bytesbackend-awareAbsolute reserve floor used with the percentage policy.
vision.max_pixelscheckpoint capMaximum resized image area.
vision.attention_budget_bytesautomaticVision attention workspace.
vision.memory_percent80Available memory used by automatic budgeting.

Scheduling policies

The scheduling policy enums default to each backend’s normal behavior. The alternatives are explicit latency trade-offs:

PolicyEffect
CachedPrefillPolicy::InterleaveOneBlockAdmits one cached queue head with at most one K/V block of work per step during resident decode. Lowers refill first-token latency at the cost of decode latency.
PrefillDecodePolicy::CompleteCohortFinishes a bounded admitted cohort before publishing its first tokens. Improves decode occupancy at the cost of early first-token latency.
PrefillRefillPolicy::ShortPromptLets queued prompts of at most 128 tokens join a running long prefill at CUDA chunk boundaries. Requires a completion-first CUDA runner and cannot be combined with CompleteCohort.

Backend-specific configuration

ModelLoadOptions::kv_blocks overrides the global K/V block count for one model. Use it to bound a small model’s cache independently of a larger chat model; the value is a positive NonZeroU32.

RuntimeConfig::metal and RuntimeConfig::cuda exist only when their Cargo feature is enabled. A Metal-only consumer cannot accidentally configure CUDA, and a backend-neutral build has neither field. See Metal and CUDA.

Metal, CUDA, and CPU

The high-level Library, Model, Session, generation, embedding, and reranking APIs are backend-neutral. Cargo features choose which native adapter is compiled.

The facade does not re-export backend objects or backend configuration types. Enable a feature, then adjust the defaults of RuntimeConfig::metal or RuntimeConfig::cuda in place. Library creates the backend lazily, and backend failures surface as libmir::Error.

Metal

libmir = { version = "0.6.2", default-features = false, features = ["metal"] }

Metal uses mirtal for explicit streams, device memory, compiled kernels, graphs, and model-neutral operations. RuntimeConfig::metal includes batch, cache, fusion, tuning, and diagnostic policies.

use libmir::{Library, MetalDecodeReservation, RuntimeConfig};

let mut config = RuntimeConfig::default();
config.metal.cache.decode_reservation = MetalDecodeReservation::GenerationBudget;
config.metal.tuning.cache_directory = Some("/var/cache/my-app/tuning".into());
let library = Library::new(config);
drop(library);

MetalDecodeReservation::OnePage is the default. GenerationBudget instead reserves up to the request’s generation budget from reclaimable spare K/V pages.

On Apple Silicon, host and accelerator share unified memory. Memory snapshots distinguish active allocations from reusable cached allocations.

CUDA

libmir = { version = "0.6.2", default-features = false, features = ["cuda"] }

CUDA uses mircuda for contexts, streams, memory, NVRTC compilation, kernels, graphs, and CUTLASS-backed plans. RuntimeConfig::cuda selects the device, pool behavior, include paths, persistent cubin cache, startup tuning, and execution-planning policy.

use libmir::{Library, RuntimeConfig};

let mut config = RuntimeConfig::default();
config.cuda.device_ordinal = 0;
config.cuda.nvrtc_cache_directory = Some("/var/cache/my-app/cubins".into());
config.cuda.tuning.cache_directory = Some("/var/cache/my-app/tuning".into());

let library = Library::new(config);
println!("backend={:?}", library.backend_target()?);
let memory = library.memory_snapshot()?;
println!("{}: free={:?} total={:?}", memory.source, memory.available_bytes, memory.total_bytes);
Ok::<(), libmir::Error>(())

The first call that needs the backend creates real device resources, so it is fallible. CUDA memory reporting includes driver-visible free and total device memory plus pool allocations.

The cuda-diagnostics feature implies cuda and compiles additional kernel diagnostics, such as activation fingerprints. Leave it disabled in production.

CPU decisions

Enable cpu to use DecisionModel with the native mircup backend. This provides Laya decisions independently of the generation scheduler; it does not make Library a general CPU chat, embedding, or reranking engine. See Typed decisions.

Capability differences

Version 0.6.2 does not imply identical realization for every encoding on both backends. A format may use direct execution on one platform and a bounded, persistent accelerator-side conversion on the other. Schema-constrained tool generation runs on both: CUDA applies the grammar mask in a fused sampling kernel, while Metal masks the step’s full device logits and reads back only the selected token, so constrained Metal steps skip the pipelined device-token lookahead. Admission comes from the discovered execution contract and backend capability, not a family-name allowlist. Inspect the exact checkpoint on the target machine before promising support.

See Model compatibility for the release matrix.

Nemotron-H’s state-space generation and MTP run on CUDA only. Laya decisions run on all three backends through the Rust API. The HTTP decision endpoint has a narrower CPU/Metal contract. Metal builds target native MLX 0.32.4.

OpenAI SDK clients

MiRMiR can be used by clients that allow an OpenAI-compatible base URL. Load a model, run mirmir serve, and point the client at http://127.0.0.1:8080/v1.

Python

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8080/v1",
    api_key="local-dev",
)

stream = client.chat.completions.create(
    model="qwen",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)

When MiRMiR has no API key on loopback, the SDK still requires a non-empty placeholder api_key locally.

JavaScript and TypeScript

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://127.0.0.1:8080/v1",
  apiKey: "local-dev",
});

const result = await client.chat.completions.create({
  model: "qwen",
  messages: [{ role: "user", content: "Hello" }],
});

console.log(result.choices[0].message.content);

Browser applications additionally need an exact server.cors_origins entry. Do not embed a remote-access API key into publicly shipped frontend code.

Compatibility boundary

Use only the endpoints and fields documented in this book. Function tool calling through tools and tool_choice is supported for chat completions; see Tool calling. Support for an OpenAI base URL does not imply support for Responses, Assistants, audio, image generation, structured response_format output, or undocumented sampling extensions. Clients that send non-zero presence_penalty or frequency_penalty, or a reasoning_effort, receive 400 and must omit those fields.

Telemetry and benchmarking

MiRMiR publishes the same runtime state to the TUI, web dashboard, CLI status, and private gRPC clients.

Metrics

MetricMeaning
TTFTTime from request start to the first generated token.
PrefillPrompt tokens processed per second.
DecodeGenerated tokens processed per second after prefill.
E2E throughputCompletion tokens divided by complete request time.
MemoryActive and cached accelerator or unified-memory occupancy.
K/V occupancyUsed blocks, free capacity, and prefix-cache effectiveness.

The server samples runtime metrics every second, keeps 900 samples, and persists telemetry atomically under ~/.local/state/mirmir/telemetry.toml. The web dashboard renders the latest 60 samples and restores them after a restart. Shutdown waits up to two seconds for the final history flush. If a system read or filesystem write stalls, the server logs a warning and continues shutdown; the last unflushed samples may be lost. The history worker holds no model or inference-runtime handles. History memory samples report host system memory; a busy logical K/V cache retains its previous reading until the next sample.

Inspect current state

mirmir status

The command prints server health, ready models, and the latest telemetry snapshot as JSON.

Benchmark a prompt

mirmir prompt \
  --model qwen \
  --prompt "Explain continuous batching." \
  --warmup 1 \
  --samples 5 \
  --seed 7 \
  --json \
  --csv benchmark.csv

The report contains individual samples and mean, median, p50, p95, population standard deviation, minimum, and maximum for E2E, TTFT, prefill, decode, and their durations.

Logging

mirmir serve emits structured lifecycle and request logs at INFO. Narrow or expand application targets explicitly:

RUST_LOG=mirmir=trace,libmir=debug mirmir serve
RUST_LOG=mirmir=warn,libmir=warn mirmir serve

Model compatibility

MiRMiR 0.6.2 supports a checkpoint only when its complete execution contract is valid for the selected backend. A repository name, architecture label, or SafeTensors container is useful context, but none is a compatibility guarantee by itself.

What MiRMiR inspects

Before admission, MiRMiR combines:

  • the task: generation, vision-language generation, embeddings, or reranking with an encoder or a causal decoder;
  • semantic architecture: attention, normalization, positional encoding, feed-forward composition, routing, pooling, and activation;
  • tokenizer, processor, chat template, and protocol tokens;
  • tensor names, roles, shapes, dtypes, packing, and companion scales;
  • physical weight encoding and quantization geometry;
  • Metal or CUDA capabilities and the available accelerator memory.

Remote search and pull use bounded header preflight. Local inspection evaluates the complete checkpoint. Compatibility is reported independently as supported, partial, unsupported, or unknown for each backend.

Validated model systems

The 0.6.2 execution contracts cover these systems when every required component is present:

SystemCovered composition
Dense decodersGrouped-query attention, RoPE variants, tied weights, and projection biases.
Routed MoETop-k routing, shared experts, dense-plus-routed blocks, and clamped routed experts.
Hybrid attentionFull and sliding softmax attention combined with linear or recurrent mixer layers.
Nemotron-H (CUDA only)Mamba-2, attention without RoPE, latent routed experts, and optional MTP for checkpoints with a prediction head. Nemotron Omni checkpoints with the RADIO vision encoder also accept images on CUDA.
Laya decisionsModernBERT decision encoder on CPU, Metal, or CUDA through the Rust API. HTTP exposes CPU and macOS Metal.
EmbeddingsEncoder checkpoints with declared pooling and optional output normalization.
Sequence scoringEncoder classification and reranking with typed pooling and output boundaries.
Causal scoringDecoder checkpoints with a sentence-transformers LogitScore module, reranked from final answer-token logits on Metal and CUDA.
Vision-languageVision tower, projector, image processor, protocol tokens, and text decoder admitted together.

These are structural categories, not a closed family allowlist. Inspect the exact checkpoint before promising support.

Gemma 4 retains its Metal path and gains accelerated CUDA prefill for sliding attention and 512-wide global heads. Support claims do not imply fresh performance measurements on every release. Historical matrices identify their model revision, backend, and measured software version.

Gemma 4’s omitted normalization and scaling settings have built-in defaults: Q/K/V RMS normalization, attention scale 1.0, and embedding multiplier sqrt(hidden_size). A standard supported checkpoint needs no manually created sidecar. Explicit configuration values override the defaults; a complete mir-model-spec.toml overrides the resolved semantic contract. Metal rejects normalization or scaling overrides its current execution path cannot implement, with an error before weights are loaded. CUDA executes its supported declared semantics. The dense Gemma 4 12B composition remains a CUDA path; this change does not add a new dense GELU runner to Metal.

The validated Nemotron-3.5-Super 120B-A12B NVFP4 text weights occupy about 66.6 GiB in the CUDA implementation, with higher runtime peaks; the vision encoder adds about 1.9 GB of resident weights. Active parameter count does not describe resident weight memory: this checkpoint is not a 64 GB Mac workload, and its state-space path is not implemented on Metal.

Checkpoint ecosystems

MiRMiR 0.6.2 reads Hugging Face and MLX Community repositories stored as SafeTensors. It interprets Transformers configuration, tokenizer assets, shard indexes, and chat templates natively; Transformers, PyTorch, bitsandbytes, and Python are not runtime dependencies.

GGUF files are not admitted by the 0.6.2 runtime.

Weight encodings

SafeTensors encodingMetalCUDANotes
F32, F16, BF16SupportedSupportedNative dense execution.
MLX grouped affine 2/3/4/5/6/8-bitSupportedSupportedPacked weights remain accelerator-resident.
Compressed Tensors INT8 and INT4SupportedSupportedTyped packing, grouping, scale, and zero-point contracts.
AWQSupportedSupportedBackend-specific direct execution or persistent device repacking.
GPTQSupportedSupportedIncludes typed format and activation-order contracts.
FP8 E4M3 and E5M2SupportedSupportedScale direction and geometry are validated explicitly.
MXFP4 and MXFP8SupportedSupportedBlock values and scale representation remain distinct.
NVFP4Supported (role-specific)Supported (native)Metal converts ordinary matrices once and executes routed banks directly.
bitsandbytes NF4 and FP4SupportedSupportedCodebook, block scales, nested quantization, and compute dtype are preserved.

Supported means the format has a complete contract, backend operation path, real-checkpoint validation, deterministic generation coverage where applicable, and bounded accelerator-memory behavior. It does not mean that every model architecture using that encoding is implemented.

Device conversion policy

MiRMiR may execute a checkpoint’s packed representation directly or convert it once into a persistent backend-native representation. Conversion is admitted only when it is bounded, runs on the accelerator, is included in memory preflight, and does not leave a full dense host-side shadow of the weights.

Check a checkpoint

mirmir model inspect OWNER/REPOSITORY
mirmir model load OWNER/REPOSITORY

Inspection explains the discovered task, format, backend admission, memory estimate, and safe context capacity. A failed structural or backend check must be resolved with a compatible checkpoint; --force cannot bypass it.

Limits and compatibility

These are the 0.6.2 product boundaries, not commitments for later releases.

HTTP and media

  • chat completions support n = 1;
  • embedding output supports encoding_format = "float";
  • one image may be attached to a generation request;
  • supported image containers are PNG, JPEG, WebP, and GIF;
  • decoded application image input is limited to 20 MiB;
  • the default HTTP body limit is 29,360,128 bytes;
  • default request timeout is 300 seconds;
  • default HTTP concurrency is 16 requests;
  • CORS is disabled until exact origins are configured;
  • /v1 inference endpoints return 503 until startup model restoration completes;
  • presence_penalty, frequency_penalty, reasoning_budget, and reasoning_effort are rejected rather than ignored;
  • response_format supports text, JSON objects, and strict JSON schemas; JSON content rejects required/named tool choices and schema-tool mode;
  • schema conformance is checked on completion; streamed content is provisional until final validation succeeds.

Tool calling

  • tool calls are validated against tools and tool_choice; violations return 422;
  • schema-constrained tool decoding runs on Metal and CUDA and requires a native XML tool template such as Qwen’s;
  • constrained root schemas are objects with at most 64 top-level properties and 512 KB; remote and file $ref retrieval is disabled;
  • thinking_token_budget requires enabled reasoning, text input, schema mode, and an explicit max_tokens.

The same budget control supports JSON content. See Structured JSON output.

Web dashboard

  • the dashboard is opt-in;
  • it is restricted to loopback;
  • its local session lasts eight hours;
  • the OpenAI bearer token cannot authorize dashboard administration.

Models

  • compatibility is determined from configuration, tokenizer, tensor schema, shapes, quantization, and backend capabilities;
  • the 0.6.2 container boundary is SafeTensors from supported Hugging Face, Transformers, MLX, and quantization-toolchain layouts;
  • a family or repository name is not a compatibility guarantee;
  • Metal and CUDA have different admitted checkpoint coverage;
  • Nemotron-H generation and MTP require CUDA; Metal rejects its state-space mixer;
  • CPU supports Laya decisions, not general chat generation;
  • /v1/decide exposes CPU/Metal only; CUDA decisions use the Rust API;
  • a forced load bypasses memory-fit policy, not structural validation;
  • model load cancellation waits for a future safe backend interruption boundary.

The complete encoding matrix is documented under Model compatibility.

libmir stability

libmir is pre-1.0. Pin 0.6.2 and review release notes before upgrading. Since 0.4.0 the top-level facade is the only supported entry point: the internal workspace crates are no longer re-exported.

Documentation roadmap

The 0.6.2 documentation covers the public control surfaces, HTTP contract, model compatibility boundary, and primary libmir lifecycle. Later passes will deepen these areas without changing their URLs.

Product guides

  • screenshot-led TUI pages for Dashboard, Models, Chat, and Settings;
  • screenshot-led web pages for search, load preflight, chat, images, settings, and cancellation;
  • complete environment-variable reference;
  • model configuration file reference;
  • recovery and troubleshooting decision trees;
  • platform-specific source build guides.

Developer reference

  • generated item links to docs.rs for each public libmir type;
  • progress event and generation metrics field reference;
  • error taxonomy and recovery guidance;
  • concurrent session and throughput patterns;
  • complete image preparation example;
  • explicit Metal and CUDA policy tables;
  • migration notes for each pre-1.0 release.

Integration policy

Tool-specific integration pages will be added only after an end-to-end test against the documented MiRMiR release. A configurable OpenAI base URL alone is not enough to claim full compatibility.