Native inference, from app to library
MiRMiR 0.6.2 is a native local model runtime with a terminal dashboard, an
optional browser dashboard, a command-line interface, and an OpenAI-compatible
HTTP server. libmir is the embeddable Rust inference library underneath it.
Use these docs to operate MiRMiR or build your own native application on the same model discovery, loading, generation, K/V cache, Metal, and CUDA layers.
See what changes in 0.6.2: faster Metal and more models on it. In 0.6.1 each model’s context is chosen before it loads, so several models fit side by side. 0.6.0 added remote access, Mirmir for Mac and images for Nemotron Omni on CUDA.
Choose a surface
| Surface | Use it when |
|---|---|
| TUI | You want an interactive terminal workflow with no browser. |
| Web dashboard | You want the same local runtime controls and telemetry in a browser. |
| CLI | You need scripts, machine-readable output, or repeatable benchmarks. |
| HTTP API | An existing application already understands OpenAI-style requests. |
libmir | A Rust application needs direct ownership of models and sessions. |
MiRMiR and
libmirare pre-1.0. These pages describe version 0.6.2. Support claims require complete checkpoint and backend validation; planned formats are not described as available.
The stack
your application
|
+-- HTTP / SSE --> mirmir server --+
+-- TUI / CLI --> private gRPC ----+--> libmir --> Metal / CUDA
+-- Rust API -----------------------+
MiRMiR owns the application, configuration, transport, dashboards, model
catalog, and telemetry. Interactive and scripted clients attach to the same
local runtime or start a temporary one when needed. libmir owns checkpoint
discovery, tokenization, generation semantics, sessions, scheduling, cache
policy, and backend adapters. Laya decisions use a separate typed API backed
by Metal, CUDA, or the native CPU module mircup.
Install MiRMiR
MiRMiR ships as one native binary. Install version 0.6.2 from crates.io:
cargo install mirmir --version 0.6.2 --locked
mirmir --version
The final command should print mirmir 0.6.2.
Platform requirements
MiRMiR currently targets accelerator-backed inference:
- Apple Silicon macOS uses the Metal backend and requires the native MLX 0.32.4 framework;
- Linux uses CUDA and requires a supported NVIDIA driver, CUDA Toolkit 13,
and
git; - no Python, PyTorch, or Transformers runtime is used in the request path.
On Linux, the build compiles native CUDA kernels. It uses the toolkit in
/usr/local/cuda, or the one named by CUDA_HOME (and NVCC for a specific
compiler); nvcc does not need to be on PATH. The first build fetches the
pinned CUTLASS and FlashAttention sources into ~/.cache/mircuda; set
MIRCUDA_CUTLASS_DIR and MIRCUDA_FLASH_ATTN_DIR to reuse existing checkouts
or build offline. Kernels target sm_120f (Blackwell) by default; set
MIRCUDA_CUDA_ARCH for another architecture.
For a source build, clone the application repository and build it using its pinned Rust toolchain:
git clone https://github.com/mirmir-tech/mirmir.git
cd mirmir
cargo build --release --locked
Initialize configuration
mirmir config init
mirmir config validate
The persistent files are stored under ~/.config/mirmir. Runtime state and the
fallback Unix socket live under ~/.local/state/mirmir.
Open the terminal dashboard
Run MiRMiR without a subcommand:
mirmir
The process attaches to an existing local runtime when one is available. If it starts a temporary runtime itself, it also shuts that runtime down when the TUI exits.
Continue with Run your first model.
Upgrading an existing installation? Read Upgrade to 0.6.2, Upgrade to 0.6.1, Upgrade from 0.3, or Upgrade from 0.2 for older installations.
Run your first model
MiRMiR can search Hugging Face, download a snapshot into its managed cache, inspect compatibility, and load it into accelerator memory.
Search and download
mirmir model search "Qwen 2.5" --limit 10
mirmir model pull Qwen/Qwen2.5-0.5B-Instruct
Search results are ranked by the execution contract discovered by libmir and
by a conservative estimate of whether the checkpoint fits this machine.
MiRMiR evaluates architecture and physical encoding separately, including
dense, integer, block-floating-point, and codebook-quantized SafeTensors.
For gated repositories, configure a Hugging Face token without placing the secret directly in shell history:
printf '%s' "$HF_TOKEN" | mirmir config set hugging_face.token
mirmir config test hugging_face.token
Inspect and load
mirmir model inspect Qwen/Qwen2.5-0.5B-Instruct
mirmir model load Qwen/Qwen2.5-0.5B-Instruct
Inspection identifies the task, tokenizer, tensor layout, memory requirement,
configured K/V cache, and safe context capacity before loading weights.
--force bypasses only the conservative memory-fit rejection; it cannot make
an unsupported execution contract valid.
See Model compatibility for the 0.6.2 format and backend matrix.
Generate
Open the TUI and select Chat, or use the non-interactive path:
mirmir prompt \
--model Qwen/Qwen2.5-0.5B-Instruct \
--prompt "Explain paged K/V cache in three sentences."
For a reproducible measurement:
mirmir prompt \
--model Qwen/Qwen2.5-0.5B-Instruct \
--prompt "Explain paged K/V cache." \
--seed 7 --temperature 0 --warmup 1 --samples 5 --json
Unload or remove
mirmir model unload Qwen/Qwen2.5-0.5B-Instruct
mirmir model remove Qwen/Qwen2.5-0.5B-Instruct
Removing is destructive and requires the model to be unloaded first.
Mirmir for Mac
Mirmir for Mac is a native SwiftUI app for macOS 26.2 or later on Apple silicon. It lives in the menu bar and has a window for throughput and memory charts, the model library, chat, sessions, logs and the servers you use. It includes the MiRMiR runtime (mirmir 0.6.2 and MLX), so nothing else needs to be installed.
Install
- Download Mirmir for Mac, open the disk image and drag Mirmir to Applications. The app is signed and notarized.
- Start Mirmir. It starts the mirmir it includes as its own server.
Its own server
Mirmir runs sandboxed, like an App Store app. Its server lives in the app’s
container, ~/Library/Containers/tech.mirmir.app/Data, with its own
configuration, models and logs, and is separate from the
mirmir CLI:
- the app does not attach to a server the CLI or TUI started, and the CLI does not see the app’s server;
- models the app downloads stay in its container; the CLI’s models are not visible until you grant their folder (see below);
- the app’s OpenAI-compatible API listens on
http://127.0.0.1:18080/v1, so it does not collide with a CLI server on port 8080. Settings → Server shows the address for Zed, Raycast orcurl.
Use the app or the CLI on a Mac; running both means two servers, two model sets and twice the memory for a loaded model. Quitting Mirmir stops its server.
Model folders
The sandbox lets Mirmir read only the folders you choose. In Settings → Model folders, Add Folder… grants one:
- a Hugging Face cache, such as
~/.cache/huggingfaceor itshubfolder: the server lists its models next to its own (throughHF_HUB_CACHE) and restarts once to pick it up; - any other folder with checkpoints: load a model from it by path.
Models → Load from Folder… grants a checkpoint folder and loads it in one step. Grants persist across launches; Remove revokes one.
The app speaks the runtime protocol 2 of MiRMiR 0.6.2 and shows Update needed for an older remote server.
The menu bar and the window
Mirmir is a menu bar app: it has no Dock icon. Its icon shows the decode rate while a model generates; click it for the loaded model, live throughput, memory, active sessions and shortcuts to the window (Open Dashboard, Models…, Instances…, Download a Model…).
Closing the window keeps Mirmir running in the menu bar. To quit, choose Quit Mirmir (⌘Q) in the panel. Quitting stops the app’s server.
Servers
Mirmir shows one server at a time: This Mac, its own server, or a remote one. The Server card at the bottom of the sidebar names it, with its address, status, MiRMiR version and device; every screen’s subtitle starts with its name. Click the card to switch to another server or to manage the list. Switching changes the whole window and the menu bar, including the chat list: each conversation belongs to the server it was started on, because a model name means nothing on another server.
This Mac is always listed. Instances → Add Server adds a remote
server over TLS: enter host or host:port (the default port is 7443) and
an API key created on that server (a pasted grpcs://host:port works too). A
CLI server on the same Mac is reachable this way too, once its remote listener
is on. The app shows the server’s certificate fingerprint before it sends the
key; compare it with mirmir remote status on the server, trust it, then test
and add. Later certificate changes are refused. See
Remote access.
Edit Server… in Instances changes a server’s name, address, API key or trusted certificate and keeps its chats; a changed connection must pass a test before it is saved. Remove Server asks first and keeps the server’s chats: they come back when a server at the same address is added again.
Create the key on the server and name it after the computer that will use it:
mirmir remote key create --name my-mac --scope control
--scope use (the default) allows chat, model lists and monitoring, and a chat
may load a model the server offers; control also loads, unloads, downloads
and removes models and changes model and runtime settings. No key can change
the server’s own settings or stored secrets. Add --system on
a machine that runs the system server; remote servers keep using the CLI.
The first time you add a server on your network, macOS asks whether Mirmir may find devices on your local network: allow it. If you declined, turn Mirmir on in System Settings → Privacy & Security → Local Network; until then the app reports the server’s address as having no route.
API keys are stored in your login Keychain, never in the app’s files. macOS may ask once whether Mirmir may read its key; choose Always Allow.
Share this Mac
Open Instances → This Mac → Remote Access… to let other computers use the
app’s server: turn on the listener, copy
its certificate fingerprint, create keys with the use or control scope
(shown once), revoke them and see which clients connected recently.
Upgrade to 0.6.2
MiRMiR 0.6.2 makes Metal faster and broader, and Mirmir for Mac smaller. All native crates and Mirmir for Mac move to 0.6.2 together; configuration, protocol 2 and the HTTP API are unchanged.
cargo install mirmir --version 0.6.2 --locked
mirmir --version
Restart the runtime after upgrading.
What changes
- Faster Metal. Prefill keeps long contexts in full chunks, decode attends over a persistent K/V history shared by concurrent requests, and K/V admission is bounded by pending prefill, so deep, concurrent requests fit in memory. On an Apple M5 Pro, mirmir is ahead of MLX-LM on Qwen3.6-35B-A3B and Gemma 4 26B-A4B; see the benchmarks.
- More models on Metal. Dense linear-attention models, dense GELU Gemma models, Nemotron-H state-space hybrids and the pixel-shuffle vision encoder of Nemotron Omni run on Metal.
- Mirmir for Mac is about 76 MB instead of 320 MB and runs on macOS 26.2 or later. Its Metal kernels compile on first use: the first request after installing or updating the app, or after a macOS update, takes a moment longer.
Building from source
MLX 0.32.4 remains the native requirement. Workmir’s scripts/mlx/setup.sh
builds it with Metal JIT for macOS 26.2 and applies a fix for its JIT
d-split attention kernel; a prebuilt MLX 0.32.4 works as before.
Upgrade to 0.6.1
MiRMiR 0.6.1 lets you choose each model’s context before it loads, so several models fit in memory side by side. All native crates and Mirmir for Mac move to 0.6.1 together.
cargo install mirmir --version 0.6.1 --locked
mirmir --version
Restart the runtime after upgrading.
What changes
- Context per model. The TUI’s load dialog, the web dashboard and Mirmir
for Mac show a Context control before Load: automatic, or a token count
(the K/V cache shared by the model’s concurrent requests), with the memory
it needs. The choice is
saved for the model and used by every load. From the CLI:
mirmir model load <model> --context 32768. See Configuration. - Loaded models report real memory. Inspecting a loaded model no longer shows a 16-token cache; it estimates a reload with the model’s own memory counted as free.
- Bounded generation settings. Max tokens, temperature, top P, top K and repetition penalty are sliders with fixed ranges in every client; max tokens is limited by the chosen context.
- Saved context survives a new download. Pulling a newer snapshot of a model keeps its saved generation settings and context.
- Mirmir for Mac. A download in progress shows its progress instead of an error. Removing a model from disk and selecting a model on a remote server no longer crash the app.
gRPC clients
The protocol stays version 2. ModelContextChoice and ModelContext are new;
InspectModelRequest, InspectModelResponse, LoadModelRequest and
UpdateModelGenerationRequest gain an optional context. A 0.6.0 server
sends no context in its inspection; clients hide the control then.
Rust consumers
libmir::Library::preview_load resolves what a load would use without
changing the library, and libmir::LoadPreview holds the result.
Library::model_config_with_options no longer replaces the library’s cached
configuration.
Upgrade to 0.6.0
MiRMiR 0.6.0 adds remote access to the runtime, the Mirmir for Mac app, and
image input for Nemotron Omni on CUDA. mirtal, mircuda, mircup, libmir,
their support crates and mirmir move to 0.6.0 together.
cargo install mirmir --version 0.6.0 --locked
mirmir --version
mirmir config validate
Restart the runtime after upgrading: attaching a new client does not replace a running server.
What changes
- Remote access.
mirmir remote enableserves the runtime over TLS with scoped API keys; clients use--remoteor Mirmir for Mac. Nothing listens remotely until you enable it. See Remote access. - Mirmir for Mac. A native menu bar and window app; see Mirmir for Mac.
- Nemotron Omni images on CUDA. Checkpoints with the RADIO vision encoder
accept images through
/v1/chat/completions. The vision weights now count toward the model’s resident memory; on a shared GB10, lower the model’skv_blocksif smaller models no longer fit beside it. Metal reports image input for these checkpoints as unavailable. Images are sized by the checkpoint processor’s rule; for the few aspect ratios where that rule would exceed the patch budget (very long, thin images), MiRMiR fits the grid to the budget instead, so those images use fewer image tokens than the Hugging Face processor reports. - Telemetry memory. Host RAM and the accelerator budget are reported separately with a typed memory pool (runtime protocol 2). The HTTP dashboard JSON keeps its shape.
- Local model paths.
ListLocalModelsreports canonical paths, so models reached through a symlinked cache show the right state.
gRPC clients
The private runtime protocol is now version 2. A client built for protocol 1
is refused with a request to update. The removed telemetry fields are listed in
mirmir/docs/protocol.md.
Configuration
New optional keys: server.remote_bind, server.remote_tls_cert and
server.remote_tls_key. Existing configuration files stay valid.
Rust consumers
Update the libmir requirement to 0.6.0. Source changes from 0.5.5:
| Public API | Required adaptation |
|---|---|
libmir_models::layout::VisionPipeline | New variant PixelShuffleEncoder; add an arm or a wildcard to exhaustive matches. |
libmir_models::layout::VisionConfig | New variant PixelShuffleEncoder(PixelShuffleVisionConfig). |
libmir_models::layout::ImageProcessorConfig | New variant PixelShuffle(PixelShuffleImageProcessorConfig). |
libmir::PreparedVisionPrompt | New variant PixelShuffle; add an arm or a wildcard to exhaustive matches. |
libmir_metal::engine::hybrid_moe | Moved to libmir_metal::engine::hybrid::moe. |
libmir_metal::engine::hybrid_linear_moe | Moved to libmir_metal::engine::hybrid::linear_moe. |
Upgrade to 0.5.5
The 0.5.5 release aligns mirtal, mircuda, mircup, libmir, their support
crates, and mirmir at one version. The website and bundled dashboard carry
the same version. Existing 0.4.x and 0.5.2 installations should review these
changes before replacing their binary.
cargo install mirmir --version 0.5.5 --locked
mirmir --version
mirmir config validate
Stop and restart the runtime after upgrading; attaching a new client does not replace an already running server. Keep your configuration and model files. For service installations, retain the service account and socket permissions.
Available capabilities
- Nemotron-H on CUDA: text generation for the validated Nemotron-3.5-Super NVFP4 checkpoint, with Mamba-2, attention without RoPE, latent MoE, and optional multi-token prediction (MTP). This does not add Nemotron-H execution to Metal or enable the checkpoint’s vision tower.
- Gemma 4 on CUDA: sliding-window attention and 512-wide global heads use accelerated prefill paths. This is separate from the existing Metal Gemma path; benchmark results remain tied to their recorded builds.
- Structured content:
response_formataccepts JSON objects and strict JSON schemas on Metal and CUDA. Reasoning is separate from JSON content; literal protocol markers inside JSON strings remain ordinary text. - Laya decisions:
libmir::decisionruns on CPU, Metal, or CUDA. The manual-testing/v1/decideendpoint exposes CPU and macOS Metal only. - Memory and scheduling: per-model K/V block limits, opt-in MTP, and the existing explicit cached-refill and short-prompt admission policies.
- Metal foundation: the native bridge targets MLX 0.32.3. Build against
the matching native installation; the Python
mlxpackage is not required.
Request compatibility
JSON content cannot be combined with schema-constrained tools or a required
or named tool_choice. Such requests return 400 before generation. Use
JSON content for a final structured answer, or a named schema-constrained tool
for a tool invocation. strict: false is not supported.
thinking_token_budget now applies to both schema tools and JSON content,
with enabled reasoning and an explicit total completion limit. It reserves
space inside that limit; it does not increase the limit. Incomplete or invalid
structured output remains an error, including an SSE error after streaming
has started.
Rust consumers
Update your libmir requirement to 0.5.5 and select metal, cuda, or
cpu as needed. The default feature set enables no backend. CPU currently
provides Laya decisions, not the general generation/embedding runtime.
Use GenerationRequest { ..Default::default() } to retain defaults for new
request fields. Cross-repository dependencies remain crates.io versions.
This release changes Rust source compatibility with 0.5.2:
| Public API | Required adaptation |
|---|---|
Error::InvalidResponseFormat | Update exhaustive error matches for invalid or incomplete structured content. |
ArchitectureCapability | Handle StateSpaceMixer, LatentRoutedExperts, ReluSquared, and SigmoidBiasedTopKRouter in exhaustive matches. |
GenerationRequest | New reasoning_effort, reasoning_history, and response_format fields; prefer ..Default::default() where appropriate. |
Runtime DecodeRequest / DecodeSequence | Initialize speculation (None for ordinary decoding). |
Runtime DecodeOutput | Initialize accepted and drafts (empty for ordinary decoding). |
DecoderConfig / Metal MixerLowering::Softmax | Initialize the new typed Q/K normalization field when constructing these structs directly. Prefer parsing checkpoint configuration and lowering the resolved spec. |
| New MTP configuration | Construct the bounded window with DraftTokens::try_from(count); values must be 1–8. |
These additions are not hidden by adding #[non_exhaustive] retroactively, which
would itself change construction and matching rules. Rebuild and test Rust
consumers when upgrading; binary configuration compatibility is separate.
See JSON output, Laya decisions, and configuration for complete examples.
Upgrade from 0.3
MiRMiR 0.4.0 keeps the 0.3 control surfaces, CLI, configuration keys, and
private protocol. It changes how the runtime is organized internally, makes the
OpenAI-compatible API stricter and more explicit, adds tool calling and
per-request reasoning control, and narrows the libmir facade. Version 0.3.1
was never published; upgrade directly from 0.3.0.
HTTP clients
Review clients that relied on silently ignored fields or on the 0.3 status codes:
- A non-zero
presence_penaltyorfrequency_penalty, and any non-nullreasoning_budgetorreasoning_effort, now return400. Omit them, or userepetition_penaltyand the MiRMiRreasoningextension instead. tool_choicemust beauto,none,required, or a named function.- A prompt that exceeds the model context, or can never fit the K/V cache, is
now
400instead of500. A request that could fit later remains429. - A cancelled operation returns
408 cancelled. - A tool call that violates
toolsortool_choice, or is malformed or truncated, returns422 invalid_model_output. - Inference endpoints return
503until startup model restoration completes. Point readiness probes at the newGET /ready; keepGET /healthfor liveness. - Closing an SSE connection now cancels the generation on the server.
New request capabilities are opt-in: tools with validated tool_calls
responses, tool_constraints: "schema" on Metal and CUDA, reasoning and
chat_template_kwargs.enable_thinking, thinking_token_budget,
return_token_ids for streams, and top_n for reranking. See
Chat completions and
Embeddings and reranking.
Configuration
Existing config.toml files remain valid; no key was renamed. New optional
runtime keys tune scheduling: prefill_batch_wait_us,
cached_prefill_policy, prefill_decode_policy, prefill_refill_policy, and,
on macOS, metal_decode_reservation. Omitting them keeps the backend defaults.
See Configuration.
Models
Decoder rerankers that ship a sentence-transformers LogitScore module are now
admitted as causal rerankers on Metal and CUDA. Qwen XML tool calls are parsed
natively. Re-run mirmir model inspect after upgrading, because admission
reports can change between releases.
libmir embedders
- Depend on
libmir = "0.4.0". The facade is now the only public entry point: thefoundation,models, andruntimemodules and the backend-specific CUDA and Metal types are no longer re-exported. ChatCompletionRequestand theChat*message types are replaced byGenerationRequest,Conversation,Message,Tool,ToolCall, and theToolChoiceenum.Model::generate*takes&GenerationRequest, andModel::preparetakes&Conversation.ProgressEventis an enum withstage(),unit(),count(), anddetail()accessors.SchedulerConfigandGenerationTokengained fields; construct them with..Default::default()or update struct literals.
See Generate and stream for the new request shape.
Upgrade checklist
- Install 0.4.1 with
cargo install mirmir --version 0.4.1 --lockedand runmirmir --version. - Run
mirmir config validateagainst the existing configuration. - Remove unsupported penalty and reasoning-effort fields from API clients and
handle
408,422, and503responses. - Switch readiness checks to
GET /ready. - Inspect each checkpoint on its target Metal or CUDA machine.
Upgrade from 0.2
MiRMiR 0.3.0 changes the application from a collection of separate command paths into several control surfaces for one local runtime. The TUI, CLI, web dashboard, and HTTP API now observe and operate the same loaded models, configuration, request activity, and telemetry.
Choose who owns the runtime
- Run
mirmir servefor a persistent runtime shared by local clients and, when enabled, the HTTP API and web dashboard. - Run
mirmirto open the TUI. It attaches to an existing runtime or starts a temporary one when none is available. - Model and prompt commands follow the same rule: attach when possible, otherwise start a temporary runtime for the operation.
A client only shuts down a temporary runtime that it started. Exiting a TUI or
CLI command attached to mirmir serve leaves the persistent runtime running.
Manage models as a lifecycle
Model discovery, storage, compatibility, and accelerator residency are separate operations in 0.3.0:
mirmir model search "Qwen 3" --limit 20
mirmir model pull Qwen/Qwen3-4B
mirmir model inspect Qwen/Qwen3-4B
mirmir model load Qwen/Qwen3-4B
mirmir model unload Qwen/Qwen3-4B
Search and pull perform bounded remote inspection. Load performs structural,
backend, and memory preflight before allocating accelerator weights. --force
can override only the conservative memory-fit decision; it cannot admit an
unsupported checkpoint.
Move settings into the shared configuration
Use mirmir config instead of maintaining per-command runtime flags:
mirmir config init
mirmir config show
mirmir config validate
mirmir config set default_model Qwen/Qwen3-4B
Configuration records effective values and their source. Secrets live in a separate protected file and remain redacted in every control surface. When a persistent server is running, configuration mutations go through it so there is one writer.
Review model compatibility
Version 0.3.0 admits a checkpoint from its task, architecture, tokenizer, tensor schema, weight encoding, and selected backend. Family names and Hub tags are not support guarantees. The release adds typed SafeTensors paths for dense, integer, block-floating-point, and codebook quantization, including bitsandbytes NF4 and FP4 checkpoints without a Python runtime.
See Model compatibility for the complete 0.3.0 matrix.
Check the public API boundary
The OpenAI-compatible surface remains deliberately smaller than the complete OpenAI platform. Existing clients should target the documented chat, embedding, reranking, model-list, and health endpoints. Management operations belong to the TUI, CLI, web dashboard, or private local control protocol.
Upgrade checklist
- Install the 0.3.0 binary and run
mirmir --version. - Run
mirmir config validateagainst the existing configuration. - Inspect each checkpoint on its target Metal or CUDA machine.
- Start
mirmir serveif the runtime must outlive an interactive client. - Confirm API authentication, CORS, and dashboard settings before binding beyond a local development workflow.
Terminal dashboard
mirmir opens the terminal dashboard. All views operate on the same local
runtime used by CLI and HTTP clients.
Views
| Key | View | Purpose |
|---|---|---|
F1 | Dashboard | Live throughput, latency, memory, K/V cache, models, and activity. |
F2 | Models | Search, download, inspect, load, unload, and remove checkpoints. |
F3 | Chat | Stream multi-turn generation, reasoning, metrics, and image input. |
F4 | Settings | Inspect and edit effective configuration and secrets. |
Use Tab and Shift+Tab, a function key, or the mouse to switch views. Press
? for context-sensitive help.
The Dashboard aggregates current and mean request rates, memory and K/V-cache history, recent operations, and the active runtime stage without forcing a host synchronization in the generation path.
Models
/starts a Hugging Face search;d,l, andudownload, load, and unload;iincludes incompatible or over-budget remote results;ropens a destructive removal confirmation;Xcancels the selected cancellable operation;fexplicitly allows an over-budget load from the load dialog.
Download progress is streamed from the server. Additional downloads wait in a FIFO queue and can be cancelled before or during transfer.
Download progress
The transfer panel names the repository and current file, and reports both
percentage and transferred bytes. Press X with the active transfer selected
to request cancellation; queued downloads can be cancelled the same way.
Memory preflight
Before loading, MiRMiR combines weight, K/V-cache, and workspace estimates and
compares the result with the accelerator budget. The dialog also reports a safe
context limit. An over-budget result is blocked unless you deliberately press
f; forcing a load does not bypass compatibility checks.
Chat
The input remains locked while a request is active. Streamed reasoning is kept separate from final content, and the view reports TTFT, prefill, decode, and end-to-end throughput.
Ctrl+Left/Ctrl+Rightselects a loaded model;Ctrl+Pedits one-off generation parameters;Ctrl+Kclears the conversation;- drop one PNG, JPEG, WebP, or GIF to attach it;
Ctrl+Dremoves the attachment;Xrequests cooperative cancellation.
During streaming, the active assistant message keeps reasoning collapsible and the composer reports TTFT, prefill rate, decode rate, and token counts. Input is locked until the request finishes or cooperative cancellation completes.
Image attachment is available only when the selected model advertises image
input. The attachment name appears above the prompt; Ctrl+D removes it before
submission.
Activity and cancellation
The Dashboard records long-running operations and their stage. Select a
cancellable operation and press X. MiRMiR marks it as cancelling immediately,
but the operation stops cooperatively at a safe boundary rather than in the
middle of an accelerator kernel.
Settings
The settings view presents the effective value, its source, and whether a
server restart is required. Press Enter to edit the selected setting. The
Hugging Face token can be tested with t, and stored secrets remain redacted.
Exit behavior
Esc opens confirmation. q exits outside the chat input, and Ctrl+C exits
immediately. Only a TUI that owns its temporary runtime shuts it down.
Web dashboard
The browser dashboard is disabled by default. Enable it and restart the server:
mirmir config set server.web_enabled true
mirmir serve
Open http://127.0.0.1:8080/ui/.
What it exposes
- Dashboard shows live and historical throughput, prefill, decode, TTFT, memory, K/V occupancy, runtime counters, and recent activity;
- Models lists local snapshots, searches Hugging Face, and performs download, inspect, load, unload, and removal actions;
- Chat streams content and reasoning, accepts one image, exposes generation parameters, reports request metrics, and keeps recent conversations;
- Settings shows effective values, their source, restart requirements, paths, raw TOML, and redacted secrets.
One WebSocket carries telemetry, model snapshots, configuration, startup readiness, and activity. The browser does not poll each view independently.
The header of every view shows whether the runtime is generating, the active model, and the accelerator.
Models
Search Hugging Face by owner or model name and manage snapshots already present on the machine. Each row shows the weight encoding, the backends that can run it, and its capabilities (tools, thinking, vision). Model actions use the same runtime queue and memory checks as the terminal and CLI clients.
Chat
Select a loaded model, optionally attach one supported image, adjust one-off generation parameters, and stream reasoning and final content independently. The latest reply reports TTFT, prefill, decode, and end-to-end throughput.
Recent conversations appear beside the chat. Start another with New chat, reopen one from the list, or delete it. The history stays in this browser’s local storage, not on the server, and keeps the 30 most recent conversations.
Settings
Settings are grouped by section. Each row shows the effective value, its source,
and whether a change applies live or after a restart. After you change a
restart-bound setting, a banner lists it until you restart mirmir serve or
dismiss it. Secrets are redacted and use dedicated mutation paths.
Local security boundary
The dashboard can only be enabled on a loopback HTTP bind. Opening it creates an
eight-hour local session in an HttpOnly, SameSite=Strict cookie. Mutations
also require a CSRF token held in page memory. The OpenAI bearer token is not
accepted as dashboard administration authority.
For remote programmatic access, keep the dashboard local and configure the HTTP API security boundary separately.
Appearance and runtime details
The theme follows your system until you choose another appearance with the sun/moon button in the sidebar. Your choice is remembered. The Dashboard view includes prefill and decode charts, device telemetry, and server platform, backend support, device and host-memory details. Use Open chat on a resident model to start chatting with it. Screenshots use demonstration data.
Mac app
MiRMiR for Mac shows the same runtime as the terminal and web dashboards, natively and from the menu bar: live throughput and memory, the model library, chat, sessions, logs and every server you use. To install and connect it, see Mirmir for Mac.
Menu bar
The menu bar icon shows the decode rate while a model generates. Its panel has the loaded model, decode rate with a sparkline of the last minute, prefill, active sessions, prefix cache hits and memory, and shortcuts to the window. Switch loads another local model.
Window
While the window is open, Mirmir is also in the Dock and the app switcher, so it is easy to get back to; closing it leaves the menu bar icon alone. The sidebar lists the screens; the Server card at its bottom names the server they show and switches to another one.
- Overview has decode, prefill, time to first token and memory, a throughput chart over 5 minutes, 1 hour or 24 hours, memory against its limit, and the latest sessions. Rates appear only while the server handles a request.
- Chat streams replies with the model’s reasoning folded above them and shows tokens, decode rate and time to first token under each reply. Each conversation belongs to the server it was started on. With a vision model, the paperclip attaches a PNG, JPEG, GIF, WebP, HEIC or TIFF image (HEIC and TIFF are converted to JPEG). The image stays with the conversation, so follow-up questions and a reopened chat still see it; attaching another replaces it, and the × on the composer’s image removes it before sending. A message with an image is not sent to a model without image input: the app says so and keeps the message.
- Models lists local checkpoints and running downloads with their
features (
tools,think,vision), size and state; All, Ready and Downloading filter the list. The search field filters it at once and, in All and Hugging Face, searches the Hub when you pause typing: results join the list with the memory each needs on this server, and Download returns to your models with the download’s progress. The inspector loads the model and chooses its Context before Load (Automatic, or a token count, with the memory it needs; give each model a context to keep several loaded at once) and edits generation defaults. The context is saved for the model and used by every load, also from the menu bar. - Instances shows each server’s status, loaded models, decode rate and memory, adds, edits and removes remote servers and opens Remote Access for this Mac.
- Sessions lists generations, loads and downloads as they run; Logs follows the app’s server.
- Settings grants model folders and shows the app’s server: its runtime, its OpenAI-compatible API address and its socket.
How it connects
The app bundles the MiRMiR runtime: the same mirmir server as the CLI, with
libmir and MLX, runs as a separate process inside it. The app’s interface is a
client of that runtime’s gRPC service, like the TUI.
The app is sandboxed, and its server inherits the sandbox. On this Mac the app
starts its bundled mirmir with the app’s container as its home, so its
configuration, models, socket and logs stay there, apart from the CLI’s, and
it stops that server when you quit. Its OpenAI-compatible API listens on
127.0.0.1:18080. The server reads only folders you grant in Settings →
Model folders; see Mirmir for Mac.
Remote servers are reached over TLS with a scoped API key from your Keychain and a pinned certificate; see Remote access.
Configuration
Initialize, inspect, and validate configuration with:
mirmir config init
mirmir config show
mirmir config validate
The main file is ~/.config/mirmir/config.toml. Secrets are stored separately
with protected permissions and are never returned in clear text by the TUI,
web dashboard, CLI, or private gRPC API.
Runtime settings
| Key | Default | Meaning |
|---|---|---|
runtime.kv_block_size | automatic | Tokens stored in one K/V block. |
runtime.kv_blocks | automatic | Total block capacity. |
runtime.speculative_draft_tokens | disabled | Draft count from 1 to 8 for a compatible CUDA MTP checkpoint. auto removes the override. |
runtime.kv_cache_dtype | auto | Backend-resolved K/V storage type. |
runtime.max_batch_requests | automatic | Concurrent decode batch limit. |
runtime.max_batch_tokens | automatic | Scheduler token budget. |
runtime.prefill_batch_wait_us | 200 | Short window used to form a prefill batch. |
runtime.decode_batch_wait_us | automatic | Short window used to form a decode batch. |
runtime.decode_priority_burst | automatic | Decode fairness before other runnable work. |
runtime.cached_prefill_policy | backend_default | interleave_one_block admits one cached continuation during resident decode, lowering refill first-token latency at some decode latency cost. |
runtime.prefill_decode_policy | backend_default | complete_cohort finishes an admitted prefill cohort before its first tokens, improving decode occupancy at some first-token latency cost. |
runtime.prefill_refill_policy | closed | short_prompt lets prompts of at most 128 tokens join a running prefill on CUDA mixed-attention models. |
runtime.metal_decode_reservation | one_page | macOS only. generation_budget reserves K/V pages for the request’s whole generation budget from reclaimable spare pages. |
runtime.memory_reserve_percent | automatic | Percentage of visible memory kept outside model admission. |
runtime.memory_reserve_bytes | automatic | Additional fixed memory reserve used by admission. |
runtime.vision_max_pixels | automatic | Hard cap on resized image area. |
runtime.vision_attention_budget_bytes | automatic | Vision attention workspace cap. |
runtime.vision_memory_percent | 80 | Memory available to automatic vision budgeting. |
Server settings
| Key | Default | Meaning |
|---|---|---|
server.http_bind | 127.0.0.1:8080 | HTTP listen address. |
server.allow_remote | false | Permits a non-loopback bind. |
server.web_enabled | false | Serves the embedded dashboard on /ui/. |
server.cors_origins | [] | Exact allowed browser origins. |
server.body_limit_bytes | 29360128 | Maximum request body. |
server.request_timeout_seconds | 300 | HTTP request timeout. |
server.max_concurrency | 16 | In-flight HTTP request limit. |
Most runtime and server settings require a restart. The UI identifies this per
row. Set any optional runtime key to auto to remove the override and return
to the backend default, for example:
mirmir config set runtime.cached_prefill_policy interleave_one_block
mirmir config set runtime.cached_prefill_policy auto
runtime.prefill_refill_policy = "short_prompt" requires the default
prefill_decode_policy; validation rejects it with complete_cohort. Other
platforms reject runtime.metal_decode_reservation.
Speculative decoding and model memory
For the supported Nemotron-H CUDA checkpoint with a prediction head:
mirmir config set runtime.speculative_draft_tokens 3
Restart the runtime after changing the setting. auto disables the override.
MTP uses extra weights and recurrent snapshots. It can improve single-stream
decode, but does not promise a gain for every prompt or concurrent batch.
It is not a generic speculative decoder for every model or a Metal feature.
Context per model
A model’s context is the K/V capacity it loads with, in tokens. By default it is automatic: the server gives a model as much cache as free memory allows, which leaves no room for a second model. To run several models side by side, give each one a context. The choice is saved per model and used by every load, including restoration after a restart:
mirmir model inspect Qwen/Qwen3-8B --context 32768 # memory at 32k tokens
mirmir model load Qwen/Qwen3-8B --context 32768 # save it and load
mirmir model load Qwen/Qwen3-8B --context automatic # back to automatic
The TUI’s load dialog, the web dashboard and Mirmir for Mac show the same
choice as a Context control before Load, with the memory it needs. A
context is rounded up to whole K/V blocks. The cache is shared by the requests
a model serves at once, so it can be larger than the model’s context length,
up to that length for each of them (runtime.max_batch_requests); one
request still uses at most the model’s context length.
The choice is stored in the model’s TOML file in ~/.config/mirmir/models/ as
a block count, which can also be edited by hand:
[runtime]
kv_blocks = 2048 # 2048 blocks × 16 tokens = 32,768 tokens
It takes effect when that model is loaded again, and it is separate from the
global runtime.kv_blocks in config.toml.
Set and remove secrets
printf '%s' "$HF_TOKEN" | mirmir config set hugging_face.token
printf '%s' "$MIRMIR_HTTP_API_KEY" | mirmir config set server.api_key
mirmir config remove server.api_key
Environment values take precedence when present. MiRMiR recognizes HF_TOKEN
and MIRMIR_HTTP_API_KEY for these two secrets.
CUDA compiler includes
On Linux, MIRMIR_CUDA_INCLUDE_PATH overrides the include directories passed
to NVRTC when MiRMiR compiles CUDA kernels at runtime. It is a PATH-style,
colon-separated list, useful when the CUDA headers are not under
/usr/local/cuda.
Remote access
A MiRMiR server can offer its runtime to other computers: Mirmir for Mac, the CLI and the TUI connect over TLS with an API key. The local Unix socket is unchanged and needs no key. This is separate from the OpenAI-compatible HTTP API, which has its own key.
Turn it on
On the server:
mirmir remote enable # listens on 0.0.0.0:7443
mirmir remote enable --bind 10.0.0.4:7443
For a system service, add --system and run as a member of the service’s
group. enable starts the listener at once and keeps it enabled across
restarts. Only mirmir serve (and the system service) opens the listener; a
server that the TUI or a command starts for itself never does. The listener
address is changed only with mirmir remote enable --bind and
mirmir remote disable, not with mirmir config set. When no key exists yet, it creates a control key and prints it
once, together with the certificate fingerprint. Open the port in the
firewall only for networks that should reach it.
mirmir remote status # listener, fingerprint, keys, recent clients
mirmir remote disable # stop listening; keys are kept
Keys and scopes
mirmir remote key create --name studio-mac --scope use
mirmir remote key create --name ops-laptop --scope control
mirmir remote key list
mirmir remote key revoke 3f9c0d6e1a2b4c5d
| Scope | Allows |
|---|---|
use | Chat, embeddings, reranking, model lists and inspection, catalog search, telemetry, activity and the redacted configuration. A chat, embedding or rerank request may load a model the server offers, and unload another to make room |
control | Everything in use, plus explicitly loading, unloading, downloading and removing models, model and runtime settings, and cancelling operations |
No key, whatever its scope, can change server.* settings (listener, TLS
certificate, HTTP binding, CORS) or the stored HTTP API key and Hugging Face
token; those change only on the server itself. A remote key names only models
the server offers (configured keys, loaded models and the paths it lists
itself); any other name or path is refused as an unknown model. The
configuration a remote key reads omits the raw TOML and file paths.
Keys and the listener are managed only on the server itself, over its local
socket: no remote key can create or revoke keys. A key reads
mmk_<id>_<secret>; the server stores only a digest of the secret. A revoked
key is refused on its next request; a stream it already opened, such as a
running generation or an activity watch, continues until it ends.
Certificates
By default the server generates a self-signed certificate once and keeps it.
Clients trust it by its SHA-256 fingerprint; compare the value they show with
mirmir remote status on the server.
- Mirmir for Mac shows the fingerprint when you add the server, pins it once you trust it and refuses a later change.
- The CLI and TUI pin nothing on their own: pass the fingerprint with
--remote-fingerprintorMIRMIR_REMOTE_FINGERPRINTon every run (read it withmirmir remote fingerprint HOST[:PORT]). Without it they verify the certificate against the system’s CA roots, which refuses the default self-signed certificate.
To use a certificate issued by a CA, which clients verify without pinning:
mirmir config set server.remote_tls_cert /etc/mirmir/remote/fullchain.pem
mirmir config set server.remote_tls_key /etc/mirmir/remote/privkey.pem
mirmir remote enable
Each command saves one key; the listener needs both. While only one is set,
mirmir config show reports the other as missing and mirmir remote enable
names it; a server that starts with half a pair keeps the listener off and
shows why in mirmir remote status. mirmir remote enable restarts a running
listener with the new certificate. To return to the self-signed certificate:
mirmir config set server.remote_tls_cert auto
mirmir config set server.remote_tls_key auto
mirmir remote enable
Connect
From Mirmir for Mac: Instances → Add Server, enter
host:7443 (a pasted grpcs://host:7443 works too) and the key, check the
fingerprint, then Test Connection and Add Server. The key is stored in
the Keychain. Edit Server… changes the address, key or trusted certificate
and keeps the server’s chats; a removed server’s chats come back when a server
at the same address is added again.
From the CLI or TUI on another computer:
mirmir remote fingerprint gpu-server.local # what the server presents
export MIRMIR_REMOTE_KEY=mmk_…
mirmir --remote gpu-server.local --remote-fingerprint 'AB:CD:…' status
mirmir --remote gpu-server.local --remote-fingerprint 'AB:CD:…' model list
mirmir --remote gpu-server.local --remote-fingerprint 'AB:CD:…' # TUI
--remote also accepts the grpcs://host:port form that mirmir remote status shows.
The port defaults to 7443. Like --system, a remote client only attaches: it
never starts or configures a local runtime.
What is enforced
- TLS 1.2 or 1.3; a pinned client accepts only the server holding that certificate’s private key.
- No key or an unknown key:
UNAUTHENTICATED. A key whose scope is too narrow, or a local-only method:PERMISSION_DENIED. - More than 20 failed attempts within a minute from one IPv4 address or IPv6 /64 network refuse further failed attempts from it for the rest of the minute; a valid key from that network keeps working. A connection without a peer address is refused.
- Rejections are logged with the address and method, never the key.
mirmir remote statuslists clients seen in the last 15 minutes with the key name, scope, address and request count.
CLI reference
Running mirmir without a subcommand opens the TUI. Commands expose the same
runtime for servers, scripts, model management, configuration, and benchmarks.
Every interactive or one-shot client first tries the local private protocol.
If no runtime exists, it starts a temporary one and shuts down only that owned
instance when the command finishes. Use mirmir serve when models and runtime
state must persist across clients.
| Command | Purpose |
|---|---|
mirmir serve | Persistent private gRPC runtime and optional HTTP/Web server. |
mirmir status | Health, loaded models, and latest telemetry as JSON. |
mirmir prompt | Stream or benchmark one prompt. |
mirmir model … | Search, inspect, download, load, unload, and remove models. |
mirmir config … | Initialize, show, edit, validate, set, remove, and test config. |
Serve
mirmir serve
mirmir serve --no-http
mirmir serve --http-bind 127.0.0.1:9090
MIRMIR_HTTP_BIND supplies the same per-process override as --http-bind.
Prompt and benchmarks
Input comes from exactly one of --prompt, --prompt-file, or --stdin.
Generation parameters include --max-tokens, --temperature, --top-p,
--top-k, --repetition-penalty, and --seed.
printf 'Explain continuous batching.' | mirmir prompt \
--stdin --model qwen --warmup 1 --samples 5 --json --csv result.csv
--no-stream buffers the final completion. JSON benchmark output contains raw
samples and aggregate E2E, TTFT, prefill, and decode distributions.
Model lifecycle
mirmir model list
mirmir model inspect MODEL
mirmir model search QUERY --limit 20
mirmir model pull OWNER/REPO --revision REVISION
mirmir model load MODEL [--force]
mirmir model unload MODEL
mirmir model remove OWNER/REPO
Selectors may be a configured alias, repository ID, local model key, or checkpoint path where the command permits it.
Configuration
mirmir config init
mirmir config show
mirmir config edit
mirmir config validate
mirmir config set KEY [VALUE]
mirmir config remove hugging_face.token
mirmir config test hugging_face.token
When a server is running, commands that change configuration use it as the sole writer.
Remote access
mirmir remote enable [--bind 0.0.0.0:7443]
mirmir remote status
mirmir remote disable
mirmir remote key create --name NAME [--scope use|control] # default: use
mirmir remote key list
mirmir remote key revoke ID
mirmir remote fingerprint HOST[:PORT]
Global options --remote HOST[:PORT], --remote-key (or MIRMIR_REMOTE_KEY)
and --remote-fingerprint (or MIRMIR_REMOTE_FINGERPRINT) attach any command
or the TUI to another computer’s server. See
Remote access.
Runtime ownership
| Invocation | Runtime behavior |
|---|---|
mirmir serve | Owns a persistent runtime until interrupted. |
mirmir | Attaches to the persistent runtime or owns a temporary TUI runtime. |
mirmir model … | Uses the shared runtime or owns one for the model operation. |
mirmir prompt … | Uses the shared runtime or owns one for generation and benchmarks. |
mirmir status | Reads an already running server and does not start one. |
OpenAI-compatible API
mirmir serve listens on http://127.0.0.1:8080 by default. The 0.6.2 public
HTTP contract contains:
| Method | Path | Purpose |
|---|---|---|
GET | /health | Liveness, server and private protocol versions. |
GET | /ready | Same body as /health, or 503 until startup model restoration completes. |
GET | /v1/models | Models ready in the shared runtime. |
POST | /v1/chat/completions | Text or single-image chat, JSON or SSE. |
POST | /v1/embeddings | Single or batched text embeddings. |
POST | /v1/rerank | TEI-style document relevance scoring. |
POST | /v1/decide | Manual Laya decisions on CPU or macOS Metal; a MiRMiR extension. |
The dashboard management API under /api/mirmir/v1 is private to the embedded
web application. It is versioned separately and is not an OpenAI client API.
Quick start
mirmir serve
curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/v1/models
Bearer authentication is optional on loopback unless an API key has been
configured. When configured, send it on every /v1 request:
curl http://127.0.0.1:8080/v1/models \
-H "Authorization: Bearer $MIRMIR_HTTP_API_KEY"
Errors
Errors use an OpenAI-style envelope:
{
"error": {
"message": "model is not loaded",
"type": "invalid_request_error",
"param": null,
"code": "not_found"
}
}
| Status | type / code | Meaning |
|---|---|---|
400 | invalid_request_error / invalid_request | Invalid or unsupported request, including a prompt that exceeds the model context or can never fit the K/V cache. |
401 | authentication_error / invalid_api_key | Missing or invalid bearer token. |
404 | invalid_request_error / not_found | Unknown or unloaded model. |
408 | request_error / cancelled | The operation was cancelled. |
409 | invalid_request_error / conflict | Model lifecycle conflict, such as a model that is loading or in use. |
422 | model_output_error / invalid_model_output | The model produced output that violates the request contract, such as a missing, undeclared, malformed, or truncated tool call. |
429 | rate_limit_error / rate_limit_exceeded | The request fits the model but not the currently available memory or vision resources; retry later. |
503 | server_error / service_unavailable | The runtime is not ready, for example while restoring models at startup. |
500 | server_error / runtime_error | Unexpected runtime failure. |
/v1/chat/completions, /v1/embeddings, and /v1/rerank return 503 until
startup restoration finishes. Use GET /ready as a readiness probe and
GET /health as a liveness probe.
Chat completions
Use POST /v1/chat/completions with a model already loaded into the shared
runtime.
curl http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $MIRMIR_HTTP_API_KEY" \
-d '{
"model": "qwen",
"messages": [{"role": "user", "content": "Hello"}],
"temperature": 0.2,
"max_tokens": 256
}'
import os
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:8080/v1",
api_key=os.environ["MIRMIR_HTTP_API_KEY"],
)
reply = client.chat.completions.create(
model="qwen",
messages=[{"role": "user", "content": "Hello"}],
temperature=0.2,
max_tokens=256,
)
print(reply.choices[0].message.content)
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "http://127.0.0.1:8080/v1",
apiKey: process.env.MIRMIR_HTTP_API_KEY,
});
const reply = await client.chat.completions.create({
model: "qwen",
messages: [{ role: "user", content: "Hello" }],
temperature: 0.2,
max_tokens: 256,
});
console.log(reply.choices[0].message.content);
Parameters
| Field | Type | Notes |
|---|---|---|
model | string | Required loaded model selector. |
messages | array | Required non-empty chat history. Assistant tool_calls arguments may be a JSON object or a JSON-encoded string. |
stream | boolean | Returns server-sent events when true. |
max_tokens | integer | Total generated-token limit, including reasoning and final content. |
max_completion_tokens | integer | Alias; must agree with max_tokens if both appear. |
min_tokens | integer | Suppress end-of-sequence before this many tokens. |
ignore_eos | boolean | Continue past end-of-sequence until max_tokens. |
temperature | number | Sampling temperature. |
top_p | number | Nucleus cutoff from 0 to 1. |
top_k | integer | Candidate count; zero leaves it unbounded. |
repetition_penalty | number | 1 disables the penalty. |
seed | integer | Reproducible sampler seed. |
n | integer | Only 1 is supported. |
tools | array | OpenAI function tool declarations. |
tool_choice | string or object | auto (default), none, required, or {"type":"function","function":{"name":…}}. |
tool_constraints | string | MiRMiR extension: none (default) or schema. See Tool calling. |
response_format | object | text, json_object, or strict json_schema. See Structured JSON output. |
reasoning | string | MiRMiR extension: model_default (default), enabled, or disabled. |
chat_template_kwargs.enable_thinking | boolean | vLLM-compatible alias for reasoning; must agree with it when both appear. |
chat_template_kwargs.medium_effort | boolean | Uses shorter reasoning when true and the template declares this control. |
chat_template_kwargs.truncate_history_thinking | boolean | Keeps or drops earlier reasoning when the template declares this control. |
thinking_token_budget | integer | Reasoning allowance inside the total limit for schema tools or JSON content. |
return_token_ids | boolean | Streaming only: adds generated token ids to each chunk. |
stream_options.include_usage | boolean | Adds usage to the final SSE chunk. |
MiRMiR rejects controls it cannot honor instead of silently ignoring them. A
non-zero presence_penalty or frequency_penalty and any non-null
reasoning_budget or reasoning_effort return 400.
Unknown chat_template_kwargs keys are rejected. Explicit effort/history
controls are validated against the selected template; they are not universal
switches for every model.
Streaming
curl -N http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen",
"messages": [{"role": "user", "content": "Hello"}],
"stream": true,
"stream_options": {"include_usage": true}
}'
Each event contains a chat.completion.chunk. The stream ends with
data: [DONE]. Reasoning-capable models emit delta.reasoning_content
separately from delta.content. Tool calls arrive as one delta.tool_calls
chunk before the final chunk, whose finish_reason is tool_calls. With
return_token_ids, each chunk carries choices[0].token_ids, and the final
chunk carries any remaining ids.
Closing the connection cancels the generation on the server.
Reasoning
reasoning selects the thinking switch for one text request without changing
other requests:
{"model": "qwen", "messages": [{"role": "user", "content": "Return 2+2."}],
"reasoning": "disabled", "max_completion_tokens": 256}
Explicit enabled and disabled modes require a model chat template that
reads enable_thinking, or a supported built-in Qwen or Gemma template. Other
templates and image requests reject explicit modes. model_default preserves
the template’s own behavior. The max_tokens limit covers reasoning and final
content together; reaching it does not guarantee a final answer.
Tool calling
Declare OpenAI function tools and select how the model may use them:
{
"model": "qwen",
"messages": [{"role": "user", "content": "What's the weather in Warsaw?"}],
"tools": [{
"type": "function",
"function": {
"name": "get_weather",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"]
}
}
}],
"tool_choice": "auto"
}
Native Qwen XML (<tool_call><function=…>) and Mistral [TOOL_CALLS] output
is normalized into OpenAI message.tool_calls with finish_reason: "tool_calls". MiRMiR validates every call against the request and returns
422 invalid_model_output when the model omits a call that required or a
named function demands, calls a tool while tool_choice is none, calls a
different or undeclared tool, or emits malformed or truncated arguments.
Schema-constrained tools
On Metal and CUDA, "tool_constraints": "schema" constrains decoding to the
named tool’s parameter schema, so the model can only produce a well-formed
call. It requires
a named tool_choice, repetition_penalty: 1, min_tokens: 0, and
ignore_eos: false; other combinations return 400.
- Root schemas must be objects with declared properties, at most 64 top-level
properties and 512 KB. Remote and file
$refretrieval is disabled. - Types, required fields, enums, and array cardinality are enforced while
decoding;
uniqueItemsis checked on the completed call (422on violation). Parameter order follows the declaration. - With
reasoning: "enabled", reasoning stays unconstrained until the model’s</think>marker, then the tool grammar takes over.thinking_token_budgetcaps reasoning insidemax_tokensand must leave at least two tokens for the delimiter and tool. When omitted, MiRMiR reserves a quarter ofmax_tokens(at least two tokens) for the tool. - A limit that truncates the tool call still returns
422.
One image
The OpenAI-style image_url content part accepts a data URL. Version 0.6.2
supports one PNG, JPEG, WebP, or GIF image per request; an animated GIF contributes its first frame.
Image requests cannot set an explicit reasoning mode, effort or history; leave
chat_template_kwargs.enable_thinking unset.
IMAGE_DATA=$(base64 < image.jpg | tr -d '\n')
curl http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d "{\"model\":\"vision-model\",\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/jpeg;base64,$IMAGE_DATA\"}},{\"type\":\"text\",\"text\":\"Describe this image\"}]}]}"
The application limit is 20 MiB per decoded image. The server body limit must also accommodate the larger base64 request.
Structured JSON output
POST /v1/chat/completions accepts response_format on Metal and CUDA:
| Value | Result |
|---|---|
{"type":"text"} or omitted | Ordinary generation. |
{"type":"json_object"} | A JSON object in message.content. |
{"type":"json_schema","json_schema":{...}} | JSON matching the supplied schema. |
curl http://127.0.0.1:8080/v1/chat/completions \
-H "Authorization: Bearer $MIRMIR_HTTP_API_KEY" \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen",
"messages": [{"role":"user","content":"Classify: the parcel arrived damaged."}],
"reasoning": "disabled",
"repetition_penalty": 1,
"max_completion_tokens": 256,
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "classification",
"strict": true,
"schema": {
"type": "object",
"properties": {"category":{"enum":["damage","delay","other"]}},
"required": ["category"],
"additionalProperties": false
}
}
}
}'
Use a model whose template supports the requested reasoning switch, or omit
the switch to retain model defaults. A JSON schema needs a nonblank name;
omit strict or set it to true. strict: false returns 400.
Validation and limits
- Sampling requires
repetition_penalty: 1,min_tokens: 0, andignore_eos: false. Omitted controls use these neutral values, overriding checkpoint defaults. - JSON content and
tool_constraints: "schema"are exclusive. Required or named tool choices are also rejected.auto/noneremain accepted; this mode constrains the final content rather than producing tool calls. - Unsupported schemas are rejected before generation. Remote/file schema
reference retrieval is disabled. The complete output is validated against
the original schema; constraints such as
uniqueItemsare checked at that final stage, not guaranteed by the token mask. - Truncated, invalid, or incomplete JSON produces
422 invalid_model_outputfor a non-streaming request. This intentionally differs from OpenAI’sfinish_reason: "length"with truncated content: MiRMiR does not return an incomplete structured document as a successful completion. Streaming clients receive an error after any partial chunks and must discard that document. Schema validity does not verify factual truth.
JSON response schemas and explicit incompatible sampling settings are rejected before generation or SSE starts. When sampling controls are omitted in schema mode, repetition penalty is 1, minimum tokens is 0, and EOS is honored, even if the checkpoint has different defaults. Valid schemas continue to constrain token selection during decode; completed content is also checked against the original schema. Model-specific tokenizer/backend failures can still occur during setup.
Reasoning and streaming
With supported native reasoning delimiters, reasoning can precede the JSON
answer and is returned separately in reasoning_content. Once JSON begins,
strings containing text such as <think> remain literal content.
With reasoning: "enabled" and an explicit completion limit,
thinking_token_budget bounds reasoning within the total. It must leave at
least two tokens for the delimiter and answer. If omitted, the HTTP adapter
reserves one quarter of the total (at least two tokens) for structured output.
Reserve more for large schemas; this does not guarantee completion.
For stream: true, accumulate delta.content before parsing JSON. Individual
chunks need not be valid JSON. The final validation can fail after bytes have
already been emitted; handle the stream’s error event rather than treating a
partial response as a validated document.
Laya decisions
POST /v1/decide is a manual-testing extension for Laya decision checkpoints.
It evaluates one typed question per property of a flat JSON schema and returns
decided values plus option probabilities. It is separate from chat completions
and the managed model catalog.
curl http://127.0.0.1:8080/v1/decide \
-H "Authorization: Bearer $MIRMIR_HTTP_API_KEY" \
-H 'Content-Type: application/json' \
-d '{
"checkpoint": "/models/laya-multilingual",
"backend": "cpu",
"state": {"text": "We were billed twice for March."},
"schema": {
"type": "object",
"properties": {
"refund": {"type": "boolean"},
"urgency": {"type": "integer", "minimum": 1, "maximum": 3}
}
}
}'
backend is cpu by default, or metal on macOS. The Rust API also supports
CUDA; the HTTP endpoint does not expose it. state accepts either text or a
conversation array of turns. Text truncation keeps the beginning;
conversation truncation keeps the newest turns.
The response contains values and a fields array with each field’s name,
options, probabilities, confidence, concentration, and, for scales,
expected_level. Supported questions include booleans, finite enums, and
bounded integer scales of at most ten levels. Unsupported schemas (such as
free strings or multipleOf), unknown backends, and unreadable paths return
400.
The endpoint uses the server’s bearer authentication and accepts a local
checkpoint path. One checkpoint/backend stays resident at a time, on a
dedicated worker. Four requests may wait behind the running job; excess work
returns 503. Abandoned queued jobs are skipped. A running decision finishes
even when its HTTP handler disappears, and its unused result is dropped.
For embedded applications, see the decision API.
Embeddings and reranking
MiRMiR discovers the task from the checkpoint structure. A generation model cannot be used as an embedding model, and it serves reranking only when the checkpoint declares a causal scoring head (see below).
Embeddings
curl http://127.0.0.1:8080/v1/embeddings \
-H 'Content-Type: application/json' \
-d '{
"model": "embedding",
"input": ["first text", "second text"],
"encoding_format": "float"
}'
| Field | Type | Notes |
|---|---|---|
model | string | Loaded embedding model. |
input | string or string[] | One input or an ordered batch. |
dimensions | integer | Optional prefix dimension, capped by the native size. |
prompt_name | string | Optional prompt preset declared by the checkpoint. |
encoding_format | string | Only float is supported. |
The output follows the OpenAI list shape and returns normalized f32 vectors
plus prompt token usage.
Reranking
curl http://127.0.0.1:8080/v1/rerank \
-H 'Content-Type: application/json' \
-d '{
"model": "reranker",
"query": "native inference",
"documents": ["Rust and Metal", "a hosted Python service"],
"return_documents": true
}'
| Field | Type | Notes |
|---|---|---|
model | string | Loaded sequence-scoring or causal-scoring model. |
query | string | Required non-empty query. |
documents | string[] | Required non-empty candidate list. |
max_length | integer | Pair token limit, capped by model context. |
top_n | integer | Return only the best n results after sorting. |
raw_scores | boolean | Return logits instead of logistic relevance scores. |
return_documents | boolean | Include each original document in results. |
Results are sorted from highest to lowest relevance while retaining the original document index.
Causal rerankers
Since 0.4.0, a decoder checkpoint that ships a sentence-transformers
LogitScore module is admitted as a causal reranker on Metal and CUDA. The
module declares the “true” and “false” answer tokens and an optional
instruction, which is rendered as a system message. MiRMiR renders each query
and document through the model’s chat template and scores the pair as the
difference between the two answer-token logits at the final position. raw_scores: true returns that difference; otherwise
it passes through the logistic function like a cross-encoder score.
Security and remote access
Loopback is the default and safest mode:
[server]
http_bind = "127.0.0.1:8080"
allow_remote = false
web_enabled = false
cors_origins = []
Require an API key
printf '%s' "$MIRMIR_HTTP_API_KEY" | mirmir config set server.api_key
Once configured, clients send:
Authorization: Bearer YOUR_KEY
Bind beyond loopback
A non-loopback bind requires both explicit remote access and an API key:
[server]
http_bind = "0.0.0.0:8080"
allow_remote = true
web_enabled = false
The embedded dashboard deliberately cannot be exposed on this bind. Put TLS, network policy, and any public rate limiting in a trusted reverse proxy or private network boundary.
Browser origins
CORS is disabled until exact origins are listed:
[server]
cors_origins = ["https://app.example.com"]
MiRMiR accepts GET and POST with Authorization and Content-Type headers
from configured origins. Do not use * as a substitute for deciding which
applications may reach a model server.
Runtime access from other computers
The HTTP API key does not grant control of the runtime. To manage models and
chat from Mirmir for Mac or mirmir --remote, enable the separate TLS
listener with its own scoped keys; see Remote access.
Embed libmir
libmir is the model-neutral Rust inference library used by MiRMiR. It
discovers Hugging Face-style checkpoints, loads a native accelerator backend,
creates independent inference sessions, and exposes generation, tool calling,
embeddings, reranking, progress, cancellation, metrics, and memory information.
It deliberately does not contain:
- CLI argument or environment parsing;
- an HTTP, gRPC, or WebSocket server;
- a terminal or browser interface;
- Hugging Face search and download policy;
- application configuration files.
Those responsibilities belong to an embedding application such as MiRMiR.
Public lifecycle
ModelDescriptor::inspect(path)
|
v
Library::new(RuntimeConfig)
|
v
Library::load(path) -> Model
|
+-- Model::generate(&GenerationRequest, ...)
+-- Model::embed(...)
+-- Model::rerank(...)
+-- Model::session() -> prefill / decode
+-- Model::unload()
Applications should depend on the top-level libmir package and import from
its root. The internal workspace crates, including the CUDA and Metal adapters
and their configuration types, are implementation details and are not
re-exported; backend settings are adjusted through RuntimeConfig fields (see
Metal and CUDA).
Choose an entry point
| Need | API |
|---|---|
| Inspect without allocating accelerator weights | ModelDescriptor::inspect |
| Load a checkpoint | Library::load |
| High-level chat generation | Model::generate |
| Tool calls and schema-constrained arguments | Conversation::tools, GenerationRequest::tool_constraints |
| JSON content | GenerationRequest::response_format |
| Laya decisions | decision::DecisionModel on CPU, Metal, or CUDA |
| Cooperative cancellation | Model::generate_cancellable |
| Single-image generation | Model::generate_image_cancellable |
| Dense vectors | Model::embed |
| Document scoring (cross-encoder or causal) | Model::rerank |
| Manual token loop | Model::prepare, Model::session, Session::prefill, Session::decode |
| Capacity planning | ModelDescriptor::memory_estimate, Library::memory_snapshot |
Start with Add libmir to a project.
Add libmir to a project
Version 0.6.2 enables no accelerator by default. Select the backend explicitly.
Apple Metal
[dependencies]
libmir = { version = "0.6.2", default-features = false, features = ["metal"] }
The Metal feature currently requires Apple Silicon, a full Xcode installation with the separately installed Metal toolchain, and native MLX.
NVIDIA CUDA
[dependencies]
libmir = { version = "0.6.2", default-features = false, features = ["cuda"] }
CUDA currently targets Linux with the NVIDIA driver, CUDA Toolkit 13.x, NVRTC, CCCL/CUB headers, and CUTLASS 4.4.2 headers.
Minimal generation
use std::io::{self, Write};
use libmir::{
Conversation, GenerationOverrides, GenerationRequest, Library, Message,
RuntimeConfig,
};
fn main() -> libmir::Result<()> {
let model_path = std::env::args_os().nth(1).expect("model path");
let library = Library::new(RuntimeConfig::default());
let model = library.load(
model_path,
GenerationOverrides::default(),
&mut |progress| eprintln!("{}: {}", progress.stage(), progress.detail()),
)?;
let request = GenerationRequest {
conversation: Conversation {
messages: vec![Message {
role: "user".into(),
content: "Explain paged K/V cache briefly.".into(),
reasoning_content: None,
tool_calls: None,
tool_call_id: None,
}],
..Conversation::default()
},
options: GenerationOverrides {
max_tokens: Some(256),
temperature: Some(0.0),
top_p: Some(1.0),
top_k: Some(0),
repetition_penalty: Some(1.0),
..GenerationOverrides::default()
},
seed: Some(7),
..GenerationRequest::default()
};
let output = model.generate(&request, &mut |_| {}, &mut |token| {
print!("{}", token.text);
let _ = io::stdout().flush();
})?;
println!("\nfinish_reason={}", output.finish_reason);
Ok(())
}
GenerationRequest separates the conversation (messages, tools,
tool_choice) from request-scoped sampling overrides. Unset overrides keep the
generation settings resolved when the model was loaded.
Library initializes its native backend lazily. Loading inspects the
checkpoint first, resolves the execution contract, then reports real loading
progress through the callback.
The same lifecycle is available as the generate example in the libmir
repository.
Inspect and load models
Inspection is separate from accelerator allocation. Use it to reject an unsupported checkpoint or present its capabilities before committing device memory.
use libmir::{GenerationOverrides, ModelDescriptor};
let descriptor = ModelDescriptor::inspect(
"/models/qwen",
GenerationOverrides::default(),
)?;
let manifest = descriptor.manifest()?;
println!("id={}", manifest.id);
println!("context={}", manifest.context_len);
println!("task={:?}", descriptor.task());
println!("quantization={:?}", manifest.quantization);
println!("generation={:?}", descriptor.generation());
Ok::<(), libmir::Error>(())
Discovery is based on configuration values, tokenizer assets, tensor names and shapes, and the resulting execution contract. A recognizable repository or architecture name alone does not admit a checkpoint.
Estimate memory
use libmir::{GenerationOverrides, ModelDescriptor, RuntimeConfig};
let descriptor = ModelDescriptor::inspect(
"/models/qwen",
GenerationOverrides::default(),
)?;
let estimate = descriptor.memory_estimate(&RuntimeConfig::default());
println!("weights={} bytes", estimate.weight_bytes);
println!("kv={} bytes", estimate.kv_cache_bytes);
println!("workspace={} bytes", estimate.workspace_bytes);
println!("required={} bytes", estimate.required_bytes);
Ok::<(), libmir::Error>(())
This is a conservative planning value, not a reservation. Compare it with
Library::memory_snapshot() on the backend that will load the model.
Load and unload
use libmir::{GenerationOverrides, Library, RuntimeConfig};
let library = Library::new(RuntimeConfig::default());
let model = library.load(
"/models/qwen",
GenerationOverrides::default(),
&mut |event| {
let count = event.count();
eprintln!("{} {}/{} {}", event.stage(), count.current(), count.total(), event.unit());
},
)?;
println!("backend={:?}", model.engine().target());
model.unload()?;
Ok::<(), libmir::Error>(())
Model is cheaply cloneable. Model::unload succeeds only when the value is
the sole remaining owner; active clones or sessions produce Error::ModelInUse.
Dropping a Session releases its backend and K/V resources automatically.
Several resident models
Library::load admits a model only when its estimate fits the accelerator
budget, counting K/V memory that already loaded models could give back;
otherwise it returns Error::MemoryAdmission. Library::load_with_options with
ModelLoadOptions { allow_memory_overcommit: true } skips that check, although
backend allocation may still fail.
Library::reclaimable_kv_bytes reports that amount. After loading or unloading,
Library::rebalance_kv_caches resizes idle models’ measured K/V caches to the
memory the others leave and returns how many changed.
Generate and stream
Model::generate is the preferred high-level text API. It renders the
checkpoint chat template, tokenizes, creates an independent session, performs
prefill, samples and decodes tokens, normalizes model-specific channels, and
returns metrics.
Content and reasoning
The token callback receives GenerationToken values with a semantic channel.
Native think markers and channel headers are removed from the public contract.
Each token carries its id and, in preceding_ids, any generated identifiers
that were folded into it without producing visible text of their own.
The final GenerationOutput contains separate text, reasoning, and
tool_calls fields, every generated token_ids entry (including a terminal
stop token when emitted), prompt_tokens, a finish_reason such as "stop",
"tool_calls", or "max_tokens", and metrics.
let output = model.generate(
&request,
&mut |progress| eprintln!("{}: {}", progress.stage(), progress.detail()),
&mut |token| println!("{:?}: {}", token.channel, token.text),
)?;
println!("answer={}", output.text);
println!("reasoning={}", output.reasoning);
println!("metrics={:#?}", output.metrics);
Ok::<(), libmir::Error>(())
ProgressEvent is an enum with one variant per stage: LoadWeights,
InitializeRuntime, Warmup, PrefillTokens, and DecodeTokens. Use
stage(), unit() (Byte, Item, or Token), count() for the
current()/total() pair, and detail() rather than parsing display text.
Reasoning controls
GenerationRequest::reasoning selects the template’s thinking switch:
ReasoningMode::ModelDefault (the default) preserves the model’s rendering,
while Enabled and Disabled require a Jinja enable_thinking input or a
supported built-in Qwen or Gemma template. Unsupported templates and image
requests reject explicit modes.
GenerationOverrides::max_tokens counts every generated token, including
reasoning. GenerationRequest::reasoning_cycle decides what happens when a
reasoning loop is detected:
| Policy | Behavior |
|---|---|
ReasoningCyclePolicy::ExitReasoning { min_tokens: 64 } | Default. Closes a supported reasoning channel after three consecutive repeated token cycles past the token gate. |
ReasoningCyclePolicy::Recover | Resumes greedy decoding with a repetition penalty. |
Scattered repetitions, such as quoting the same source twice, do not force an exit. The completion limit still bounds reasoning that never reaches an answer.
Tool calls
Declare functions in Conversation::tools and select a ToolChoice: Auto
(the default), None, Required, or Function(name).
use libmir::{FunctionDefinition, Tool, ToolChoice};
request.conversation.tools = vec![Tool {
kind: "function".into(),
function: FunctionDefinition {
name: "weather".into(),
description: Some("Current weather for a city".into()),
parameters: serde_json::json!({
"type": "object",
"properties": { "city": { "type": "string" } },
"required": ["city"]
}),
},
}];
request.conversation.tool_choice = ToolChoice::Function("weather".into());
Native JSON (Mistral-style) and Qwen XML tool calls are parsed and normalized
into GenerationOutput::tool_calls as a JSON array of
{id, type, function: {name, arguments}} objects, and
finish_reason becomes "tool_calls". Malformed output, an undeclared tool, a
call that contradicts tool_choice, or a missing required call returns
Error::InvalidToolCall. Feed prior calls back through Message::tool_calls
and results through Message::tool_call_id.
Schema-constrained tools
Set GenerationRequest::tool_constraints to ToolConstraints::Schema to mask
sampling with the named tool’s JSON Schema. The mode is opt-in and currently
requires:
- the Metal or CUDA backend with device sampling;
ToolChoice::Function(name)naming a declared tool;- a native XML
<tool_call>/<function=...>template, such as Qwen’s; repetition_penalty = 1,min_tokens = 0, andignore_eos = false.
Schemas the grammar compiler cannot express are rejected before generation.
Completed arguments are validated against the original schema, including
uniqueItems, and violations return Error::InvalidToolCall.
With reasoning enabled, the model may think before the constrained call.
GenerationRequest::reasoning_token_budget optionally caps that reasoning; at
the cap only the native reasoning delimiter can be sampled. The budget requires
ReasoningMode::Enabled and ToolConstraints::Schema, and max_tokens must
leave room for the delimiter and the tool output.
JSON content
Set request.response_format = libmir::ResponseFormat::JsonSchema(schema)
to constrain final content instead of a tool call. Use {"type":"object"}
for an arbitrary JSON object. The default is ResponseFormat::Text.
The Metal/CUDA sampling restrictions are the same as schema tools. JSON
content rejects ToolConstraints::Schema and required/named ToolChoice.
Native reasoning remains separate; markers inside JSON strings stay literal.
The complete content is checked against the original schema. A partial token
callback is not a validated document. Invalid final content returns
Error::InvalidResponseFormat, mapped to invalid_model_output by the application.
reasoning_token_budget also supports JSON content with enabled reasoning and
compatible atomic reasoning delimiters. Direct Rust callers set this allowance
explicitly; the HTTP adapter’s default quarter-budget reservation is not a
library default.
Cooperative cancellation
CancellationToken is thread-safe and all clones observe the same state. Text
prefill observes it at cache waits and scheduler boundaries, and decode checks it
between steps.
use libmir::CancellationToken;
let cancellation = CancellationToken::new();
let signal = cancellation.clone();
std::thread::spawn(move || {
std::thread::sleep(std::time::Duration::from_secs(2));
signal.cancel();
});
let result = model.generate_cancellable(
&request,
&mut |_| {},
&mut |token| print!("{}", token.text),
&cancellation,
);
drop(result);
Cancellation is cooperative and returns Error::Cancelled. It does not
interrupt an individual accelerator kernel in the middle of execution.
One image
Insert IMAGE_PLACEHOLDER exactly once in the message content and provide an
encoded PNG, JPEG, WebP, or GIF:
use libmir::{CancellationToken, IMAGE_PLACEHOLDER};
let mut request = request;
request.conversation.messages[0].content =
format!("{IMAGE_PLACEHOLDER}\nDescribe this image.");
let image = std::fs::read("image.jpg")?;
let output = model.generate_image_cancellable(
&request,
&image,
&mut |_| {},
&mut |token| print!("{}", token.text),
&CancellationToken::new(),
)?;
drop(output);
Ok::<(), Box<dyn std::error::Error>>(())
The checkpoint must expose a complete supported vision contract. Resource
limits come from RuntimeConfig::vision. Image prefill observes cancellation
before and after the complete prefill, then between decode steps.
Embeddings and reranking
The same loaded Model type exposes task-specific methods. Calling the wrong
method for a checkpoint returns Error::TaskMismatch.
Embeddings
use libmir::{EmbeddingRequest, GenerationOverrides, Library, RuntimeConfig};
let model = Library::new(RuntimeConfig::default()).load(
"/models/embedding",
GenerationOverrides::default(),
&mut |_| {},
)?;
let output = model.embed(EmbeddingRequest {
inputs: vec!["first text".into(), "second text".into()],
dimensions: None,
prompt_name: None,
})?;
println!("vectors={}", output.embeddings.len());
println!("tokens={}", output.prompt_tokens);
Ok::<(), libmir::Error>(())
dimensions may retain a prefix no larger than the checkpoint’s native
dimension. prompt_name must name a preset declared by the checkpoint.
Reranking
use libmir::{GenerationOverrides, Library, RerankRequest, RuntimeConfig};
let model = Library::new(RuntimeConfig::default()).load(
"/models/reranker",
GenerationOverrides::default(),
&mut |_| {},
)?;
let output = model.rerank(RerankRequest {
query: "native inference".into(),
documents: vec!["Rust and Metal".into(), "a hosted service".into()],
max_length: None,
raw_scores: false,
})?;
for result in output.results {
println!("{} {:.4} {}", result.index, result.score, result.document);
}
Ok::<(), libmir::Error>(())
Results are relevance-sorted but preserve each candidate’s original index.
Set raw_scores to return the raw score instead of its logistic transform.
Two checkpoint contracts are served by Model::rerank, reported by
ModelDescriptor::task_plan():
| Plan | Checkpoint | Raw score |
|---|---|---|
TaskExecutionPlan::SequenceScoring | Sequence-classification cross-encoder | Classifier logit for the query/document pair. |
TaskExecutionPlan::CausalScoring | Causal decoder with a sentence-transformers LogitScore module | Final-position logit of the true token minus that of the false token. |
Causal scoring runs on Metal and CUDA. It renders the checkpoint chat template
with query and document turns, preceded by the checkpoint’s default prompt
as a system instruction when one is declared. The rendered prompt is never
truncated; a candidate longer than max_length returns Error::Context.
Typed decisions
libmir::decision loads Laya checkpoints independently of Library and the
generation scheduler. Its ModernBERT encoder answers option, boolean, and
scale questions about text or conversation state.
libmir = { version = "0.6.2", default-features = false, features = ["cpu"] }
serde_json = "1"
use libmir::decision::{DecisionBackend, DecisionModel, DecisionSchema, DecisionState};
let model = DecisionModel::load("/models/laya-multilingual", DecisionBackend::Cpu)?;
let schema = DecisionSchema::from_json_schema(&serde_json::json!({
"type": "object",
"properties": {"refund": {"type": "boolean"}}
}))?;
let values = model.decide_schema(&DecisionState::text("We were billed twice."), &schema)?;
println!("{}", serde_json::Value::Object(values));
Ok::<(), libmir::Error>(())
Select DecisionBackend::Metal with the metal feature for Apple GPUs, or
DecisionBackend::Cuda(DecisionPrecision::F32) / Bf16 with cuda for NVIDIA
GPUs. CPU and Metal evaluate in FP32; CUDA offers FP32 or BF16 matrix products
with FP32 normalization, softmax, and residuals. These precisions have separate
numerical tolerances in the reference tests.
decide returns typed Answer values with option probabilities and verdicts;
decide_schema projects them into a JSON object. rows and logits expose
tokenized questions and raw option logits for validation.
The CPU backend uses mircup: Apple Accelerate on macOS and multithreaded
GEMM elsewhere. Enabling cpu does not provide general-purpose text generation,
embeddings, or reranking through Library.
Sessions and K/V cache
Use Model::generate unless an application needs direct control over the token
loop. The low-level Session API owns independent K/V state while sharing the
loaded model and its cache arena.
use libmir::{RuntimeError, SamplingLogits};
let prepared = model.prepare(&request.conversation)?;
let mut session = model.session();
let prefill = session.prefill(
&prepared.tokens.token_ids,
SamplingLogits::None,
&mut |_| {},
)?;
let mut next = prefill.next_token.ok_or_else(|| {
RuntimeError::Backend("device sampling returned no token".into())
})?;
for _ in 0..32 {
print!("{}", model.descriptor().tokenizer().decode(&[next])?);
if model.descriptor().tokenizer().stop_token_ids().contains(&next) {
break;
}
next = session
.decode(next, SamplingLogits::None)?
.event
.token_id
.ok_or_else(|| RuntimeError::Backend("no token".into()))?;
}
println!("{:#?}", session.cache_stats());
Ok::<(), libmir::Error>(())
Model::prepare renders and tokenizes a Conversation and validates it against
the context window. SamplingLogits::None requests device sampling when
supported. Other variants can expose full or candidate logits for an
application-owned sampler.
Session::prefill_cancellable accepts a CancellationToken and observes it
between scheduled accelerator steps. Drop a session whose prefill was cancelled
instead of reusing it.
Cache behavior
The library uses page-backed K/V storage. Full ready blocks can become shared
prefix entries, and sessions use copy-on-write when a shared tail must diverge.
Session::cache_stats reports:
- resolved storage dtype and quantization mode;
- total, free, and used blocks;
- cached prefix count;
- prefix probes, hits, misses, and matching token counts.
Dropping the session releases its block table. A multimodal prefill deliberately does not publish reusable prefix-cache entries.
Runtime configuration
RuntimeConfig is an explicit Rust value. libmir never reads command-line
arguments, dotenv files, or environment variables.
use libmir::{CachedPrefillPolicy, KvCacheDType, Library, RuntimeConfig};
let mut config = RuntimeConfig::default();
config.kv_cache.block_size = 32;
config.kv_cache.block_count = 2_048;
config.kv_cache.dtype = KvCacheDType::Auto;
config.automatic_kv_cache = false;
config.scheduler.max_batch_requests = 8;
config.scheduler.max_batch_tokens = 4_096;
config.scheduler.decode_batch_wait_us = 200;
config.scheduler.decode_priority_burst = 8;
config.scheduler.cached_prefill_policy = CachedPrefillPolicy::BackendDefault;
config.memory.reserve_percent = Some(20);
config.vision.max_pixels = Some(1_048_576);
config.vision.memory_percent = 70;
let library = Library::new(config);
drop(library);
Shared configuration
| Field | Default | Purpose |
|---|---|---|
kv_cache.block_size | 16 | Tokens per page-backed cache block. |
kv_cache.block_count | 4096 | Physical block count. |
kv_cache.dtype | Auto | Backend-resolved cache storage. |
automatic_kv_cache | true | Derive block count from model shape and available memory. |
scheduler.max_batch_requests | 16 | Maximum requests in a decode batch. |
scheduler.max_batch_tokens | 8192 | Prompt plus output token budget. |
scheduler.prefill_batch_wait_us | 200 | Prefill collection window. |
scheduler.decode_batch_wait_us | 200 | Short batching window. |
scheduler.decode_priority_burst | 8 | Decode fairness control. |
scheduler.cached_prefill_policy | BackendDefault | Admission of cached continuations during decode. |
scheduler.prefill_decode_policy | BackendDefault | When newly admitted prompts begin decoding. |
scheduler.prefill_refill_policy | Closed | Admission of short prompts into a running long prefill. |
scheduler.speculation | None | Optional Speculation { draft_tokens: NonZeroUsize } for a compatible CUDA MTP checkpoint. |
memory.reserve_percent | backend-aware | Percentage of total accelerator memory retained as headroom. |
memory.reserve_bytes | backend-aware | Absolute reserve floor used with the percentage policy. |
vision.max_pixels | checkpoint cap | Maximum resized image area. |
vision.attention_budget_bytes | automatic | Vision attention workspace. |
vision.memory_percent | 80 | Available memory used by automatic budgeting. |
Scheduling policies
The scheduling policy enums default to each backend’s normal behavior. The alternatives are explicit latency trade-offs:
| Policy | Effect |
|---|---|
CachedPrefillPolicy::InterleaveOneBlock | Admits one cached queue head with at most one K/V block of work per step during resident decode. Lowers refill first-token latency at the cost of decode latency. |
PrefillDecodePolicy::CompleteCohort | Finishes a bounded admitted cohort before publishing its first tokens. Improves decode occupancy at the cost of early first-token latency. |
PrefillRefillPolicy::ShortPrompt | Lets queued prompts of at most 128 tokens join a running long prefill at CUDA chunk boundaries. Requires a completion-first CUDA runner and cannot be combined with CompleteCohort. |
Backend-specific configuration
ModelLoadOptions::kv_blocks overrides the global K/V block count for one
model. Use it to bound a small model’s cache independently of a larger chat
model; the value is a positive NonZeroU32.
RuntimeConfig::metal and RuntimeConfig::cuda exist only when their Cargo
feature is enabled. A Metal-only consumer cannot accidentally configure CUDA, and
a backend-neutral build has neither field. See Metal and CUDA.
Metal, CUDA, and CPU
The high-level Library, Model, Session, generation, embedding, and
reranking APIs are backend-neutral. Cargo features choose which native adapter
is compiled.
The facade does not re-export backend objects or backend configuration types.
Enable a feature, then adjust the defaults of RuntimeConfig::metal or
RuntimeConfig::cuda in place. Library creates the backend lazily, and
backend failures surface as libmir::Error.
Metal
libmir = { version = "0.6.2", default-features = false, features = ["metal"] }
Metal uses mirtal for explicit streams, device memory, compiled kernels,
graphs, and model-neutral operations. RuntimeConfig::metal includes batch,
cache, fusion, tuning, and diagnostic policies.
use libmir::{Library, MetalDecodeReservation, RuntimeConfig};
let mut config = RuntimeConfig::default();
config.metal.cache.decode_reservation = MetalDecodeReservation::GenerationBudget;
config.metal.tuning.cache_directory = Some("/var/cache/my-app/tuning".into());
let library = Library::new(config);
drop(library);
MetalDecodeReservation::OnePage is the default. GenerationBudget instead
reserves up to the request’s generation budget from reclaimable spare K/V pages.
On Apple Silicon, host and accelerator share unified memory. Memory snapshots distinguish active allocations from reusable cached allocations.
CUDA
libmir = { version = "0.6.2", default-features = false, features = ["cuda"] }
CUDA uses mircuda for contexts, streams, memory, NVRTC compilation, kernels,
graphs, and CUTLASS-backed plans. RuntimeConfig::cuda selects the device,
pool behavior, include paths, persistent cubin cache, startup tuning, and
execution-planning policy.
use libmir::{Library, RuntimeConfig};
let mut config = RuntimeConfig::default();
config.cuda.device_ordinal = 0;
config.cuda.nvrtc_cache_directory = Some("/var/cache/my-app/cubins".into());
config.cuda.tuning.cache_directory = Some("/var/cache/my-app/tuning".into());
let library = Library::new(config);
println!("backend={:?}", library.backend_target()?);
let memory = library.memory_snapshot()?;
println!("{}: free={:?} total={:?}", memory.source, memory.available_bytes, memory.total_bytes);
Ok::<(), libmir::Error>(())
The first call that needs the backend creates real device resources, so it is fallible. CUDA memory reporting includes driver-visible free and total device memory plus pool allocations.
The cuda-diagnostics feature implies cuda and compiles additional kernel
diagnostics, such as activation fingerprints. Leave it disabled in production.
CPU decisions
Enable cpu to use DecisionModel with the native mircup backend. This
provides Laya decisions independently of the generation scheduler; it does
not make Library a general CPU chat, embedding, or reranking engine.
See Typed decisions.
Capability differences
Version 0.6.2 does not imply identical realization for every encoding on both backends. A format may use direct execution on one platform and a bounded, persistent accelerator-side conversion on the other. Schema-constrained tool generation runs on both: CUDA applies the grammar mask in a fused sampling kernel, while Metal masks the step’s full device logits and reads back only the selected token, so constrained Metal steps skip the pipelined device-token lookahead. Admission comes from the discovered execution contract and backend capability, not a family-name allowlist. Inspect the exact checkpoint on the target machine before promising support.
See Model compatibility for the release matrix.
Nemotron-H’s state-space generation and MTP run on CUDA only. Laya decisions run on all three backends through the Rust API. The HTTP decision endpoint has a narrower CPU/Metal contract. Metal builds target native MLX 0.32.4.
OpenAI SDK clients
MiRMiR can be used by clients that allow an OpenAI-compatible base URL. Load a
model, run mirmir serve, and point the client at http://127.0.0.1:8080/v1.
Python
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:8080/v1",
api_key="local-dev",
)
stream = client.chat.completions.create(
model="qwen",
messages=[{"role": "user", "content": "Hello"}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta
if delta.content:
print(delta.content, end="", flush=True)
When MiRMiR has no API key on loopback, the SDK still requires a non-empty
placeholder api_key locally.
JavaScript and TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "http://127.0.0.1:8080/v1",
apiKey: "local-dev",
});
const result = await client.chat.completions.create({
model: "qwen",
messages: [{ role: "user", content: "Hello" }],
});
console.log(result.choices[0].message.content);
Browser applications additionally need an exact server.cors_origins entry.
Do not embed a remote-access API key into publicly shipped frontend code.
Compatibility boundary
Use only the endpoints and fields documented in this book. Function tool
calling through tools and tool_choice is supported for chat completions;
see Tool calling. Support for an OpenAI base URL
does not imply support for Responses, Assistants, audio, image generation,
structured response_format output, or undocumented sampling extensions.
Clients that send non-zero presence_penalty or frequency_penalty, or a
reasoning_effort, receive 400 and must omit those fields.
Telemetry and benchmarking
MiRMiR publishes the same runtime state to the TUI, web dashboard, CLI status, and private gRPC clients.
Metrics
| Metric | Meaning |
|---|---|
| TTFT | Time from request start to the first generated token. |
| Prefill | Prompt tokens processed per second. |
| Decode | Generated tokens processed per second after prefill. |
| E2E throughput | Completion tokens divided by complete request time. |
| Memory | Active and cached accelerator or unified-memory occupancy. |
| K/V occupancy | Used blocks, free capacity, and prefix-cache effectiveness. |
The server samples runtime metrics every second, keeps 900 samples, and
persists telemetry atomically under ~/.local/state/mirmir/telemetry.toml.
The web dashboard renders the latest 60 samples and restores them after a
restart. Shutdown waits up to two seconds for the final history flush. If a
system read or filesystem write stalls, the server logs a warning and continues
shutdown; the last unflushed samples may be lost. The history worker holds no
model or inference-runtime handles. History memory samples report host system
memory; a busy logical K/V cache retains its previous reading until the next sample.
Inspect current state
mirmir status
The command prints server health, ready models, and the latest telemetry snapshot as JSON.
Benchmark a prompt
mirmir prompt \
--model qwen \
--prompt "Explain continuous batching." \
--warmup 1 \
--samples 5 \
--seed 7 \
--json \
--csv benchmark.csv
The report contains individual samples and mean, median, p50, p95, population standard deviation, minimum, and maximum for E2E, TTFT, prefill, decode, and their durations.
Logging
mirmir serve emits structured lifecycle and request logs at INFO. Narrow or
expand application targets explicitly:
RUST_LOG=mirmir=trace,libmir=debug mirmir serve
RUST_LOG=mirmir=warn,libmir=warn mirmir serve
Model compatibility
MiRMiR 0.6.2 supports a checkpoint only when its complete execution contract is valid for the selected backend. A repository name, architecture label, or SafeTensors container is useful context, but none is a compatibility guarantee by itself.
What MiRMiR inspects
Before admission, MiRMiR combines:
- the task: generation, vision-language generation, embeddings, or reranking with an encoder or a causal decoder;
- semantic architecture: attention, normalization, positional encoding, feed-forward composition, routing, pooling, and activation;
- tokenizer, processor, chat template, and protocol tokens;
- tensor names, roles, shapes, dtypes, packing, and companion scales;
- physical weight encoding and quantization geometry;
- Metal or CUDA capabilities and the available accelerator memory.
Remote search and pull use bounded header preflight. Local inspection evaluates
the complete checkpoint. Compatibility is reported independently as
supported, partial, unsupported, or unknown for each backend.
Validated model systems
The 0.6.2 execution contracts cover these systems when every required component is present:
| System | Covered composition |
|---|---|
| Dense decoders | Grouped-query attention, RoPE variants, tied weights, and projection biases. |
| Routed MoE | Top-k routing, shared experts, dense-plus-routed blocks, and clamped routed experts. |
| Hybrid attention | Full and sliding softmax attention combined with linear or recurrent mixer layers. |
| Nemotron-H (CUDA only) | Mamba-2, attention without RoPE, latent routed experts, and optional MTP for checkpoints with a prediction head. Nemotron Omni checkpoints with the RADIO vision encoder also accept images on CUDA. |
| Laya decisions | ModernBERT decision encoder on CPU, Metal, or CUDA through the Rust API. HTTP exposes CPU and macOS Metal. |
| Embeddings | Encoder checkpoints with declared pooling and optional output normalization. |
| Sequence scoring | Encoder classification and reranking with typed pooling and output boundaries. |
| Causal scoring | Decoder checkpoints with a sentence-transformers LogitScore module, reranked from final answer-token logits on Metal and CUDA. |
| Vision-language | Vision tower, projector, image processor, protocol tokens, and text decoder admitted together. |
These are structural categories, not a closed family allowlist. Inspect the exact checkpoint before promising support.
Gemma 4 retains its Metal path and gains accelerated CUDA prefill for sliding attention and 512-wide global heads. Support claims do not imply fresh performance measurements on every release. Historical matrices identify their model revision, backend, and measured software version.
Gemma 4’s omitted normalization and scaling settings have built-in defaults:
Q/K/V RMS normalization, attention scale 1.0, and embedding multiplier
sqrt(hidden_size). A standard supported checkpoint needs no manually created
sidecar. Explicit configuration values override the defaults; a complete
mir-model-spec.toml overrides the resolved semantic contract. Metal rejects
normalization or scaling overrides its current execution path cannot implement,
with an error before weights are loaded. CUDA executes its supported declared
semantics. The dense Gemma 4 12B composition remains a CUDA path; this change
does not add a new dense GELU runner to Metal.
The validated Nemotron-3.5-Super 120B-A12B NVFP4 text weights occupy about 66.6 GiB in the CUDA implementation, with higher runtime peaks; the vision encoder adds about 1.9 GB of resident weights. Active parameter count does not describe resident weight memory: this checkpoint is not a 64 GB Mac workload, and its state-space path is not implemented on Metal.
Checkpoint ecosystems
MiRMiR 0.6.2 reads Hugging Face and MLX Community repositories stored as SafeTensors. It interprets Transformers configuration, tokenizer assets, shard indexes, and chat templates natively; Transformers, PyTorch, bitsandbytes, and Python are not runtime dependencies.
GGUF files are not admitted by the 0.6.2 runtime.
Weight encodings
| SafeTensors encoding | Metal | CUDA | Notes |
|---|---|---|---|
| F32, F16, BF16 | Supported | Supported | Native dense execution. |
| MLX grouped affine 2/3/4/5/6/8-bit | Supported | Supported | Packed weights remain accelerator-resident. |
| Compressed Tensors INT8 and INT4 | Supported | Supported | Typed packing, grouping, scale, and zero-point contracts. |
| AWQ | Supported | Supported | Backend-specific direct execution or persistent device repacking. |
| GPTQ | Supported | Supported | Includes typed format and activation-order contracts. |
| FP8 E4M3 and E5M2 | Supported | Supported | Scale direction and geometry are validated explicitly. |
| MXFP4 and MXFP8 | Supported | Supported | Block values and scale representation remain distinct. |
| NVFP4 | Supported (role-specific) | Supported (native) | Metal converts ordinary matrices once and executes routed banks directly. |
| bitsandbytes NF4 and FP4 | Supported | Supported | Codebook, block scales, nested quantization, and compute dtype are preserved. |
Supported means the format has a complete contract, backend operation path,
real-checkpoint validation, deterministic generation coverage where applicable,
and bounded accelerator-memory behavior. It does not mean that every model
architecture using that encoding is implemented.
Device conversion policy
MiRMiR may execute a checkpoint’s packed representation directly or convert it once into a persistent backend-native representation. Conversion is admitted only when it is bounded, runs on the accelerator, is included in memory preflight, and does not leave a full dense host-side shadow of the weights.
Check a checkpoint
mirmir model inspect OWNER/REPOSITORY
mirmir model load OWNER/REPOSITORY
Inspection explains the discovered task, format, backend admission, memory
estimate, and safe context capacity. A failed structural or backend check must
be resolved with a compatible checkpoint; --force cannot bypass it.
Limits and compatibility
These are the 0.6.2 product boundaries, not commitments for later releases.
HTTP and media
- chat completions support
n = 1; - embedding output supports
encoding_format = "float"; - one image may be attached to a generation request;
- supported image containers are PNG, JPEG, WebP, and GIF;
- decoded application image input is limited to 20 MiB;
- the default HTTP body limit is 29,360,128 bytes;
- default request timeout is 300 seconds;
- default HTTP concurrency is 16 requests;
- CORS is disabled until exact origins are configured;
/v1inference endpoints return503until startup model restoration completes;presence_penalty,frequency_penalty,reasoning_budget, andreasoning_effortare rejected rather than ignored;response_formatsupports text, JSON objects, and strict JSON schemas; JSON content rejects required/named tool choices and schema-tool mode;- schema conformance is checked on completion; streamed content is provisional until final validation succeeds.
Tool calling
- tool calls are validated against
toolsandtool_choice; violations return422; - schema-constrained tool decoding runs on Metal and CUDA and requires a native XML tool template such as Qwen’s;
- constrained root schemas are objects with at most 64 top-level properties
and 512 KB; remote and file
$refretrieval is disabled; thinking_token_budgetrequires enabled reasoning, text input, schema mode, and an explicitmax_tokens.
The same budget control supports JSON content. See Structured JSON output.
Web dashboard
- the dashboard is opt-in;
- it is restricted to loopback;
- its local session lasts eight hours;
- the OpenAI bearer token cannot authorize dashboard administration.
Models
- compatibility is determined from configuration, tokenizer, tensor schema, shapes, quantization, and backend capabilities;
- the 0.6.2 container boundary is SafeTensors from supported Hugging Face, Transformers, MLX, and quantization-toolchain layouts;
- a family or repository name is not a compatibility guarantee;
- Metal and CUDA have different admitted checkpoint coverage;
- Nemotron-H generation and MTP require CUDA; Metal rejects its state-space mixer;
- CPU supports Laya decisions, not general chat generation;
/v1/decideexposes CPU/Metal only; CUDA decisions use the Rust API;- a forced load bypasses memory-fit policy, not structural validation;
- model load cancellation waits for a future safe backend interruption boundary.
The complete encoding matrix is documented under Model compatibility.
libmir stability
libmir is pre-1.0. Pin 0.6.2 and review release notes before upgrading.
Since 0.4.0 the top-level facade is the only supported entry point: the
internal workspace crates are no longer re-exported.
Documentation roadmap
The 0.6.2 documentation covers the public control surfaces, HTTP contract,
model compatibility boundary, and primary libmir lifecycle. Later passes
will deepen these areas without changing their URLs.
Product guides
- screenshot-led TUI pages for Dashboard, Models, Chat, and Settings;
- screenshot-led web pages for search, load preflight, chat, images, settings, and cancellation;
- complete environment-variable reference;
- model configuration file reference;
- recovery and troubleshooting decision trees;
- platform-specific source build guides.
Developer reference
- generated item links to docs.rs for each public
libmirtype; - progress event and generation metrics field reference;
- error taxonomy and recovery guidance;
- concurrent session and throughput patterns;
- complete image preparation example;
- explicit Metal and CUDA policy tables;
- migration notes for each pre-1.0 release.
Integration policy
Tool-specific integration pages will be added only after an end-to-end test against the documented MiRMiR release. A configurable OpenAI base URL alone is not enough to claim full compatibility.