Native inference, from app to library

MiRMiR 0.3.0 is a native local model runtime with a terminal dashboard, an optional browser dashboard, a command-line interface, and an OpenAI-compatible HTTP server. libmir is the embeddable Rust inference library underneath it.

Use these docs to operate MiRMiR or build your own native application on the same model discovery, loading, generation, K/V cache, Metal, and CUDA layers.

Choose a surface

MiRMiR and libmir are pre-1.0. These pages describe version 0.3.0. Support claims require complete checkpoint and backend validation; planned formats are not described as available.

The stack

your application
      |
      +-- HTTP / SSE --> mirmir server --+
      +-- TUI / CLI --> private gRPC ----+--> libmir --> Metal / CUDA
      +-- Rust API -----------------------+

MiRMiR owns the application, configuration, transport, dashboards, model catalog, and telemetry. Interactive and scripted clients attach to the same local runtime or start a temporary one when needed. libmir owns checkpoint discovery, tokenization, generation semantics, sessions, scheduling, cache policy, and backend adapters.

Install MiRMiR

MiRMiR ships as one native binary. Install version 0.3.0 from crates.io:

cargo install mirmir --version 0.3.0 --locked
mirmir --version

The final command should print mirmir 0.3.0.

Platform requirements

MiRMiR currently targets accelerator-backed inference:

• Apple Silicon macOS uses the Metal backend and requires the native MLX framework;
• Linux uses CUDA and requires a supported NVIDIA driver and CUDA Toolkit;
• no Python, PyTorch, or Transformers runtime is used in the request path.

For a source build, clone the application repository and build it using its pinned Rust toolchain:

git clone https://github.com/mirmir-tech/mirmir.git
cd mirmir
cargo build --release --locked

Initialize configuration

mirmir config init
mirmir config validate

The persistent files are stored under ~/.config/mirmir. Runtime state and the fallback Unix socket live under ~/.local/state/mirmir.

Open the terminal dashboard

Run MiRMiR without a subcommand:

mirmir

The process attaches to an existing local runtime when one is available. If it starts a temporary runtime itself, it also shuts that runtime down when the TUI exits.

Continue with Run your first model.

Upgrading an existing installation? Read Upgrade from 0.2.

Run your first model

MiRMiR can search Hugging Face, download a snapshot into its managed cache, inspect compatibility, and load it into accelerator memory.

Search and download

mirmir model search "Qwen 2.5" --limit 10
mirmir model pull Qwen/Qwen2.5-0.5B-Instruct

Search results are ranked by the execution contract discovered by libmir and by a conservative estimate of whether the checkpoint fits this machine. MiRMiR evaluates architecture and physical encoding separately, including dense, integer, block-floating-point, and codebook-quantized SafeTensors.

For gated repositories, configure a Hugging Face token without placing the secret directly in shell history:

printf '%s' "$HF_TOKEN" | mirmir config set hugging_face.token
mirmir config test hugging_face.token

Inspect and load

mirmir model inspect Qwen/Qwen2.5-0.5B-Instruct
mirmir model load Qwen/Qwen2.5-0.5B-Instruct

Inspection identifies the task, tokenizer, tensor layout, memory requirement, configured K/V cache, and safe context capacity before loading weights. --force bypasses only the conservative memory-fit rejection; it cannot make an unsupported execution contract valid.

See Model compatibility for the 0.3.0 format and backend matrix.

Generate

Open the TUI and select Chat, or use the non-interactive path:

mirmir prompt \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --prompt "Explain paged K/V cache in three sentences."

For a reproducible measurement:

mirmir prompt \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --prompt "Explain paged K/V cache." \
  --seed 7 --temperature 0 --warmup 1 --samples 5 --json

Unload or remove

mirmir model unload Qwen/Qwen2.5-0.5B-Instruct
mirmir model remove Qwen/Qwen2.5-0.5B-Instruct

Removing is destructive and requires the model to be unloaded first.

Upgrade from 0.2

MiRMiR 0.3.0 changes the application from a collection of separate command paths into several control surfaces for one local runtime. The TUI, CLI, web dashboard, and HTTP API now observe and operate the same loaded models, configuration, request activity, and telemetry.

Choose who owns the runtime

• Run mirmir serve for a persistent runtime shared by local clients and, when enabled, the HTTP API and web dashboard.
• Run mirmir to open the TUI. It attaches to an existing runtime or starts a temporary one when none is available.
• Model and prompt commands follow the same rule: attach when possible, otherwise start a temporary runtime for the operation.

A client only shuts down a temporary runtime that it started. Exiting a TUI or CLI command attached to mirmir serve leaves the persistent runtime running.

Manage models as a lifecycle

Model discovery, storage, compatibility, and accelerator residency are separate operations in 0.3.0:

mirmir model search "Qwen 3" --limit 20
mirmir model pull Qwen/Qwen3-4B
mirmir model inspect Qwen/Qwen3-4B
mirmir model load Qwen/Qwen3-4B
mirmir model unload Qwen/Qwen3-4B

Search and pull perform bounded remote inspection. Load performs structural, backend, and memory preflight before allocating accelerator weights. --force can override only the conservative memory-fit decision; it cannot admit an unsupported checkpoint.

Move settings into the shared configuration

Use mirmir config instead of maintaining per-command runtime flags:

mirmir config init
mirmir config show
mirmir config validate
mirmir config set default_model Qwen/Qwen3-4B

Configuration records effective values and their source. Secrets live in a separate protected file and remain redacted in every control surface. When a persistent server is running, configuration mutations go through it so there is one writer.

Review model compatibility

Version 0.3.0 admits a checkpoint from its task, architecture, tokenizer, tensor schema, weight encoding, and selected backend. Family names and Hub tags are not support guarantees. The release adds typed SafeTensors paths for dense, integer, block-floating-point, and codebook quantization, including bitsandbytes NF4 and FP4 checkpoints without a Python runtime.

See Model compatibility for the complete 0.3.0 matrix.

Check the public API boundary

The OpenAI-compatible surface remains deliberately smaller than the complete OpenAI platform. Existing clients should target the documented chat, embedding, reranking, model-list, and health endpoints. Management operations belong to the TUI, CLI, web dashboard, or private local control protocol.

Upgrade checklist

1. Install the 0.3.0 binary and run mirmir --version.
2. Run mirmir config validate against the existing configuration.
3. Inspect each checkpoint on its target Metal or CUDA machine.
4. Start mirmir serve if the runtime must outlive an interactive client.
5. Confirm API authentication, CORS, and dashboard settings before binding beyond a local development workflow.

Terminal dashboard

mirmir opens the terminal dashboard. All views operate on the same local runtime used by CLI and HTTP clients.

Views

Use Tab and Shift+Tab, a function key, or the mouse to switch views. Press ? for context-sensitive help.

The Dashboard aggregates current and mean request rates, memory and K/V-cache history, recent operations, and the active runtime stage without forcing a host synchronization in the generation path.

Models

• / starts a Hugging Face search;
• d, l, and u download, load, and unload;
• i includes incompatible or over-budget remote results;
• r opens a destructive removal confirmation;
• X cancels the selected cancellable operation;
• f explicitly allows an over-budget load from the load dialog.

Download progress is streamed from the server. Additional downloads wait in a FIFO queue and can be cancelled before or during transfer.

Download progress

The transfer panel names the repository and current file, and reports both percentage and transferred bytes. Press X with the active transfer selected to request cancellation; queued downloads can be cancelled the same way.

Memory preflight

Before loading, MiRMiR combines weight, K/V-cache, and workspace estimates and compares the result with the accelerator budget. The dialog also reports a safe context limit. An over-budget result is blocked unless you deliberately press f; forcing a load does not bypass compatibility checks.

Chat

The input remains locked while a request is active. Streamed reasoning is kept separate from final content, and the view reports TTFT, prefill, decode, and end-to-end throughput.

• Ctrl+Left / Ctrl+Right selects a loaded model;
• Ctrl+P edits one-off generation parameters;
• Ctrl+K clears the conversation;
• drop one PNG, JPEG, WebP, or GIF to attach it;
• Ctrl+D removes the attachment;
• X requests cooperative cancellation.

During streaming, the active assistant message keeps reasoning collapsible and the composer reports TTFT, prefill rate, decode rate, and token counts. Input is locked until the request finishes or cooperative cancellation completes.

Image attachment is available only when the selected model advertises image input. The attachment name appears above the prompt; Ctrl+D removes it before submission.

Activity and cancellation

The Dashboard records long-running operations and their stage. Select a cancellable operation and press X. MiRMiR marks it as cancelling immediately, but the operation stops cooperatively at a safe boundary rather than in the middle of an accelerator kernel.

Settings

The settings view presents the effective value, its source, and whether a server restart is required. Press Enter to edit the selected setting. The Hugging Face token can be tested with t, and stored secrets remain redacted.

Exit behavior

Esc opens confirmation. q exits outside the chat input, and Ctrl+C exits immediately. Only a TUI that owns its temporary runtime shuts it down.

Web dashboard

The browser dashboard is disabled by default. Enable it and restart the server:

mirmir config set server.web_enabled true
mirmir serve

Open http://127.0.0.1:8080/ui/.

What it exposes

• Dashboard shows live and historical throughput, prefill, decode, TTFT, memory, K/V occupancy, runtime counters, and recent activity;
• Models lists local snapshots, searches Hugging Face, and performs download, inspect, load, unload, and removal actions;
• Chat streams content and reasoning, accepts one image, exposes generation parameters, and reports request metrics;
• Settings shows effective values, their source, restart requirements, paths, raw TOML, and redacted secrets.

One WebSocket carries telemetry, model snapshots, configuration, startup readiness, and activity. The browser does not poll each view independently.

Models

Search Hugging Face by owner or model name and manage snapshots already present on the machine. Model actions use the same runtime queue and memory checks as the terminal and CLI clients.

Chat

Select a loaded model, optionally attach one supported image, adjust one-off generation parameters, and stream reasoning and final content independently. The footer reports TTFT, prefill, decode, and end-to-end throughput.

Settings

Each row shows the effective value, its source, and whether changing it requires a restart. Secrets are redacted and use dedicated mutation paths.

Local security boundary

The dashboard can only be enabled on a loopback HTTP bind. Opening it creates an eight-hour local session in an HttpOnly, SameSite=Strict cookie. Mutations also require a CSRF token held in page memory. The OpenAI bearer token is not accepted as dashboard administration authority.

For remote programmatic access, keep the dashboard local and configure the HTTP API security boundary separately.

Appearance and runtime details

The theme follows your system until you choose another appearance with the sun/moon button in the sidebar. Your choice is remembered. The Dashboard view includes prefill and decode charts with gradient fills, plus server platform, backend support, device and host-memory details. Each model row shows its format, backend compatibility and capabilities directly. Use Open chat for a resident model. Generation details in the chat composer reveal the most recent response metrics. Screenshots use demonstration data.

Configuration

Initialize, inspect, and validate configuration with:

mirmir config init
mirmir config show
mirmir config validate

The main file is ~/.config/mirmir/config.toml. Secrets are stored separately with protected permissions and are never returned in clear text by the TUI, web dashboard, CLI, or private gRPC API.

Runtime settings

Server settings

Most runtime and server settings require a restart. The UI identifies this per row.

Set and remove secrets

printf '%s' "$HF_TOKEN" | mirmir config set hugging_face.token
printf '%s' "$MIRMIR_HTTP_API_KEY" | mirmir config set server.api_key
mirmir config remove server.api_key

Environment values take precedence when present. MiRMiR recognizes HF_TOKEN and MIRMIR_HTTP_API_KEY for these two secrets.

CLI reference

Running mirmir without a subcommand opens the TUI. Commands expose the same runtime for servers, scripts, model management, configuration, and benchmarks.

Every interactive or one-shot client first tries the local private protocol. If no runtime exists, it starts a temporary one and shuts down only that owned instance when the command finishes. Use mirmir serve when models and runtime state must persist across clients.

Serve

mirmir serve
mirmir serve --no-http
mirmir serve --http-bind 127.0.0.1:9090

MIRMIR_HTTP_BIND supplies the same per-process override as --http-bind.

Prompt and benchmarks

Input comes from exactly one of --prompt, --prompt-file, or --stdin. Generation parameters include --max-tokens, --temperature, --top-p, --top-k, --repetition-penalty, and --seed.

printf 'Explain continuous batching.' | mirmir prompt \
  --stdin --model qwen --warmup 1 --samples 5 --json --csv result.csv

--no-stream buffers the final completion. JSON benchmark output contains raw samples and aggregate E2E, TTFT, prefill, and decode distributions.

Model lifecycle

mirmir model list
mirmir model inspect MODEL
mirmir model search QUERY --limit 20
mirmir model pull OWNER/REPO --revision REVISION
mirmir model load MODEL [--force]
mirmir model unload MODEL
mirmir model remove OWNER/REPO

Selectors may be a configured alias, repository ID, local model key, or checkpoint path where the command permits it.

Configuration

mirmir config init
mirmir config show
mirmir config edit
mirmir config validate
mirmir config set KEY [VALUE]
mirmir config remove hugging_face.token
mirmir config test hugging_face.token

When a server is running, commands that change configuration use it as the sole writer.

Runtime ownership

OpenAI-compatible API

mirmir serve listens on http://127.0.0.1:8080 by default. The 0.3.0 public HTTP contract contains:

The dashboard management API under /api/mirmir/v1 is private to the embedded web application. It is versioned separately and is not an OpenAI client API.

Quick start

mirmir serve
curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/v1/models

Bearer authentication is optional on loopback unless an API key has been configured. When configured, send it on every /v1 request:

curl http://127.0.0.1:8080/v1/models \
  -H "Authorization: Bearer $MIRMIR_HTTP_API_KEY"

Errors

Errors use an OpenAI-style envelope:

{
  "error": {
    "message": "model is not loaded",
    "type": "invalid_request_error",
    "param": null,
    "code": "not_found"
  }
}

MiRMiR maps invalid requests to 400, authentication to 401, permission failures to 403, missing resources to 404, state conflicts to 409, capacity exhaustion to 429, unavailable runtime state to 503, and unexpected runtime failures to 500.

Chat completions

Use POST /v1/chat/completions with a model already loaded into the shared runtime.

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $MIRMIR_HTTP_API_KEY" \
  -d '{
    "model": "qwen",
    "messages": [{"role": "user", "content": "Hello"}],
    "temperature": 0.2,
    "max_tokens": 256
  }'

Parameters

Streaming

curl -N http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

Each event contains a chat.completion.chunk. The stream ends with data: [DONE]. Reasoning-capable models emit delta.reasoning_content separately from delta.content.

One image

The OpenAI-style image_url content part accepts a data URL. Version 0.3.0 supports one PNG, JPEG, WebP, or GIF per request.

IMAGE_DATA=$(base64 < image.jpg | tr -d '\n')
curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"vision-model\",\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/jpeg;base64,$IMAGE_DATA\"}},{\"type\":\"text\",\"text\":\"Describe this image\"}]}]}"

The application limit is 20 MiB per decoded image. The server body limit must also accommodate the larger base64 request.

Embeddings and reranking

MiRMiR discovers the task from the checkpoint structure. A generation model cannot be used as an embedding or sequence-scoring model.

Embeddings

curl http://127.0.0.1:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "embedding",
    "input": ["first text", "second text"],
    "encoding_format": "float"
  }'

The output follows the OpenAI list shape and returns normalized f32 vectors plus prompt token usage.

Reranking

curl http://127.0.0.1:8080/v1/rerank \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "reranker",
    "query": "native inference",
    "documents": ["Rust and Metal", "a hosted Python service"],
    "return_documents": true
  }'

Results are sorted from highest to lowest relevance while retaining the original document index.

Security and remote access

Loopback is the default and safest mode:

[server]
http_bind = "127.0.0.1:8080"
allow_remote = false
web_enabled = false
cors_origins = []

Require an API key

printf '%s' "$MIRMIR_HTTP_API_KEY" | mirmir config set server.api_key

Once configured, clients send:

Authorization: Bearer YOUR_KEY

Bind beyond loopback

A non-loopback bind requires both explicit remote access and an API key:

[server]
http_bind = "0.0.0.0:8080"
allow_remote = true
web_enabled = false

The embedded dashboard deliberately cannot be exposed on this bind. Put TLS, network policy, and any public rate limiting in a trusted reverse proxy or private network boundary.

Browser origins

CORS is disabled until exact origins are listed:

[server]
cors_origins = ["https://app.example.com"]

MiRMiR accepts GET and POST with Authorization and Content-Type headers from configured origins. Do not use * as a substitute for deciding which applications may reach a model server.

Embed libmir

libmir is the model-neutral Rust inference library used by MiRMiR. It discovers Hugging Face-style checkpoints, loads a native accelerator backend, creates independent inference sessions, and exposes generation, embeddings, reranking, progress, cancellation, metrics, and memory information.

It deliberately does not contain:

• CLI argument or environment parsing;
• an HTTP, gRPC, or WebSocket server;
• a terminal or browser interface;
• Hugging Face search and download policy;
• application configuration files.

Those responsibilities belong to an embedding application such as MiRMiR.

Public lifecycle

ModelDescriptor::inspect(path)
        |
        v
Library::new(RuntimeConfig)
        |
        v
Library::load(path) -> Model
        |
        +-- Model::generate(...)
        +-- Model::embed(...)
        +-- Model::rerank(...)
        +-- Model::session() -> prefill / decode
        +-- Model::unload()

Applications should depend on the top-level libmir package. Its internal workspace crates are implementation details, even when they expose useful low-level types during repository development.

Choose an entry point

Start with Add libmir to a project.

Add libmir to a project

Version 0.3.0 enables no accelerator by default. Select the backend explicitly.

Apple Metal

[dependencies]
libmir = { version = "0.3.0", default-features = false, features = ["metal"] }

The Metal feature currently requires Apple Silicon, a full Xcode installation with the separately installed Metal toolchain, and native MLX.

NVIDIA CUDA

[dependencies]
libmir = { version = "0.3.0", default-features = false, features = ["cuda"] }

CUDA currently targets Linux with the NVIDIA driver, CUDA Toolkit 13.x, NVRTC, CCCL/CUB headers, and CUTLASS 4.4.2 headers.

Minimal generation

use std::io::{self, Write};

use libmir::{
    ChatCompletionRequest, ChatMessage, GenerationOverrides, Library,
    RuntimeConfig,
};

fn main() -> libmir::Result<()> {
    let model_path = std::env::args_os().nth(1).expect("model path");
    let library = Library::new(RuntimeConfig::default());
    let model = library.load(
        model_path,
        GenerationOverrides::default(),
        &mut |progress| eprintln!("{progress:?}"),
    )?;

    let request = ChatCompletionRequest {
        model: model.handle().id.clone(),
        messages: vec![ChatMessage {
            role: "user".into(),
            content: "Explain paged K/V cache briefly.".into(),
            reasoning_content: None,
        }],
        stream: true,
        max_tokens: Some(256),
        temperature: Some(0.0),
        top_p: Some(1.0),
        top_k: Some(0),
        repetition_penalty: Some(1.0),
        seed: Some(7),
    };

    let output = model.generate(&request, &mut |_| {}, &mut |token| {
        print!("{}", token.text);
        let _ = io::stdout().flush();
    })?;

    println!("\nfinish_reason={}", output.finish_reason);
    Ok(())
}

Library initializes its native backend lazily. Loading inspects the checkpoint first, resolves the execution contract, then reports real loading progress through the callback.

The same lifecycle is available as the generate example in the libmir repository.

Inspect and load models

Inspection is separate from accelerator allocation. Use it to reject an unsupported checkpoint or present its capabilities before committing device memory.

use libmir::{GenerationOverrides, ModelDescriptor};

let descriptor = ModelDescriptor::inspect(
    "/models/qwen",
    GenerationOverrides::default(),
)?;
let manifest = descriptor.manifest()?;

println!("id={}", manifest.id);
println!("context={}", manifest.context_len);
println!("task={:?}", descriptor.task());
println!("quantization={:?}", manifest.quantization);
println!("generation={:?}", descriptor.generation());
Ok::<(), libmir::Error>(())

Discovery is based on configuration values, tokenizer assets, tensor names and shapes, and the resulting execution contract. A recognizable repository or architecture name alone does not admit a checkpoint.

Estimate memory

use libmir::{GenerationOverrides, ModelDescriptor, RuntimeConfig};

let descriptor = ModelDescriptor::inspect(
    "/models/qwen",
    GenerationOverrides::default(),
)?;
let estimate = descriptor.memory_estimate(&RuntimeConfig::default());

println!("weights={} bytes", estimate.weight_bytes);
println!("kv={} bytes", estimate.kv_cache_bytes);
println!("workspace={} bytes", estimate.workspace_bytes);
println!("required={} bytes", estimate.required_bytes);
Ok::<(), libmir::Error>(())

This is a conservative planning value, not a reservation. Compare it with Library::memory_snapshot() on the backend that will load the model.

Load and unload

use libmir::{GenerationOverrides, Library, RuntimeConfig};

let library = Library::new(RuntimeConfig::default());
let model = library.load(
    "/models/qwen",
    GenerationOverrides::default(),
    &mut |event| eprintln!("{event:?}"),
)?;

println!("backend={:?}", model.engine().target());
model.unload()?;
Ok::<(), libmir::Error>(())

Model is cheaply cloneable. Model::unload succeeds only when the value is the sole remaining owner; active clones or sessions produce Error::ModelInUse. Dropping a Session releases its backend and K/V resources automatically.

Generate and stream

Model::generate is the preferred high-level text API. It renders the checkpoint chat template, tokenizes, creates an independent session, performs prefill, samples and decodes tokens, normalizes model-specific channels, and returns metrics.

Content and reasoning

The token callback receives GenerationToken values with a semantic channel. Native think markers and channel headers are removed from the public contract. The final GenerationOutput contains separate text and reasoning fields.

let output = model.generate(
    &request,
    &mut |progress| eprintln!("{progress:?}"),
    &mut |token| println!("{:?}: {}", token.channel, token.text),
)?;

println!("answer={}", output.text);
println!("reasoning={}", output.reasoning);
println!("metrics={:#?}", output.metrics);
Ok::<(), libmir::Error>(())

Cooperative cancellation

CancellationToken is thread-safe and all clones observe the same state. The generation loop checks it before prefill, after prefill, and between decode steps.

use libmir::CancellationToken;

let cancellation = CancellationToken::new();
let signal = cancellation.clone();

std::thread::spawn(move || {
    std::thread::sleep(std::time::Duration::from_secs(2));
    signal.cancel();
});

let result = model.generate_cancellable(
    &request,
    &mut |_| {},
    &mut |token| print!("{}", token.text),
    &cancellation,
);
drop(result);

Cancellation is cooperative. It does not interrupt an individual accelerator kernel in the middle of execution.

One image

Insert IMAGE_PLACEHOLDER exactly once in the message content and provide an encoded PNG, JPEG, WebP, or GIF:

use libmir::{CancellationToken, IMAGE_PLACEHOLDER};

let mut request = request;
request.messages[0].content = format!("{IMAGE_PLACEHOLDER}\nDescribe this image.");
let image = std::fs::read("image.jpg")?;

let output = model.generate_image_cancellable(
    &request,
    &image,
    &mut |_| {},
    &mut |token| print!("{}", token.text),
    &CancellationToken::new(),
)?;
drop(output);
Ok::<(), Box<dyn std::error::Error>>(())

The checkpoint must expose a complete supported vision contract. Resource limits come from RuntimeConfig::vision.

Embeddings and reranking

The same loaded Model type exposes task-specific methods. Calling the wrong method for a checkpoint returns Error::TaskMismatch.

Embeddings

use libmir::{EmbeddingRequest, GenerationOverrides, Library, RuntimeConfig};

let model = Library::new(RuntimeConfig::default()).load(
    "/models/embedding",
    GenerationOverrides::default(),
    &mut |_| {},
)?;

let output = model.embed(EmbeddingRequest {
    inputs: vec!["first text".into(), "second text".into()],
    dimensions: None,
    prompt_name: None,
})?;

println!("vectors={}", output.embeddings.len());
println!("tokens={}", output.prompt_tokens);
Ok::<(), libmir::Error>(())

dimensions may retain a prefix no larger than the checkpoint’s native dimension. prompt_name must name a preset declared by the checkpoint.

Reranking

use libmir::{GenerationOverrides, Library, RerankRequest, RuntimeConfig};

let model = Library::new(RuntimeConfig::default()).load(
    "/models/reranker",
    GenerationOverrides::default(),
    &mut |_| {},
)?;

let output = model.rerank(RerankRequest {
    query: "native inference".into(),
    documents: vec!["Rust and Metal".into(), "a hosted service".into()],
    max_length: None,
    raw_scores: false,
})?;

for result in output.results {
    println!("{} {:.4} {}", result.index, result.score, result.document);
}
Ok::<(), libmir::Error>(())

Results are relevance-sorted but preserve each candidate’s original index. Set raw_scores to return classifier logits instead of a logistic score.

Sessions and K/V cache

Use Model::generate unless an application needs direct control over the token loop. The low-level Session API owns independent K/V state while sharing the loaded model and its cache arena.

use libmir::{SamplingLogits, runtime::RuntimeError};

let prepared = model.prepare(&request)?;
let mut session = model.session();
let prefill = session.prefill(
    &prepared.tokens.token_ids,
    SamplingLogits::None,
    &mut |_| {},
)?;
let mut next = prefill.next_token.ok_or_else(|| {
    RuntimeError::Backend("device sampling returned no token".into())
})?;

for _ in 0..32 {
    print!("{}", model.descriptor().tokenizer().decode(&[next])?);
    if model.descriptor().tokenizer().stop_token_ids().contains(&next) {
        break;
    }
    next = session
        .decode(next, SamplingLogits::None)?
        .event
        .token_id
        .ok_or_else(|| RuntimeError::Backend("no token".into()))?;
}

println!("{:#?}", session.cache_stats());
Ok::<(), libmir::Error>(())

SamplingLogits::None requests device sampling when supported. Other variants can expose full or candidate logits for an application-owned sampler.

Cache behavior

The library uses page-backed K/V storage. Full ready blocks can become shared prefix entries, and sessions use copy-on-write when a shared tail must diverge. Session::cache_stats reports:

• resolved storage dtype and quantization mode;
• total, free, and used blocks;
• cached prefix count;
• prefix probes, hits, misses, and matching token counts.

Dropping the session releases its block table. A multimodal prefill deliberately does not publish reusable prefix-cache entries.

Runtime configuration

RuntimeConfig is an explicit Rust value. libmir never reads command-line arguments, dotenv files, or environment variables.

use libmir::{KvCacheDType, Library, RuntimeConfig};

let mut config = RuntimeConfig::default();
config.kv_cache.block_size = 32;
config.kv_cache.block_count = 2_048;
config.kv_cache.dtype = KvCacheDType::Auto;
config.automatic_kv_cache = false;
config.scheduler.max_batch_requests = 8;
config.scheduler.max_batch_tokens = 4_096;
config.scheduler.decode_batch_wait_us = 200;
config.scheduler.decode_priority_burst = 8;
config.memory.reserve_percent = Some(20);
config.vision.max_pixels = Some(1_048_576);
config.vision.memory_percent = 70;

let library = Library::new(config);
drop(library);

Shared configuration

Backend-specific fields exist only when their Cargo feature is enabled. A Metal-only consumer cannot accidentally configure CUDA, and a backend-neutral build has neither field.

Metal and CUDA

The high-level Library, Model, Session, generation, embedding, and reranking APIs are backend-neutral. Cargo features choose which native adapter is compiled.

Metal

libmir = { version = "0.3.0", default-features = false, features = ["metal"] }

Metal uses mirtal for explicit streams, device memory, compiled kernels, graphs, and model-neutral operations. RuntimeConfig::metal includes batch, cache, fusion, and diagnostic policies.

On Apple Silicon, host and accelerator share unified memory. Memory snapshots distinguish active allocations from reusable cached allocations.

CUDA

libmir = { version = "0.3.0", default-features = false, features = ["cuda"] }

CUDA uses mircuda for contexts, streams, memory, NVRTC compilation, kernels, graphs, and CUTLASS-backed plans. RuntimeConfig::cuda selects the device, pool behavior, include paths, persistent PTX cache, and execution-planning policy.

use libmir::{CudaBackend, CudaConfig};

let backend = CudaBackend::new(CudaConfig::default())?;
let device = backend.device_info();
println!("CUDA device: {}", device.name);
Ok::<(), libmir::CudaError>(())

Backend construction is fallible because it creates real device resources. CUDA memory reporting includes driver-visible free and total device memory plus pool allocations.

Capability differences

Version 0.3.0 does not imply identical realization for every encoding on both backends. A format may use direct execution on one platform and a bounded, persistent accelerator-side conversion on the other. Admission comes from the discovered execution contract and backend capability, not a family-name allowlist. Inspect the exact checkpoint on the target machine before promising support.

See Model compatibility for the release matrix.

OpenAI SDK clients

MiRMiR can be used by clients that allow an OpenAI-compatible base URL. Load a model, run mirmir serve, and point the client at http://127.0.0.1:8080/v1.

Python

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8080/v1",
    api_key="local-dev",
)

stream = client.chat.completions.create(
    model="qwen",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)

When MiRMiR has no API key on loopback, the SDK still requires a non-empty placeholder api_key locally.

JavaScript and TypeScript

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://127.0.0.1:8080/v1",
  apiKey: "local-dev",
});

const result = await client.chat.completions.create({
  model: "qwen",
  messages: [{ role: "user", content: "Hello" }],
});

console.log(result.choices[0].message.content);

Browser applications additionally need an exact server.cors_origins entry. Do not embed a remote-access API key into publicly shipped frontend code.

Compatibility boundary

Use only the endpoints and fields documented in this book. Support for an OpenAI base URL does not imply support for Responses, Assistants, audio, image generation, tool calling, or undocumented sampling extensions.

Telemetry and benchmarking

MiRMiR publishes the same runtime state to the TUI, web dashboard, CLI status, and private gRPC clients.

Metrics

The server samples runtime metrics every second, keeps 900 samples, and persists telemetry atomically under ~/.local/state/mirmir/telemetry.toml. The web dashboard renders the latest 60 samples and restores them after a restart.

Inspect current state

mirmir status

The command prints server health, ready models, and the latest telemetry snapshot as JSON.

Benchmark a prompt

mirmir prompt \
  --model qwen \
  --prompt "Explain continuous batching." \
  --warmup 1 \
  --samples 5 \
  --seed 7 \
  --json \
  --csv benchmark.csv

The report contains individual samples and mean, median, p50, p95, population standard deviation, minimum, and maximum for E2E, TTFT, prefill, decode, and their durations.

Logging

mirmir serve emits structured lifecycle and request logs at INFO. Narrow or expand application targets explicitly:

RUST_LOG=mirmir=trace,libmir=debug mirmir serve
RUST_LOG=mirmir=warn,libmir=warn mirmir serve

Model compatibility

MiRMiR 0.3.0 supports a checkpoint only when its complete execution contract is valid for the selected backend. A repository name, architecture label, or SafeTensors container is useful context, but none is a compatibility guarantee by itself.

What MiRMiR inspects

Before admission, MiRMiR combines:

• the task: generation, vision-language generation, embeddings, or reranking;
• semantic architecture: attention, normalization, positional encoding, feed-forward composition, routing, pooling, and activation;
• tokenizer, processor, chat template, and protocol tokens;
• tensor names, roles, shapes, dtypes, packing, and companion scales;
• physical weight encoding and quantization geometry;
• Metal or CUDA capabilities and the available accelerator memory.

Remote search and pull use bounded header preflight. Local inspection evaluates the complete checkpoint. Compatibility is reported independently as supported, partial, unsupported, or unknown for each backend.

Validated model systems

The 0.3.0 execution contracts cover these systems when every required component is present:

These are structural categories, not a closed family allowlist. Inspect the exact checkpoint before promising support.

Checkpoint ecosystems

MiRMiR 0.3.0 reads Hugging Face and MLX Community repositories stored as SafeTensors. It interprets Transformers configuration, tokenizer assets, shard indexes, and chat templates natively; Transformers, PyTorch, bitsandbytes, and Python are not runtime dependencies.

GGUF files are not admitted by the 0.3.0 runtime.

Weight encodings

Supported means the format has a complete contract, backend operation path, real-checkpoint validation, deterministic generation coverage where applicable, and bounded accelerator-memory behavior. It does not mean that every model architecture using that encoding is implemented.

Device conversion policy

MiRMiR may execute a checkpoint’s packed representation directly or convert it once into a persistent backend-native representation. Conversion is admitted only when it is bounded, runs on the accelerator, is included in memory preflight, and does not leave a full dense host-side shadow of the weights.

Check a checkpoint

mirmir model inspect OWNER/REPOSITORY
mirmir model load OWNER/REPOSITORY

Inspection explains the discovered task, format, backend admission, memory estimate, and safe context capacity. A failed structural or backend check must be resolved with a compatible checkpoint; --force cannot bypass it.

Limits and compatibility

These are the 0.3.0 product boundaries, not commitments for later releases.

HTTP and media

• chat completions support n = 1;
• embedding output supports encoding_format = "float";
• one image may be attached to a generation request;
• supported image containers are PNG, JPEG, WebP, and GIF;
• decoded application image input is limited to 20 MiB;
• the default HTTP body limit is 29,360,128 bytes;
• default request timeout is 300 seconds;
• default HTTP concurrency is 16 requests;
• CORS is disabled until exact origins are configured.

Web dashboard

• the dashboard is opt-in;
• it is restricted to loopback;
• its local session lasts eight hours;
• the OpenAI bearer token cannot authorize dashboard administration.

Models

• compatibility is determined from configuration, tokenizer, tensor schema, shapes, quantization, and backend capabilities;
• the 0.3.0 container boundary is SafeTensors from supported Hugging Face, Transformers, MLX, and quantization-toolchain layouts;
• a family or repository name is not a compatibility guarantee;
• Metal and CUDA have different admitted checkpoint coverage;
• a forced load bypasses memory-fit policy, not structural validation;
• model load cancellation waits for a future safe backend interruption boundary.

The complete encoding matrix is documented under Model compatibility.

libmir stability

libmir is pre-1.0. Pin 0.3.0, review release notes before upgrading, and prefer the top-level facade over internal workspace crates.

Documentation roadmap

The 0.3.0 documentation covers the public control surfaces, HTTP contract, model compatibility boundary, and primary libmir lifecycle. Later passes will deepen these areas without changing their URLs.

Product guides

• screenshot-led TUI pages for Dashboard, Models, Chat, and Settings;
• screenshot-led web pages for search, load preflight, chat, images, settings, and cancellation;
• complete environment-variable reference;
• model configuration file reference;
• recovery and troubleshooting decision trees;
• platform-specific source build guides.

Developer reference

• generated item links to docs.rs for each public libmir type;
• progress event and generation metrics field reference;
• error taxonomy and recovery guidance;
• concurrent session and throughput patterns;
• complete image preparation example;
• explicit Metal and CUDA policy tables;
• migration notes for each pre-1.0 release.

Integration policy

Tool-specific integration pages will be added only after an end-to-end test against MiRMiR 0.3.x. A configurable OpenAI base URL alone is not enough to claim full compatibility.