Configuration
Initialize, inspect, and validate configuration with:
mirmir config init
mirmir config show
mirmir config validate
The main file is ~/.config/mirmir/config.toml. Secrets are stored separately
with protected permissions and are never returned in clear text by the TUI,
web dashboard, CLI, or private gRPC API.
Runtime settings
| Key | Default | Meaning |
|---|---|---|
runtime.kv_block_size | automatic | Tokens stored in one K/V block. |
runtime.kv_blocks | automatic | Total block capacity. |
runtime.speculative_draft_tokens | disabled | Draft count from 1 to 8 for a compatible CUDA MTP checkpoint. auto removes the override. |
runtime.kv_cache_dtype | auto | Backend-resolved K/V storage type. |
runtime.max_batch_requests | automatic | Concurrent decode batch limit. |
runtime.max_batch_tokens | automatic | Scheduler token budget. |
runtime.prefill_batch_wait_us | 200 | Short window used to form a prefill batch. |
runtime.decode_batch_wait_us | automatic | Short window used to form a decode batch. |
runtime.decode_priority_burst | automatic | Decode fairness before other runnable work. |
runtime.cached_prefill_policy | backend_default | interleave_one_block admits one cached continuation during resident decode, lowering refill first-token latency at some decode latency cost. |
runtime.prefill_decode_policy | backend_default | complete_cohort finishes an admitted prefill cohort before its first tokens, improving decode occupancy at some first-token latency cost. |
runtime.prefill_refill_policy | closed | short_prompt lets prompts of at most 128 tokens join a running prefill on CUDA mixed-attention models. |
runtime.metal_decode_reservation | one_page | macOS only. generation_budget reserves K/V pages for the request’s whole generation budget from reclaimable spare pages. |
runtime.memory_reserve_percent | automatic | Percentage of visible memory kept outside model admission. |
runtime.memory_reserve_bytes | automatic | Additional fixed memory reserve used by admission. |
runtime.vision_max_pixels | automatic | Hard cap on resized image area. |
runtime.vision_attention_budget_bytes | automatic | Vision attention workspace cap. |
runtime.vision_memory_percent | 80 | Memory available to automatic vision budgeting. |
Server settings
| Key | Default | Meaning |
|---|---|---|
server.http_bind | 127.0.0.1:8080 | HTTP listen address. |
server.allow_remote | false | Permits a non-loopback bind. |
server.web_enabled | false | Serves the embedded dashboard on /ui/. |
server.cors_origins | [] | Exact allowed browser origins. |
server.body_limit_bytes | 29360128 | Maximum request body. |
server.request_timeout_seconds | 300 | HTTP request timeout. |
server.max_concurrency | 16 | In-flight HTTP request limit. |
Most runtime and server settings require a restart. The UI identifies this per
row. Set any optional runtime key to auto to remove the override and return
to the backend default, for example:
mirmir config set runtime.cached_prefill_policy interleave_one_block
mirmir config set runtime.cached_prefill_policy auto
runtime.prefill_refill_policy = "short_prompt" requires the default
prefill_decode_policy; validation rejects it with complete_cohort. Other
platforms reject runtime.metal_decode_reservation.
Speculative decoding and model memory
For the supported Nemotron-H CUDA checkpoint with a prediction head:
mirmir config set runtime.speculative_draft_tokens 3
Restart the runtime after changing the setting. auto disables the override.
MTP uses extra weights and recurrent snapshots. It can improve single-stream
decode, but does not promise a gain for every prompt or concurrent batch.
It is not a generic speculative decoder for every model or a Metal feature.
Context per model
A model’s context is the K/V capacity it loads with, in tokens. By default it is automatic: the server gives a model as much cache as free memory allows, which leaves no room for a second model. To run several models side by side, give each one a context. The choice is saved per model and used by every load, including restoration after a restart:
mirmir model inspect Qwen/Qwen3-8B --context 32768 # memory at 32k tokens
mirmir model load Qwen/Qwen3-8B --context 32768 # save it and load
mirmir model load Qwen/Qwen3-8B --context automatic # back to automatic
The TUI’s load dialog, the web dashboard and Mirmir for Mac show the same
choice as a Context control before Load, with the memory it needs. A
context is rounded up to whole K/V blocks. The cache is shared by the requests
a model serves at once, so it can be larger than the model’s context length,
up to that length for each of them (runtime.max_batch_requests); one
request still uses at most the model’s context length.
The choice is stored in the model’s TOML file in ~/.config/mirmir/models/ as
a block count, which can also be edited by hand:
[runtime]
kv_blocks = 2048 # 2048 blocks × 16 tokens = 32,768 tokens
It takes effect when that model is loaded again, and it is separate from the
global runtime.kv_blocks in config.toml.
Set and remove secrets
printf '%s' "$HF_TOKEN" | mirmir config set hugging_face.token
printf '%s' "$MIRMIR_HTTP_API_KEY" | mirmir config set server.api_key
mirmir config remove server.api_key
Environment values take precedence when present. MiRMiR recognizes HF_TOKEN
and MIRMIR_HTTP_API_KEY for these two secrets.
CUDA compiler includes
On Linux, MIRMIR_CUDA_INCLUDE_PATH overrides the include directories passed
to NVRTC when MiRMiR compiles CUDA kernels at runtime. It is a PATH-style,
colon-separated list, useful when the CUDA headers are not under
/usr/local/cuda.