Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Configuration

Initialize, inspect, and validate configuration with:

mirmir config init
mirmir config show
mirmir config validate

The main file is ~/.config/mirmir/config.toml. Secrets are stored separately with protected permissions and are never returned in clear text by the TUI, web dashboard, CLI, or private gRPC API.

Runtime settings

KeyDefaultMeaning
runtime.kv_block_sizeautomaticTokens stored in one K/V block.
runtime.kv_blocksautomaticTotal block capacity.
runtime.speculative_draft_tokensdisabledDraft count from 1 to 8 for a compatible CUDA MTP checkpoint. auto removes the override.
runtime.kv_cache_dtypeautoBackend-resolved K/V storage type.
runtime.max_batch_requestsautomaticConcurrent decode batch limit.
runtime.max_batch_tokensautomaticScheduler token budget.
runtime.prefill_batch_wait_us200Short window used to form a prefill batch.
runtime.decode_batch_wait_usautomaticShort window used to form a decode batch.
runtime.decode_priority_burstautomaticDecode fairness before other runnable work.
runtime.cached_prefill_policybackend_defaultinterleave_one_block admits one cached continuation during resident decode, lowering refill first-token latency at some decode latency cost.
runtime.prefill_decode_policybackend_defaultcomplete_cohort finishes an admitted prefill cohort before its first tokens, improving decode occupancy at some first-token latency cost.
runtime.prefill_refill_policyclosedshort_prompt lets prompts of at most 128 tokens join a running prefill on CUDA mixed-attention models.
runtime.metal_decode_reservationone_pagemacOS only. generation_budget reserves K/V pages for the request’s whole generation budget from reclaimable spare pages.
runtime.memory_reserve_percentautomaticPercentage of visible memory kept outside model admission.
runtime.memory_reserve_bytesautomaticAdditional fixed memory reserve used by admission.
runtime.vision_max_pixelsautomaticHard cap on resized image area.
runtime.vision_attention_budget_bytesautomaticVision attention workspace cap.
runtime.vision_memory_percent80Memory available to automatic vision budgeting.

Server settings

KeyDefaultMeaning
server.http_bind127.0.0.1:8080HTTP listen address.
server.allow_remotefalsePermits a non-loopback bind.
server.web_enabledfalseServes the embedded dashboard on /ui/.
server.cors_origins[]Exact allowed browser origins.
server.body_limit_bytes29360128Maximum request body.
server.request_timeout_seconds300HTTP request timeout.
server.max_concurrency16In-flight HTTP request limit.

Most runtime and server settings require a restart. The UI identifies this per row. Set any optional runtime key to auto to remove the override and return to the backend default, for example:

mirmir config set runtime.cached_prefill_policy interleave_one_block
mirmir config set runtime.cached_prefill_policy auto

runtime.prefill_refill_policy = "short_prompt" requires the default prefill_decode_policy; validation rejects it with complete_cohort. Other platforms reject runtime.metal_decode_reservation.

Speculative decoding and model memory

For the supported Nemotron-H CUDA checkpoint with a prediction head:

mirmir config set runtime.speculative_draft_tokens 3

Restart the runtime after changing the setting. auto disables the override. MTP uses extra weights and recurrent snapshots. It can improve single-stream decode, but does not promise a gain for every prompt or concurrent batch. It is not a generic speculative decoder for every model or a Metal feature.

Context per model

A model’s context is the K/V capacity it loads with, in tokens. By default it is automatic: the server gives a model as much cache as free memory allows, which leaves no room for a second model. To run several models side by side, give each one a context. The choice is saved per model and used by every load, including restoration after a restart:

mirmir model inspect Qwen/Qwen3-8B --context 32768   # memory at 32k tokens
mirmir model load Qwen/Qwen3-8B --context 32768      # save it and load
mirmir model load Qwen/Qwen3-8B --context automatic  # back to automatic

The TUI’s load dialog, the web dashboard and Mirmir for Mac show the same choice as a Context control before Load, with the memory it needs. A context is rounded up to whole K/V blocks. The cache is shared by the requests a model serves at once, so it can be larger than the model’s context length, up to that length for each of them (runtime.max_batch_requests); one request still uses at most the model’s context length.

The choice is stored in the model’s TOML file in ~/.config/mirmir/models/ as a block count, which can also be edited by hand:

[runtime]
kv_blocks = 2048   # 2048 blocks × 16 tokens = 32,768 tokens

It takes effect when that model is loaded again, and it is separate from the global runtime.kv_blocks in config.toml.

Set and remove secrets

printf '%s' "$HF_TOKEN" | mirmir config set hugging_face.token
printf '%s' "$MIRMIR_HTTP_API_KEY" | mirmir config set server.api_key
mirmir config remove server.api_key

Environment values take precedence when present. MiRMiR recognizes HF_TOKEN and MIRMIR_HTTP_API_KEY for these two secrets.

CUDA compiler includes

On Linux, MIRMIR_CUDA_INCLUDE_PATH overrides the include directories passed to NVRTC when MiRMiR compiles CUDA kernels at runtime. It is a PATH-style, colon-separated list, useful when the CUDA headers are not under /usr/local/cuda.