Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Runtime configuration

RuntimeConfig is an explicit Rust value. libmir never reads command-line arguments, dotenv files, or environment variables.

use libmir::{CachedPrefillPolicy, KvCacheDType, Library, RuntimeConfig};

let mut config = RuntimeConfig::default();
config.kv_cache.block_size = 32;
config.kv_cache.block_count = 2_048;
config.kv_cache.dtype = KvCacheDType::Auto;
config.automatic_kv_cache = false;
config.scheduler.max_batch_requests = 8;
config.scheduler.max_batch_tokens = 4_096;
config.scheduler.decode_batch_wait_us = 200;
config.scheduler.decode_priority_burst = 8;
config.scheduler.cached_prefill_policy = CachedPrefillPolicy::BackendDefault;
config.memory.reserve_percent = Some(20);
config.vision.max_pixels = Some(1_048_576);
config.vision.memory_percent = 70;

let library = Library::new(config);
drop(library);

Shared configuration

FieldDefaultPurpose
kv_cache.block_size16Tokens per page-backed cache block.
kv_cache.block_count4096Physical block count.
kv_cache.dtypeAutoBackend-resolved cache storage.
automatic_kv_cachetrueDerive block count from model shape and available memory.
scheduler.max_batch_requests16Maximum requests in a decode batch.
scheduler.max_batch_tokens8192Prompt plus output token budget.
scheduler.prefill_batch_wait_us200Prefill collection window.
scheduler.decode_batch_wait_us200Short batching window.
scheduler.decode_priority_burst8Decode fairness control.
scheduler.cached_prefill_policyBackendDefaultAdmission of cached continuations during decode.
scheduler.prefill_decode_policyBackendDefaultWhen newly admitted prompts begin decoding.
scheduler.prefill_refill_policyClosedAdmission of short prompts into a running long prefill.
scheduler.speculationNoneOptional Speculation { draft_tokens: NonZeroUsize } for a compatible CUDA MTP checkpoint.
memory.reserve_percentbackend-awarePercentage of total accelerator memory retained as headroom.
memory.reserve_bytesbackend-awareAbsolute reserve floor used with the percentage policy.
vision.max_pixelscheckpoint capMaximum resized image area.
vision.attention_budget_bytesautomaticVision attention workspace.
vision.memory_percent80Available memory used by automatic budgeting.

Scheduling policies

The scheduling policy enums default to each backend’s normal behavior. The alternatives are explicit latency trade-offs:

PolicyEffect
CachedPrefillPolicy::InterleaveOneBlockAdmits one cached queue head with at most one K/V block of work per step during resident decode. Lowers refill first-token latency at the cost of decode latency.
PrefillDecodePolicy::CompleteCohortFinishes a bounded admitted cohort before publishing its first tokens. Improves decode occupancy at the cost of early first-token latency.
PrefillRefillPolicy::ShortPromptLets queued prompts of at most 128 tokens join a running long prefill at CUDA chunk boundaries. Requires a completion-first CUDA runner and cannot be combined with CompleteCohort.

Backend-specific configuration

ModelLoadOptions::kv_blocks overrides the global K/V block count for one model. Use it to bound a small model’s cache independently of a larger chat model; the value is a positive NonZeroU32.

RuntimeConfig::metal and RuntimeConfig::cuda exist only when their Cargo feature is enabled. A Metal-only consumer cannot accidentally configure CUDA, and a backend-neutral build has neither field. See Metal and CUDA.