Runtime configuration
RuntimeConfig is an explicit Rust value. libmir never reads command-line
arguments, dotenv files, or environment variables.
use libmir::{CachedPrefillPolicy, KvCacheDType, Library, RuntimeConfig};
let mut config = RuntimeConfig::default();
config.kv_cache.block_size = 32;
config.kv_cache.block_count = 2_048;
config.kv_cache.dtype = KvCacheDType::Auto;
config.automatic_kv_cache = false;
config.scheduler.max_batch_requests = 8;
config.scheduler.max_batch_tokens = 4_096;
config.scheduler.decode_batch_wait_us = 200;
config.scheduler.decode_priority_burst = 8;
config.scheduler.cached_prefill_policy = CachedPrefillPolicy::BackendDefault;
config.memory.reserve_percent = Some(20);
config.vision.max_pixels = Some(1_048_576);
config.vision.memory_percent = 70;
let library = Library::new(config);
drop(library);
Shared configuration
| Field | Default | Purpose |
|---|---|---|
kv_cache.block_size | 16 | Tokens per page-backed cache block. |
kv_cache.block_count | 4096 | Physical block count. |
kv_cache.dtype | Auto | Backend-resolved cache storage. |
automatic_kv_cache | true | Derive block count from model shape and available memory. |
scheduler.max_batch_requests | 16 | Maximum requests in a decode batch. |
scheduler.max_batch_tokens | 8192 | Prompt plus output token budget. |
scheduler.prefill_batch_wait_us | 200 | Prefill collection window. |
scheduler.decode_batch_wait_us | 200 | Short batching window. |
scheduler.decode_priority_burst | 8 | Decode fairness control. |
scheduler.cached_prefill_policy | BackendDefault | Admission of cached continuations during decode. |
scheduler.prefill_decode_policy | BackendDefault | When newly admitted prompts begin decoding. |
scheduler.prefill_refill_policy | Closed | Admission of short prompts into a running long prefill. |
scheduler.speculation | None | Optional Speculation { draft_tokens: NonZeroUsize } for a compatible CUDA MTP checkpoint. |
memory.reserve_percent | backend-aware | Percentage of total accelerator memory retained as headroom. |
memory.reserve_bytes | backend-aware | Absolute reserve floor used with the percentage policy. |
vision.max_pixels | checkpoint cap | Maximum resized image area. |
vision.attention_budget_bytes | automatic | Vision attention workspace. |
vision.memory_percent | 80 | Available memory used by automatic budgeting. |
Scheduling policies
The scheduling policy enums default to each backend’s normal behavior. The alternatives are explicit latency trade-offs:
| Policy | Effect |
|---|---|
CachedPrefillPolicy::InterleaveOneBlock | Admits one cached queue head with at most one K/V block of work per step during resident decode. Lowers refill first-token latency at the cost of decode latency. |
PrefillDecodePolicy::CompleteCohort | Finishes a bounded admitted cohort before publishing its first tokens. Improves decode occupancy at the cost of early first-token latency. |
PrefillRefillPolicy::ShortPrompt | Lets queued prompts of at most 128 tokens join a running long prefill at CUDA chunk boundaries. Requires a completion-first CUDA runner and cannot be combined with CompleteCohort. |
Backend-specific configuration
ModelLoadOptions::kv_blocks overrides the global K/V block count for one
model. Use it to bound a small model’s cache independently of a larger chat
model; the value is a positive NonZeroU32.
RuntimeConfig::metal and RuntimeConfig::cuda exist only when their Cargo
feature is enabled. A Metal-only consumer cannot accidentally configure CUDA, and
a backend-neutral build has neither field. See Metal and CUDA.