Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Inspect and load models

Inspection is separate from accelerator allocation. Use it to reject an unsupported checkpoint or present its capabilities before committing device memory.

use libmir::{GenerationOverrides, ModelDescriptor};

let descriptor = ModelDescriptor::inspect(
    "/models/qwen",
    GenerationOverrides::default(),
)?;
let manifest = descriptor.manifest()?;

println!("id={}", manifest.id);
println!("context={}", manifest.context_len);
println!("task={:?}", descriptor.task());
println!("quantization={:?}", manifest.quantization);
println!("generation={:?}", descriptor.generation());
Ok::<(), libmir::Error>(())

Discovery is based on configuration values, tokenizer assets, tensor names and shapes, and the resulting execution contract. A recognizable repository or architecture name alone does not admit a checkpoint.

Estimate memory

use libmir::{GenerationOverrides, ModelDescriptor, RuntimeConfig};

let descriptor = ModelDescriptor::inspect(
    "/models/qwen",
    GenerationOverrides::default(),
)?;
let estimate = descriptor.memory_estimate(&RuntimeConfig::default());

println!("weights={} bytes", estimate.weight_bytes);
println!("kv={} bytes", estimate.kv_cache_bytes);
println!("workspace={} bytes", estimate.workspace_bytes);
println!("required={} bytes", estimate.required_bytes);
Ok::<(), libmir::Error>(())

This is a conservative planning value, not a reservation. Compare it with Library::memory_snapshot() on the backend that will load the model.

Load and unload

use libmir::{GenerationOverrides, Library, RuntimeConfig};

let library = Library::new(RuntimeConfig::default());
let model = library.load(
    "/models/qwen",
    GenerationOverrides::default(),
    &mut |event| {
        let count = event.count();
        eprintln!("{} {}/{} {}", event.stage(), count.current(), count.total(), event.unit());
    },
)?;

println!("backend={:?}", model.engine().target());
model.unload()?;
Ok::<(), libmir::Error>(())

Model is cheaply cloneable. Model::unload succeeds only when the value is the sole remaining owner; active clones or sessions produce Error::ModelInUse. Dropping a Session releases its backend and K/V resources automatically.

Several resident models

Library::load admits a model only when its estimate fits the accelerator budget, counting K/V memory that already loaded models could give back; otherwise it returns Error::MemoryAdmission. Library::load_with_options with ModelLoadOptions { allow_memory_overcommit: true } skips that check, although backend allocation may still fail. Library::reclaimable_kv_bytes reports that amount. After loading or unloading, Library::rebalance_kv_caches resizes idle models’ measured K/V caches to the memory the others leave and returns how many changed.