Sessions and K/V cache

Use Model::generate unless an application needs direct control over the token loop. The low-level Session API owns independent K/V state while sharing the loaded model and its cache arena.

use libmir::{SamplingLogits, runtime::RuntimeError};

let prepared = model.prepare(&request)?;
let mut session = model.session();
let prefill = session.prefill(
    &prepared.tokens.token_ids,
    SamplingLogits::None,
    &mut |_| {},
)?;
let mut next = prefill.next_token.ok_or_else(|| {
    RuntimeError::Backend("device sampling returned no token".into())
})?;

for _ in 0..32 {
    print!("{}", model.descriptor().tokenizer().decode(&[next])?);
    if model.descriptor().tokenizer().stop_token_ids().contains(&next) {
        break;
    }
    next = session
        .decode(next, SamplingLogits::None)?
        .event
        .token_id
        .ok_or_else(|| RuntimeError::Backend("no token".into()))?;
}

println!("{:#?}", session.cache_stats());
Ok::<(), libmir::Error>(())

SamplingLogits::None requests device sampling when supported. Other variants can expose full or candidate logits for an application-owned sampler.

Cache behavior

The library uses page-backed K/V storage. Full ready blocks can become shared prefix entries, and sessions use copy-on-write when a shared tail must diverge. Session::cache_stats reports:

• resolved storage dtype and quantization mode;
• total, free, and used blocks;
• cached prefix count;
• prefix probes, hits, misses, and matching token counts.

Dropping the session releases its block table. A multimodal prefill deliberately does not publish reusable prefix-cache entries.