Generate and stream

Model::generate is the preferred high-level text API. It renders the checkpoint chat template, tokenizes, creates an independent session, performs prefill, samples and decodes tokens, normalizes model-specific channels, and returns metrics.

Content and reasoning

The token callback receives GenerationToken values with a semantic channel. Native think markers and channel headers are removed from the public contract. The final GenerationOutput contains separate text and reasoning fields.

let output = model.generate(
    &request,
    &mut |progress| eprintln!("{progress:?}"),
    &mut |token| println!("{:?}: {}", token.channel, token.text),
)?;

println!("answer={}", output.text);
println!("reasoning={}", output.reasoning);
println!("metrics={:#?}", output.metrics);
Ok::<(), libmir::Error>(())

Cooperative cancellation

CancellationToken is thread-safe and all clones observe the same state. The generation loop checks it before prefill, after prefill, and between decode steps.

use libmir::CancellationToken;

let cancellation = CancellationToken::new();
let signal = cancellation.clone();

std::thread::spawn(move || {
    std::thread::sleep(std::time::Duration::from_secs(2));
    signal.cancel();
});

let result = model.generate_cancellable(
    &request,
    &mut |_| {},
    &mut |token| print!("{}", token.text),
    &cancellation,
);
drop(result);

Cancellation is cooperative. It does not interrupt an individual accelerator kernel in the middle of execution.

One image

Insert IMAGE_PLACEHOLDER exactly once in the message content and provide an encoded PNG, JPEG, WebP, or GIF:

use libmir::{CancellationToken, IMAGE_PLACEHOLDER};

let mut request = request;
request.messages[0].content = format!("{IMAGE_PLACEHOLDER}\nDescribe this image.");
let image = std::fs::read("image.jpg")?;

let output = model.generate_image_cancellable(
    &request,
    &image,
    &mut |_| {},
    &mut |token| print!("{}", token.text),
    &CancellationToken::new(),
)?;
drop(output);
Ok::<(), Box<dyn std::error::Error>>(())

The checkpoint must expose a complete supported vision contract. Resource limits come from RuntimeConfig::vision.