Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Generate and stream

Model::generate is the preferred high-level text API. It renders the checkpoint chat template, tokenizes, creates an independent session, performs prefill, samples and decodes tokens, normalizes model-specific channels, and returns metrics.

Content and reasoning

The token callback receives GenerationToken values with a semantic channel. Native think markers and channel headers are removed from the public contract. Each token carries its id and, in preceding_ids, any generated identifiers that were folded into it without producing visible text of their own.

The final GenerationOutput contains separate text, reasoning, and tool_calls fields, every generated token_ids entry (including a terminal stop token when emitted), prompt_tokens, a finish_reason such as "stop", "tool_calls", or "max_tokens", and metrics.

let output = model.generate(
    &request,
    &mut |progress| eprintln!("{}: {}", progress.stage(), progress.detail()),
    &mut |token| println!("{:?}: {}", token.channel, token.text),
)?;

println!("answer={}", output.text);
println!("reasoning={}", output.reasoning);
println!("metrics={:#?}", output.metrics);
Ok::<(), libmir::Error>(())

ProgressEvent is an enum with one variant per stage: LoadWeights, InitializeRuntime, Warmup, PrefillTokens, and DecodeTokens. Use stage(), unit() (Byte, Item, or Token), count() for the current()/total() pair, and detail() rather than parsing display text.

Reasoning controls

GenerationRequest::reasoning selects the template’s thinking switch: ReasoningMode::ModelDefault (the default) preserves the model’s rendering, while Enabled and Disabled require a Jinja enable_thinking input or a supported built-in Qwen or Gemma template. Unsupported templates and image requests reject explicit modes.

GenerationOverrides::max_tokens counts every generated token, including reasoning. GenerationRequest::reasoning_cycle decides what happens when a reasoning loop is detected:

PolicyBehavior
ReasoningCyclePolicy::ExitReasoning { min_tokens: 64 }Default. Closes a supported reasoning channel after three consecutive repeated token cycles past the token gate.
ReasoningCyclePolicy::RecoverResumes greedy decoding with a repetition penalty.

Scattered repetitions, such as quoting the same source twice, do not force an exit. The completion limit still bounds reasoning that never reaches an answer.

Tool calls

Declare functions in Conversation::tools and select a ToolChoice: Auto (the default), None, Required, or Function(name).

use libmir::{FunctionDefinition, Tool, ToolChoice};

request.conversation.tools = vec![Tool {
    kind: "function".into(),
    function: FunctionDefinition {
        name: "weather".into(),
        description: Some("Current weather for a city".into()),
        parameters: serde_json::json!({
            "type": "object",
            "properties": { "city": { "type": "string" } },
            "required": ["city"]
        }),
    },
}];
request.conversation.tool_choice = ToolChoice::Function("weather".into());

Native JSON (Mistral-style) and Qwen XML tool calls are parsed and normalized into GenerationOutput::tool_calls as a JSON array of {id, type, function: {name, arguments}} objects, and finish_reason becomes "tool_calls". Malformed output, an undeclared tool, a call that contradicts tool_choice, or a missing required call returns Error::InvalidToolCall. Feed prior calls back through Message::tool_calls and results through Message::tool_call_id.

Schema-constrained tools

Set GenerationRequest::tool_constraints to ToolConstraints::Schema to mask sampling with the named tool’s JSON Schema. The mode is opt-in and currently requires:

  • the Metal or CUDA backend with device sampling;
  • ToolChoice::Function(name) naming a declared tool;
  • a native XML <tool_call>/<function=...> template, such as Qwen’s;
  • repetition_penalty = 1, min_tokens = 0, and ignore_eos = false.

Schemas the grammar compiler cannot express are rejected before generation. Completed arguments are validated against the original schema, including uniqueItems, and violations return Error::InvalidToolCall.

With reasoning enabled, the model may think before the constrained call. GenerationRequest::reasoning_token_budget optionally caps that reasoning; at the cap only the native reasoning delimiter can be sampled. The budget requires ReasoningMode::Enabled and ToolConstraints::Schema, and max_tokens must leave room for the delimiter and the tool output.

JSON content

Set request.response_format = libmir::ResponseFormat::JsonSchema(schema) to constrain final content instead of a tool call. Use {"type":"object"} for an arbitrary JSON object. The default is ResponseFormat::Text.

The Metal/CUDA sampling restrictions are the same as schema tools. JSON content rejects ToolConstraints::Schema and required/named ToolChoice. Native reasoning remains separate; markers inside JSON strings stay literal. The complete content is checked against the original schema. A partial token callback is not a validated document. Invalid final content returns Error::InvalidResponseFormat, mapped to invalid_model_output by the application.

reasoning_token_budget also supports JSON content with enabled reasoning and compatible atomic reasoning delimiters. Direct Rust callers set this allowance explicitly; the HTTP adapter’s default quarter-budget reservation is not a library default.

Cooperative cancellation

CancellationToken is thread-safe and all clones observe the same state. Text prefill observes it at cache waits and scheduler boundaries, and decode checks it between steps.

use libmir::CancellationToken;

let cancellation = CancellationToken::new();
let signal = cancellation.clone();

std::thread::spawn(move || {
    std::thread::sleep(std::time::Duration::from_secs(2));
    signal.cancel();
});

let result = model.generate_cancellable(
    &request,
    &mut |_| {},
    &mut |token| print!("{}", token.text),
    &cancellation,
);
drop(result);

Cancellation is cooperative and returns Error::Cancelled. It does not interrupt an individual accelerator kernel in the middle of execution.

One image

Insert IMAGE_PLACEHOLDER exactly once in the message content and provide an encoded PNG, JPEG, WebP, or GIF:

use libmir::{CancellationToken, IMAGE_PLACEHOLDER};

let mut request = request;
request.conversation.messages[0].content =
    format!("{IMAGE_PLACEHOLDER}\nDescribe this image.");
let image = std::fs::read("image.jpg")?;

let output = model.generate_image_cancellable(
    &request,
    &image,
    &mut |_| {},
    &mut |token| print!("{}", token.text),
    &CancellationToken::new(),
)?;
drop(output);
Ok::<(), Box<dyn std::error::Error>>(())

The checkpoint must expose a complete supported vision contract. Resource limits come from RuntimeConfig::vision. Image prefill observes cancellation before and after the complete prefill, then between decode steps.