Generate and stream
Model::generate is the preferred high-level text API. It renders the
checkpoint chat template, tokenizes, creates an independent session, performs
prefill, samples and decodes tokens, normalizes model-specific channels, and
returns metrics.
Content and reasoning
The token callback receives GenerationToken values with a semantic channel.
Native think markers and channel headers are removed from the public contract.
Each token carries its id and, in preceding_ids, any generated identifiers
that were folded into it without producing visible text of their own.
The final GenerationOutput contains separate text, reasoning, and
tool_calls fields, every generated token_ids entry (including a terminal
stop token when emitted), prompt_tokens, a finish_reason such as "stop",
"tool_calls", or "max_tokens", and metrics.
let output = model.generate(
&request,
&mut |progress| eprintln!("{}: {}", progress.stage(), progress.detail()),
&mut |token| println!("{:?}: {}", token.channel, token.text),
)?;
println!("answer={}", output.text);
println!("reasoning={}", output.reasoning);
println!("metrics={:#?}", output.metrics);
Ok::<(), libmir::Error>(())
ProgressEvent is an enum with one variant per stage: LoadWeights,
InitializeRuntime, Warmup, PrefillTokens, and DecodeTokens. Use
stage(), unit() (Byte, Item, or Token), count() for the
current()/total() pair, and detail() rather than parsing display text.
Reasoning controls
GenerationRequest::reasoning selects the template’s thinking switch:
ReasoningMode::ModelDefault (the default) preserves the model’s rendering,
while Enabled and Disabled require a Jinja enable_thinking input or a
supported built-in Qwen or Gemma template. Unsupported templates and image
requests reject explicit modes.
GenerationOverrides::max_tokens counts every generated token, including
reasoning. GenerationRequest::reasoning_cycle decides what happens when a
reasoning loop is detected:
| Policy | Behavior |
|---|---|
ReasoningCyclePolicy::ExitReasoning { min_tokens: 64 } | Default. Closes a supported reasoning channel after three consecutive repeated token cycles past the token gate. |
ReasoningCyclePolicy::Recover | Resumes greedy decoding with a repetition penalty. |
Scattered repetitions, such as quoting the same source twice, do not force an exit. The completion limit still bounds reasoning that never reaches an answer.
Tool calls
Declare functions in Conversation::tools and select a ToolChoice: Auto
(the default), None, Required, or Function(name).
use libmir::{FunctionDefinition, Tool, ToolChoice};
request.conversation.tools = vec![Tool {
kind: "function".into(),
function: FunctionDefinition {
name: "weather".into(),
description: Some("Current weather for a city".into()),
parameters: serde_json::json!({
"type": "object",
"properties": { "city": { "type": "string" } },
"required": ["city"]
}),
},
}];
request.conversation.tool_choice = ToolChoice::Function("weather".into());
Native JSON (Mistral-style) and Qwen XML tool calls are parsed and normalized
into GenerationOutput::tool_calls as a JSON array of
{id, type, function: {name, arguments}} objects, and
finish_reason becomes "tool_calls". Malformed output, an undeclared tool, a
call that contradicts tool_choice, or a missing required call returns
Error::InvalidToolCall. Feed prior calls back through Message::tool_calls
and results through Message::tool_call_id.
Schema-constrained tools
Set GenerationRequest::tool_constraints to ToolConstraints::Schema to mask
sampling with the named tool’s JSON Schema. The mode is opt-in and currently
requires:
- the Metal or CUDA backend with device sampling;
ToolChoice::Function(name)naming a declared tool;- a native XML
<tool_call>/<function=...>template, such as Qwen’s; repetition_penalty = 1,min_tokens = 0, andignore_eos = false.
Schemas the grammar compiler cannot express are rejected before generation.
Completed arguments are validated against the original schema, including
uniqueItems, and violations return Error::InvalidToolCall.
With reasoning enabled, the model may think before the constrained call.
GenerationRequest::reasoning_token_budget optionally caps that reasoning; at
the cap only the native reasoning delimiter can be sampled. The budget requires
ReasoningMode::Enabled and ToolConstraints::Schema, and max_tokens must
leave room for the delimiter and the tool output.
JSON content
Set request.response_format = libmir::ResponseFormat::JsonSchema(schema)
to constrain final content instead of a tool call. Use {"type":"object"}
for an arbitrary JSON object. The default is ResponseFormat::Text.
The Metal/CUDA sampling restrictions are the same as schema tools. JSON
content rejects ToolConstraints::Schema and required/named ToolChoice.
Native reasoning remains separate; markers inside JSON strings stay literal.
The complete content is checked against the original schema. A partial token
callback is not a validated document. Invalid final content returns
Error::InvalidResponseFormat, mapped to invalid_model_output by the application.
reasoning_token_budget also supports JSON content with enabled reasoning and
compatible atomic reasoning delimiters. Direct Rust callers set this allowance
explicitly; the HTTP adapter’s default quarter-budget reservation is not a
library default.
Cooperative cancellation
CancellationToken is thread-safe and all clones observe the same state. Text
prefill observes it at cache waits and scheduler boundaries, and decode checks it
between steps.
use libmir::CancellationToken;
let cancellation = CancellationToken::new();
let signal = cancellation.clone();
std::thread::spawn(move || {
std::thread::sleep(std::time::Duration::from_secs(2));
signal.cancel();
});
let result = model.generate_cancellable(
&request,
&mut |_| {},
&mut |token| print!("{}", token.text),
&cancellation,
);
drop(result);
Cancellation is cooperative and returns Error::Cancelled. It does not
interrupt an individual accelerator kernel in the middle of execution.
One image
Insert IMAGE_PLACEHOLDER exactly once in the message content and provide an
encoded PNG, JPEG, WebP, or GIF:
use libmir::{CancellationToken, IMAGE_PLACEHOLDER};
let mut request = request;
request.conversation.messages[0].content =
format!("{IMAGE_PLACEHOLDER}\nDescribe this image.");
let image = std::fs::read("image.jpg")?;
let output = model.generate_image_cancellable(
&request,
&image,
&mut |_| {},
&mut |token| print!("{}", token.text),
&CancellationToken::new(),
)?;
drop(output);
Ok::<(), Box<dyn std::error::Error>>(())
The checkpoint must expose a complete supported vision contract. Resource
limits come from RuntimeConfig::vision. Image prefill observes cancellation
before and after the complete prefill, then between decode steps.