Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Chat completions

Use POST /v1/chat/completions with a model already loaded into the shared runtime.

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $MIRMIR_HTTP_API_KEY" \
  -d '{
    "model": "qwen",
    "messages": [{"role": "user", "content": "Hello"}],
    "temperature": 0.2,
    "max_tokens": 256
  }'
import os
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8080/v1",
    api_key=os.environ["MIRMIR_HTTP_API_KEY"],
)

reply = client.chat.completions.create(
    model="qwen",
    messages=[{"role": "user", "content": "Hello"}],
    temperature=0.2,
    max_tokens=256,
)
print(reply.choices[0].message.content)
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://127.0.0.1:8080/v1",
  apiKey: process.env.MIRMIR_HTTP_API_KEY,
});

const reply = await client.chat.completions.create({
  model: "qwen",
  messages: [{ role: "user", content: "Hello" }],
  temperature: 0.2,
  max_tokens: 256,
});
console.log(reply.choices[0].message.content);

Parameters

FieldTypeNotes
modelstringRequired loaded model selector.
messagesarrayRequired non-empty chat history. Assistant tool_calls arguments may be a JSON object or a JSON-encoded string.
streambooleanReturns server-sent events when true.
max_tokensintegerTotal generated-token limit, including reasoning and final content.
max_completion_tokensintegerAlias; must agree with max_tokens if both appear.
min_tokensintegerSuppress end-of-sequence before this many tokens.
ignore_eosbooleanContinue past end-of-sequence until max_tokens.
temperaturenumberSampling temperature.
top_pnumberNucleus cutoff from 0 to 1.
top_kintegerCandidate count; zero leaves it unbounded.
repetition_penaltynumber1 disables the penalty.
seedintegerReproducible sampler seed.
nintegerOnly 1 is supported.
toolsarrayOpenAI function tool declarations.
tool_choicestring or objectauto (default), none, required, or {"type":"function","function":{"name":…}}.
tool_constraintsstringMiRMiR extension: none (default) or schema. See Tool calling.
response_formatobjecttext, json_object, or strict json_schema. See Structured JSON output.
reasoningstringMiRMiR extension: model_default (default), enabled, or disabled.
chat_template_kwargs.enable_thinkingbooleanvLLM-compatible alias for reasoning; must agree with it when both appear.
chat_template_kwargs.medium_effortbooleanUses shorter reasoning when true and the template declares this control.
chat_template_kwargs.truncate_history_thinkingbooleanKeeps or drops earlier reasoning when the template declares this control.
thinking_token_budgetintegerReasoning allowance inside the total limit for schema tools or JSON content.
return_token_idsbooleanStreaming only: adds generated token ids to each chunk.
stream_options.include_usagebooleanAdds usage to the final SSE chunk.

MiRMiR rejects controls it cannot honor instead of silently ignoring them. A non-zero presence_penalty or frequency_penalty and any non-null reasoning_budget or reasoning_effort return 400. Unknown chat_template_kwargs keys are rejected. Explicit effort/history controls are validated against the selected template; they are not universal switches for every model.

Streaming

curl -N http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

Each event contains a chat.completion.chunk. The stream ends with data: [DONE]. Reasoning-capable models emit delta.reasoning_content separately from delta.content. Tool calls arrive as one delta.tool_calls chunk before the final chunk, whose finish_reason is tool_calls. With return_token_ids, each chunk carries choices[0].token_ids, and the final chunk carries any remaining ids.

Closing the connection cancels the generation on the server.

Reasoning

reasoning selects the thinking switch for one text request without changing other requests:

{"model": "qwen", "messages": [{"role": "user", "content": "Return 2+2."}],
 "reasoning": "disabled", "max_completion_tokens": 256}

Explicit enabled and disabled modes require a model chat template that reads enable_thinking, or a supported built-in Qwen or Gemma template. Other templates and image requests reject explicit modes. model_default preserves the template’s own behavior. The max_tokens limit covers reasoning and final content together; reaching it does not guarantee a final answer.

Tool calling

Declare OpenAI function tools and select how the model may use them:

{
  "model": "qwen",
  "messages": [{"role": "user", "content": "What's the weather in Warsaw?"}],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "parameters": {
        "type": "object",
        "properties": {"city": {"type": "string"}},
        "required": ["city"]
      }
    }
  }],
  "tool_choice": "auto"
}

Native Qwen XML (<tool_call><function=…>) and Mistral [TOOL_CALLS] output is normalized into OpenAI message.tool_calls with finish_reason: "tool_calls". MiRMiR validates every call against the request and returns 422 invalid_model_output when the model omits a call that required or a named function demands, calls a tool while tool_choice is none, calls a different or undeclared tool, or emits malformed or truncated arguments.

Schema-constrained tools

On Metal and CUDA, "tool_constraints": "schema" constrains decoding to the named tool’s parameter schema, so the model can only produce a well-formed call. It requires a named tool_choice, repetition_penalty: 1, min_tokens: 0, and ignore_eos: false; other combinations return 400.

  • Root schemas must be objects with declared properties, at most 64 top-level properties and 512 KB. Remote and file $ref retrieval is disabled.
  • Types, required fields, enums, and array cardinality are enforced while decoding; uniqueItems is checked on the completed call (422 on violation). Parameter order follows the declaration.
  • With reasoning: "enabled", reasoning stays unconstrained until the model’s </think> marker, then the tool grammar takes over. thinking_token_budget caps reasoning inside max_tokens and must leave at least two tokens for the delimiter and tool. When omitted, MiRMiR reserves a quarter of max_tokens (at least two tokens) for the tool.
  • A limit that truncates the tool call still returns 422.

One image

The OpenAI-style image_url content part accepts a data URL. Version 0.6.2 supports one PNG, JPEG, WebP, or GIF image per request; an animated GIF contributes its first frame. Image requests cannot set an explicit reasoning mode, effort or history; leave chat_template_kwargs.enable_thinking unset.

IMAGE_DATA=$(base64 < image.jpg | tr -d '\n')
curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"vision-model\",\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/jpeg;base64,$IMAGE_DATA\"}},{\"type\":\"text\",\"text\":\"Describe this image\"}]}]}"

The application limit is 20 MiB per decoded image. The server body limit must also accommodate the larger base64 request.