Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Metal, CUDA, and CPU

The high-level Library, Model, Session, generation, embedding, and reranking APIs are backend-neutral. Cargo features choose which native adapter is compiled.

The facade does not re-export backend objects or backend configuration types. Enable a feature, then adjust the defaults of RuntimeConfig::metal or RuntimeConfig::cuda in place. Library creates the backend lazily, and backend failures surface as libmir::Error.

Metal

libmir = { version = "0.6.2", default-features = false, features = ["metal"] }

Metal uses mirtal for explicit streams, device memory, compiled kernels, graphs, and model-neutral operations. RuntimeConfig::metal includes batch, cache, fusion, tuning, and diagnostic policies.

use libmir::{Library, MetalDecodeReservation, RuntimeConfig};

let mut config = RuntimeConfig::default();
config.metal.cache.decode_reservation = MetalDecodeReservation::GenerationBudget;
config.metal.tuning.cache_directory = Some("/var/cache/my-app/tuning".into());
let library = Library::new(config);
drop(library);

MetalDecodeReservation::OnePage is the default. GenerationBudget instead reserves up to the request’s generation budget from reclaimable spare K/V pages.

On Apple Silicon, host and accelerator share unified memory. Memory snapshots distinguish active allocations from reusable cached allocations.

CUDA

libmir = { version = "0.6.2", default-features = false, features = ["cuda"] }

CUDA uses mircuda for contexts, streams, memory, NVRTC compilation, kernels, graphs, and CUTLASS-backed plans. RuntimeConfig::cuda selects the device, pool behavior, include paths, persistent cubin cache, startup tuning, and execution-planning policy.

use libmir::{Library, RuntimeConfig};

let mut config = RuntimeConfig::default();
config.cuda.device_ordinal = 0;
config.cuda.nvrtc_cache_directory = Some("/var/cache/my-app/cubins".into());
config.cuda.tuning.cache_directory = Some("/var/cache/my-app/tuning".into());

let library = Library::new(config);
println!("backend={:?}", library.backend_target()?);
let memory = library.memory_snapshot()?;
println!("{}: free={:?} total={:?}", memory.source, memory.available_bytes, memory.total_bytes);
Ok::<(), libmir::Error>(())

The first call that needs the backend creates real device resources, so it is fallible. CUDA memory reporting includes driver-visible free and total device memory plus pool allocations.

The cuda-diagnostics feature implies cuda and compiles additional kernel diagnostics, such as activation fingerprints. Leave it disabled in production.

CPU decisions

Enable cpu to use DecisionModel with the native mircup backend. This provides Laya decisions independently of the generation scheduler; it does not make Library a general CPU chat, embedding, or reranking engine. See Typed decisions.

Capability differences

Version 0.6.2 does not imply identical realization for every encoding on both backends. A format may use direct execution on one platform and a bounded, persistent accelerator-side conversion on the other. Schema-constrained tool generation runs on both: CUDA applies the grammar mask in a fused sampling kernel, while Metal masks the step’s full device logits and reads back only the selected token, so constrained Metal steps skip the pipelined device-token lookahead. Admission comes from the discovered execution contract and backend capability, not a family-name allowlist. Inspect the exact checkpoint on the target machine before promising support.

See Model compatibility for the release matrix.

Nemotron-H’s state-space generation and MTP run on CUDA only. Laya decisions run on all three backends through the Rust API. The HTTP decision endpoint has a narrower CPU/Metal contract. Metal builds target native MLX 0.32.4.