Projects

Infer Runtime

Published: 2026-08-14Last edited: 2026-08-27active

inferencecontrol-planelocal-firstmultimodalconsumer-contract

infer-runtime is an AI inference runtime for local applications. A Consumer submits an Intent with capability, latency, placement, privacy, and fallback constraints. Runtime selects a Provider and Deployment, then manages queueing, quotas, model residency, cancellation, failover, and execution records.

It unifies the control plane while text, audio, and vision retain separate typed protocols.

An application submits an inference intent and constraints for Runtime to schedule

Current implementation

TextResponses-style unary requests and SSE, encrypted local background work, and local, cloud, and subscription Providers. The subscription bridge remains experimental.

AudioTranscription, forced alignment, speech synthesis, sound generation, voice cloning, event detection, and audio-text retrieval. Streaming and some generation and retrieval paths remain experimental.

VisionNarrow ONNX and Core ML capabilities for faces, image-text embeddings, subject segmentation, and face parsing. Some model weights are restricted to research use.

Scheduling and resourcesIntent routing, priority queues, deadlines, cancellation, retry and fallback, circuit breaking, quotas, and Ollama/ONNX resource management.

Infer Console showing runtime, job, and resource state with synthetic data — The Console is real; all Providers, Deployments, metrics, and instance identities shown here are synthetic.

Consumer integration

Each application uses its own identity and ACL. Intents are separate from physical model names, and Runtime chooses a Provider, Build, and Deployment within the request constraints. local_only, offline mode, modality-export permissions, and fallback grants are not relaxed silently.

Local Consumers can discover the current infer-runtime.consumer-core endpoint through Infra Discovery. The infer-runtime-client library handles discovery, generation changes, credentials, contract headers, and stable error parsing.

The Models and Resources view in Infer Console

Status and boundaries

The project is a development preview. Core scheduling, local-resource management, access control, and the Console form a working path; several audio, vision, and subscription capabilities remain experimental. Automatic eviction is disabled by default, and probes that may cost money or change model residency require an explicit Operator action.

Providers, Deployments, and model files are configured by each installation. Runtime covers inference-resource access, selection, and scheduling.

On this pageCurrent implementation Consumer integration Status and boundaries