# Infer Runtime

An AI inference runtime for local applications that schedules resources from an Intent and its placement, privacy, latency, and fallback constraints.

## Metadata

- HTML: https://glenzli.com/en/projects/infer-runtime/
- Markdown: https://glenzli.com/en/projects/infer-runtime.md
- Collection: Projects
- Language: en
- Published: 2026-08-14
- Updated: 2026-08-27
- Status: active
- Tags: inference, control-plane, local-first, multimodal, consumer-contract

## Content

`infer-runtime` is an AI inference runtime for local applications. A Consumer submits an Intent with capability, latency, placement, privacy, and fallback constraints. Runtime selects a Provider and Deployment, then manages queueing, quotas, model residency, cancellation, failover, and execution records.

It unifies the control plane while text, audio, and vision retain separate typed protocols.

  ![An application submits an inference intent and constraints for Runtime to schedule](/images/projects/infer-runtime-banner.webp)

## Current implementation

    **Text**
    Responses-style unary requests and SSE, encrypted local background work, and local, cloud, and subscription Providers. The subscription bridge remains experimental.

    **Audio**
    Transcription, forced alignment, speech synthesis, sound generation, voice cloning, event detection, and audio-text retrieval. Streaming and some generation and retrieval paths remain experimental.

    **Vision**
    Narrow ONNX and Core ML capabilities for faces, image-text embeddings, subject segmentation, and face parsing. Some model weights are restricted to research use.

    **Scheduling and resources**
    Intent routing, priority queues, deadlines, cancellation, retry and fallback, circuit breaking, quotas, and Ollama/ONNX resource management.

  ![Infer Console showing runtime, job, and resource state with synthetic data](/images/projects/infer-runtime-overview.webp)
  The Console is real; all Providers, Deployments, metrics, and instance identities shown here are synthetic.

## Consumer integration

Each application uses its own identity and ACL. Intents are separate from physical model names, and Runtime chooses a Provider, Build, and Deployment within the request constraints. `local_only`, offline mode, modality-export permissions, and fallback grants are not relaxed silently.

Local Consumers can discover the current `infer-runtime.consumer-core` endpoint through Infra Discovery. The `infer-runtime-client` library handles discovery, generation changes, credentials, contract headers, and stable error parsing.

  ![The Models and Resources view in Infer Console](/images/projects/infer-runtime-models.webp)

## Status and boundaries

The project is a development preview. Core scheduling, local-resource management, access control, and the Console form a working path; several audio, vision, and subscription capabilities remain experimental. Automatic eviction is disabled by default, and probes that may cost money or change model residency require an explicit Operator action.

Providers, Deployments, and model files are configured by each installation. Runtime covers inference-resource access, selection, and scheduling.