The AI gateway that runs where your data lives.

One name, ordered fallbacks.

Clients call a published name. Re-point it, add fallbacks or route by rule without touching a client. auto runs ordered rules (tools present, estimated tokens, message count, content match), then an optional small local classifier, then a default.

Circuit breakers and budgets per provider.

Background probes and live traffic feed circuit breakers; fallback chains skip a provider that is down or over budget. Credentials are stored encrypted and never displayed again.

Proxy · current interface, demonstration data

Every model lands where it fits.

OneVir sizes weights and KV cache against each device's free memory, then picks a GPU, a split across GPUs, a MoE hybrid with experts in system RAM, or the CPU. When nothing fits, it refuses rather than over-committing.

Gateway · local models with per-model device selection, demonstration data

Local-only in one action.

Turn external providers off and their models disappear from the model list. Local routes and local fallbacks keep serving. A single request can also opt out with a header.

Fast on the second request, too.

Concurrent requests share one decode loop. A finished request keeps its KV cache, so the next turn only processes new tokens. An identical deterministic request replays from the response cache without loading the model.

Time to first token on an RTX 4080 SUPER with Qwen2.5-0.5B-Instruct Q4_K_M, a 0.5-billion-parameter model. Larger models are slower. · response cache: exact or semantic

Search and triage from the same GGUF files.

BERT-family encoders serve embeddings, classification and reranking. Vectors come back L2-normalised, ready for any retrieval stack.

What runs locally.

Transcribe and speak.

Whisper transcribes files and live audio. Piper and Kokoro read text aloud. Audio workers run as private processes with no open port. Speech translation is available through provider deployments.

Agents can call tools on local models.

Send OpenAI tools to a local model. OneVir parses the model's tool calls out of the stream and returns standard tool_calls deltas. Tool results go back on the next turn, and a routing rule can send tool requests wherever you choose.

# example stream · local model
data: {"choices":[{"delta":{"tool_calls":[{"index":0,"type":"function",
  "function":{"name":"lookup_order","arguments":"{\"order_id\":\"4471\"}"}}]}}]}

data: {"choices":[{"delta":{},"finish_reason":"tool_calls"}]}

Prompt injection is caught before it reaches a model.

Local classifier and safety models inspect prompts, retrieved context and replies. If a detector fails, the request fails closed unless an administrator chooses otherwise.

Personal data is redacted on the way out.

Detect email addresses, phone numbers, payment cards, IBANs, US SSNs and IP addresses. Choose what happens when one is found.

A policy is tested before it goes live, and every activation is an immutable revision.

1. DraftEdit request rules, safety profiles and assignments.
2. ValidateCheck the schema and installed guard packages.
3. SimulateCompare active and proposed decisions for a request.
4. SaveStore the draft. Live traffic is unchanged.
5. ActivateNew requests get the new revision. Roll back to any of the last 20.

Governed traffic stops at once.

One action cancels governed responses in flight and returns 503 to new chat, messages and completions requests until an administrator resumes. A remote policy push cannot clear it.

• chat completion · streaming200 · streaming
• messages · streaming200 · streaming
• completion · queued202 · queued

Requests running normally

Stop governed requests

Demonstration control. It does not contact a server.

Budgets that hold before a request is sent.

Limits that hold under load.

Allowances are reserved before dispatch, so parallel calls cannot overspend.

Know where the time went.

Each request is one trace, with spans for routing, model load, queueing, prefill, generation and every provider attempt. Export stays off until you turn it on.

Scrape it like anything else.

Requests, tokens, prompt-cache reuse, queue depth, time to first token, provider health, monthly spend and kill-switch counters.

# example scrape
OneVir_active_requests 3
OneVir_loaded_models 2
OneVir_prompt_tokens_total 1284112
OneVir_prompt_tokens_cached_total 902331
OneVir_completion_tokens_total 412907
OneVir_errors_total 21
OneVir_gateway_external_providers 1
OneVir_proxy_provider_health{provider="deepseek"} 2

Watch the hardware work.

CPU, memory, GPU, disk and network for the server machine, with OneVir's own share shown separately.

Performance · current interface, dark theme, demonstration data

Usage you can explain.

Tokens, requests, cache hits, errors and spend across local and provider traffic, by model and by day. Download any view as CSV.

Activity · demonstration data, not a benchmark

Publish what applications may use.

Local and provider models share one catalogue. A model that is not published stays hidden from applications and cannot be called.

Models · catalogue with publication state, demonstration data

Keys that can be trusted with production.

• Stored only as digests, never in readable form.
• Model grants and operation scopes decide exactly what a key may call.
• Expiry, rotation and revocation, with revocation taking effect mid-stream.
• Never administrative. Application keys can never reach administration endpoints.

Credentials stay encrypted.

Saved provider and Hugging Face keys are encrypted with AES-256-GCM and are write-only in the console. On Windows, DPAPI protects the vault key.

• deepseek · sk-…739write-only
• Hugging Face token · hf_…Q2cwrite-only
• server API key · environmentmasked

An inventory you can hand to auditors.

Export every model, deployment and declared provenance as CycloneDX JSON, CSV or a printable report. Details nobody declared are marked unknown, never guessed.

AI BOM · export formats

from openai import OpenAI

client = OpenAI(base_url=ONEVIR_BASE_URL, api_key=APP_KEY)
reply = client.chat.completions.create(
    model="auto",  # routing rules decide; or a published model or alias
    messages=[{"role": "user", "content": "Summarise our refund policy in one line."}],
)
print(reply.choices[0].message.content)

ANTHROPIC_BASE_URL=$ONEVIR_URL ANTHROPIC_AUTH_TOKEN=$APP_KEY claude

From one workstation to a managed node.

Validated on Windows 11. Linux and macOS code paths and deployment files exist but are not yet certified. The replicated topology has been tested on one host only.

Data-centre throughput, as a Python service.

vLLM is built for high-throughput serving on GPU clusters and comes with a large Python dependency set. Choose it for cluster-scale serving. OneVir targets a single node you control, runs as one native server rather than a Python service, and adds the gateway, policy and provider layers that vLLM leaves to other tools.

A local runner for individual developers.

Ollama makes running a model on a laptop easy and is built on the same llama.cpp/ggml stack. OneVir is built for putting local models in front of applications and teams: application keys with model grants and limits, a published catalogue, provider routing with fallback and budgets, guard models, PII redaction, an emergency stop, Prometheus metrics and OpenTelemetry export.

llama.cpp's own HTTP server.

An excellent raw engine with a minimal management layer, typically one model per process. OneVir wraps the same engine with continuous batching across resident models, hot swap and LRU eviction, automatic placement, prompt and response caches, encoders and speech, plus the full gateway around it.

Routing and metering in front of models served elsewhere.

Gateways such as LiteLLM, Portkey, Kong AI Gateway or Cloudflare AI Gateway route and meter requests to models served elsewhere and add keys, logging and sometimes guardrails, but they do not run models themselves. OneVir runs models and routes to providers from the same process, so one policy, one cache and one audit trail cover both paths.