Deployments overview

A deployment turns an optimized pipeline into a live HTTP endpoint. You get a URL, a scoped API key, autoscaling replicas, Instant Start cold-start caching, and supported OpenAI-compatible routes where the deployment exposes them.

Deployment unlocks once your lifetime credit purchases reach $50. There is no plan to buy: add credits at Settings > Cost and the Deploy tab opens. Before that you can still build, optimize, call the Model APIs, and test in the playground. See Pricing and credits.

Request lifecycle

Every request hits the edge, authenticates against your API key, routes to a replica with headroom, and streams tokens back. Cold starts pull from Instant Start; warm requests skip it.

Where it runs

Four deployment targets, picked when you click Deploy:

Target	When to pick it
Managed RunInfra Cloud (default)	Scale-to-zero with Instant Start. Fastest path; RunInfra runs the infra.
Your RunPod (bring your own cloud)	Delivery to your RunPod account after RunInfra Modal measurement evidence exists.
Self-hosted Modal	Reserved GPUs on your Modal account, billed to you.
Custom GPU	Bare metal, on-prem, regulated industries, existing GPU contracts.

See Deployment targets for the full picking guide.

Deploy

From chat
From the Deploy tab

Deploy this pipeline

The agent picks the winning variant, provisions a GPU, and returns the endpoint URL and key.

RunInfra uses your intake choices to order the deployment paths. If you chose a local machine, edge device, own server, or on-premises target, Your hardware appears first and shows the captured device memory budget. Local and edge exports default to Local GPU. Own-server and on-premises exports default to Container. Cloud-focused intake keeps managed cloud first. Your manual target choice remains selected after the session finishes loading.

Optimized artifact deploys

When you deploy an optimized variant, RunInfra Cloud uses the deployable artifact from the selected pipeline version. The deploy request carries the pipeline files, model revision, serving backend, and optimization settings that produced the winning variant. Managed deployments reject missing or stale artifact snapshots before provisioning. This prevents a deployment from silently falling back to a generic base image and losing the optimization that you selected. Each custom artifact build returns a deployment receipt with:

Field	Meaning
`buildSource`	Shows whether the image came from a custom artifact build.
`artifactSha256`	Stable identity for the pipeline files used as build input.
`contentHash`	Image build content hash used for cache and audit checks.
`imageTag`	Container image tag deployed to the serving backend.
`servingBackend`	Backend selected for the deployed model.

The Deploy tab refreshes the live capability contract before provisioning. If a backend or modality is not currently enabled, deployment stops before creating GPU resources and shows the required action.

Modality verification and API keys

Managed deployments verify through a modality-native gateway canary before they go active. Voice pipeline, speech-to-text, text-to-speech, audio-language, and vision-language deployments verify this way and can mint working API keys immediately after the deployment verifies; previously that immediate key flow covered LLM and embedding deployments only. Minted keys pair with the flat base URL https://api.runinfra.ai/v1. Image generation, rerank, and classification managed deployments remain unavailable. Rerank and classification are delivered through deployment kits instead; see Deployment kits.

Deployment kits

The Deploy tab packages the same selected pipeline version that deployment uses. Its Deployment kit area centers on one optimized pipeline package: build it, download it, run the setup guide, and verify before serving. Downloads include these files:

File	Purpose
`README.md`	Orientation and quick start.
`FETCH_MODEL.sh`	Downloads the optimized weights into `./model` from a signed, time-limited URL.
`runinfra/manifest.json`	Provenance, runtime, GPU, optimization, proof level, and portability contract.
`runinfra/verify.sh`	Runs the bundle’s canary verification against the manifest.
`runinfra/Dockerfile`	Builds an image pinned to the runtime used by the selected artifact.
`runinfra/serve.sh`	Starts the selected runtime against `./model`.
`runinfra/docker-compose.yml`	Local validation with the same runtime image and model mount.
`runinfra/benchmarks.json`	Benchmark projection from the manifest.
`runinfra/eval.json`	Quality projection from the manifest.
`runinfra/canary_inputs/sample.json`	Modality-aware sample request for verification.
`runinfra/checksums.txt`	SHA-256 checksums for the generated bundle files.

Runtime files are generated for the selected backend. vLLM and SGLang launch their Python serving modules, vLLM Omni pins the audio runtime, TensorRT-LLM uses Triton Server, and Diffusers or TEI preserve the base image entrypoint and required environment. Rerank and classification kits use the explicit tei export format: the generated serving setup runs text-embeddings-inference, and the kit’s serve script answers the kit’s own documented endpoints, /rerank for rerank kits and /predict for classification kits. The manifest exposes a proof_level so downstream teams can see how much measured evidence backs the export:

Proof level	Meaning
`benchmarked_and_quality_gated`	Benchmarks and a quality gate are present for the exported artifact.
`benchmarked_only`	Benchmarks are present, but no quality gate was recorded for this export.
`uncharacterized`	The export is usable, but benchmark or quality evidence is missing.

When proof is incomplete, the Deploy tab and generated manifest show the degraded proof level instead of filling in confident numbers. Missing benchmark, quality, library-version, or out-of-envelope data is rendered as uncharacterized. Export job status preserves the selected optimizationVersionId and runbook executionId. Use those ids to confirm the downloaded bundle or Hugging Face push belongs to the accepted optimization run before treating it as production evidence. The signed weight URL in FETCH_MODEL.sh expires. If the link is stale, rebuild the bundle from the Deploy tab to get a fresh URL. Before building a deployment kit, RunInfra scans customer-authored and edited files for sensitive values. Files that exactly match the current generated Code artifact pass without being misclassified because they contain safe placeholders, environment variable names, or setup instructions. If the generated snapshot is stale, the Deploy tab asks you to sync or regenerate the Code artifact. If a customer-added or edited file contains a real secret, export still stops and identifies the affected file paths without exposing the secret value.

The kit pins the weights the run measured

A model repository name is a moving target. Its default branch advances and tags can be re-pointed, so a kit that names only the repository loads whatever that repository holds on the day you run it, while the kit’s own report quotes numbers measured on some earlier state. When the run resolved the exact weights revision it measured, every generated launch surface pins it. The serve script, the container build, the Compose file, the Kubernetes manifests, and the managed-cloud targets all carry the revision through an HF_REVISION setting, passed to each serving backend with that backend’s own revision flag. The revision that gets pinned is the commit the run actually loaded, recorded by the worker, not merely the reference the run asked for. The applied-optimization report states it plainly, pinned to commit <sha>, the weights this run measured, and OPTIMIZATION.lock records weights_pin as commit with weights_reproducible true. When the run did not resolve an immutable commit, the kit says so rather than leaving a missing field to be read as reproducibility. Every buyer-facing surface, README.md, OPTIMIZATIONS.md, APPLIED-OPTIMIZATION.md, and optimization-report.md, carries this banner:

UNVERIFIED: this run recorded no weights revision, so the kit loads whatever
the model repository's default branch holds on the day you run it. Those may
not be the weights the reported numbers were measured on. Set HF_REVISION to
a commit id to lock the kit to one state.

A recorded branch or tag gets its own banner, because a branch is a serving instruction and not a pin:

UNVERIFIED: the run recorded `main`, which is a branch or tag rather than a
commit. Branches and tags can be moved, so this kit can load different weights
than the reported numbers were measured on. Set HF_REVISION to a commit id to
lock it to one state.

The launch files still pass that branch, because it is what the run asked for, but OPTIMIZATION.lock records weights_pin as moving-ref with weights_reproducible false, and nothing in the kit describes it as pinned.

The weights pin and the optimization claim are separate facts. A pipeline no run ever measured ships no claim surfaces at all: no OPTIMIZATIONS.md, APPLIED-OPTIMIZATION.md, OPTIMIZATION.lock, or optimization-report.md. Its README.md still states the missing pin.

Before relying on a kit’s numbers, do what the banner asks: set HF_REVISION to an immutable commit id of the model repository. A kit that already carries a resolved commit needs no action.

Deployment modes

Flex (Core)
Active (Core)

Scale-to-zero. Replicas shed after 5 idle minutes, spin back up in under 2 seconds on next request.

Property	Value
Cost	Per-token only, nothing when idle
Cold start	Under 2 s on RunInfra Cloud
Idle timeout	5 minutes of no traffic
Best for	Development, bursty traffic, cost-sensitive workloads

Always-on. Replicas stay warm 24/7, zero cold start on any request.

Property	Value
Cost	Per-token rate + flat base fee per warm replica
Cold start	None
Best for	Production APIs with SLAs, latency-critical apps

Calling the endpoint

OpenAI-compatible. Use the OpenAI Python or JavaScript SDK by setting the RunInfra base URL and API key:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.runinfra.ai/v1",
    api_key="YOUR_RUNINFRA_API_KEY",
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.runinfra.ai/v1",
  apiKey: "YOUR_RUNINFRA_API_KEY",
});

const response = await client.chat.completions.create({
  model: "llama-3.3-70b",
  messages: [{ role: "user", content: "Hello" }],
});
console.log(response.choices[0].message.content);

curl https://api.runinfra.ai/v1/chat/completions \
  -H "Authorization: Bearer YOUR_RUNINFRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"llama-3.3-70b","messages":[{"role":"user","content":"Hello"}]}'

Verified clients include the OpenAI Python and JavaScript SDKs, LangChain, LlamaIndex, Vercel AI SDK, Instructor, and direct HTTP for supported endpoints. See OpenAI compatibility for the full contract.

Capacity

Each replica serves up to 30 concurrent requests. Replica budgets step with your lifetime credit purchases:

Lifetime credit purchases	Max replicas	Max concurrent requests
$50 or more	4	120
$250 or more	16	480
$10,000 or more	32	960
Enterprise	Custom	Custom

Exceeding the budget returns 429 with Retry-After. See Autoscaling to raise caps and tune concurrency.

Playground

Test before or after deploying from the Deploy tab. Send prompts, inspect token counts, compare variants, measure latency, without writing a line of code. After deployment, the playground targets the selected endpoint row, so the latency and output come from the endpoint you are inspecting.

Workspace	Playground requests
Below the $50 milestone	100/day
$50 or more in lifetime purchases	Unlimited
Enterprise	Unlimited

The first playground request after an idle period pays a cold start (up to 2 s on Flex deployments). Subsequent requests are fast.

Manage endpoints

From chat or Deployments:

Stop

Pause the endpoint. No charges while stopped.

Start

Resume from stopped. Cached weights make restarts fast.

Change GPU

Switch tier. The agent warns if re-optimization is recommended.

Worker counts persist across these actions. The worker count requested at deploy time is reapplied on restart, start, and GPU changes: an explicit count on the request always wins, otherwise the deployment’s original sizing applies, otherwise the workspace default. Worker counts are capped at 32 per deployment, and your spend tier or an operational cap can be lower. See Pricing and credits for the replica and GPU ceiling at each tier.

Known limitations

Request timeout is long but finite. For large max_tokens, stream the response so you don’t hit the timeout.
First deploy of a pipeline pays the full weight warm-up (minutes). Subsequent cold starts reuse Instant Start’s weight cache and are much faster.
Active mode unlocks with deploy, at $50 in lifetime credit purchases.

Common questions

How do I know when the deployment is ready?

The Deployments dashboard shows provisioning and transitions to active when the replica is serving. You can watch logs from the deployment’s detail page to confirm the model finished loading.

Is there a dry-run or preview mode?

The playground under the Deploy tab is the preview. Send real prompts, inspect quality and latency, compare variants, before you commit to deploying an endpoint that serves external traffic.

How do I roll back to a previous variant?

From the pipeline page, pick any prior optimization variant and redeploy it. Weights are usually still cached from the earlier run, so the rollback is fast.

Next steps

Deployment targets

Managed RunInfra Cloud, your RunPod, self-hosted Modal, or custom GPU.

Autoscaling

Replica budget, concurrency, Flex vs Active knobs.

Instant Start

Cold-start weight caching explained.

Speculation

Draft-model speculative decoding for throughput.

Get started

Using the agent

Optimization

Platform

Use cases

Deployments

Runbooks

Guides

Using with other libraries

Cookbook

Deployments overview

Request lifecycle

Where it runs

Deploy

Optimized artifact deploys

Modality verification and API keys

Deployment kits

The kit pins the weights the run measured

Deployment modes

Calling the endpoint

Capacity

Playground

Manage endpoints

Stop

Start

Change GPU

Known limitations

Common questions

Next steps

Deployment targets

Autoscaling

Instant Start

Speculation

​Request lifecycle

​Where it runs

​Deploy

​Optimized artifact deploys

​Modality verification and API keys

​Deployment kits

​The kit pins the weights the run measured

​Deployment modes

​Calling the endpoint

​Capacity

​Playground

​Manage endpoints

Stop

Start

Change GPU

​Known limitations

​Common questions

​Next steps

Deployment targets

Autoscaling

Instant Start

Speculation

Request lifecycle

Where it runs

Deploy

Optimized artifact deploys

Modality verification and API keys

Deployment kits

The kit pins the weights the run measured

Deployment modes

Calling the endpoint

Capacity

Playground

Manage endpoints

Known limitations

Common questions

Next steps