Deployments overview

A deployment turns an optimized pipeline into a live HTTP endpoint. You get a URL, a scoped API key, autoscaling replicas, Instant Start cold-start caching, and supported OpenAI-compatible routes where the deployment exposes them.

Deployment requires a paid Core plan (custom amount from

50/mo). Your

10 in free credits let you build, optimize, and test in the playground without deploying.

Request lifecycle

Every request hits the edge, authenticates against your API key, routes to a replica with headroom, and streams tokens back. Cold starts pull from Instant Start; warm requests skip it.

Where it runs

Four deployment targets, picked when you click Deploy:

Target	When to pick it
Managed RunInfra Cloud (default)	Scale-to-zero with Instant Start. Fastest path; RunInfra runs the infra.
Your RunPod (bring your own cloud)	Delivery to your RunPod account after RunInfra Modal measurement evidence exists.
Self-hosted Modal	Reserved GPUs on your Modal account, billed to you.
Custom GPU	Bare metal, on-prem, regulated industries, existing GPU contracts.

See Deployment targets for the full picking guide.

Deploy

From chat
From the Deploy tab

Deploy this pipeline

The agent picks the winning variant, provisions a GPU, and returns the endpoint URL and key.

Optimized artifact deploys

When you deploy an optimized variant, RunInfra Cloud uses the deployable artifact from the selected pipeline version. The deploy request carries the pipeline files, model revision, serving backend, and optimization settings that produced the winning variant. Managed deployments reject missing or stale artifact snapshots before provisioning. This prevents a deployment from silently falling back to a generic base image and losing the optimization that you selected. Each custom artifact build returns a deployment receipt with:

Field	Meaning
`buildSource`	Shows whether the image came from a custom artifact build.
`artifactSha256`	Stable identity for the pipeline files used as build input.
`contentHash`	Image build content hash used for cache and audit checks.
`imageTag`	Container image tag deployed to the serving backend.
`servingBackend`	Backend selected for the deployed model.

The Deploy tab refreshes the live capability contract before provisioning. If a backend or modality is not currently enabled, deployment stops before creating GPU resources and shows the required action.

Modality verification and API keys

Managed deployments verify through a modality-native gateway canary before they go active. Voice pipeline, speech-to-text, text-to-speech, audio-language, and vision-language deployments verify this way and can mint working API keys immediately after the deployment verifies; previously that immediate key flow covered LLM and embedding deployments only. Minted keys pair with the flat base URL https://api.runinfra.ai/v1. Image generation, rerank, and classification managed deployments remain unavailable. Rerank and classification are delivered through export kits instead; see Export packages.

Export packages

The Export tab packages the same selected pipeline version that deployment uses. It now centers on one optimized pipeline package: build it, download it, run the setup guide, and verify before serving. Downloads include these files:

File	Purpose
`README.md`	Orientation and quick start.
`FETCH_MODEL.sh`	Downloads the optimized weights into `./model` from a signed, time-limited URL.
`runinfra/manifest.json`	Provenance, runtime, GPU, optimization, proof level, and portability contract.
`runinfra/verify.sh`	Runs the bundle’s canary verification against the manifest.
`runinfra/Dockerfile`	Builds an image pinned to the runtime used by the selected artifact.
`runinfra/serve.sh`	Starts the selected runtime against `./model`.
`runinfra/docker-compose.yml`	Local validation with the same runtime image and model mount.
`runinfra/benchmarks.json`	Benchmark projection from the manifest.
`runinfra/eval.json`	Quality projection from the manifest.
`runinfra/canary_inputs/sample.json`	Modality-aware sample request for verification.
`runinfra/checksums.txt`	SHA-256 checksums for the generated bundle files.

Runtime files are generated for the selected backend. vLLM and SGLang launch their Python serving modules, vLLM Omni pins the audio runtime, TensorRT-LLM uses Triton Server, and Diffusers or TEI preserve the base image entrypoint and required environment. Rerank and classification kits use the explicit tei export format: the generated serving setup runs text-embeddings-inference, and the kit’s serve script answers the kit’s own documented endpoints, /rerank for rerank kits and /predict for classification kits. The manifest exposes a proof_level so downstream teams can see how much measured evidence backs the export:

Proof level	Meaning
`benchmarked_and_quality_gated`	Benchmarks and a quality gate are present for the exported artifact.
`benchmarked_only`	Benchmarks are present, but no quality gate was recorded for this export.
`uncharacterized`	The export is usable, but benchmark or quality evidence is missing.

When proof is incomplete, the Export tab and generated manifest show the degraded proof level instead of filling in confident numbers. Missing benchmark, quality, library-version, or out-of-envelope data is rendered as uncharacterized. Export job status preserves the selected optimizationVersionId and runbook executionId. Use those ids to confirm the downloaded bundle or Hugging Face push belongs to the accepted optimization run before treating it as production evidence. The signed weight URL in FETCH_MODEL.sh expires. If the link is stale, rebuild the bundle from the Export tab to get a fresh URL.

Deployment modes

Flex (Core)
Active (Core)

Scale-to-zero. Replicas shed after 5 idle minutes, spin back up in under 2 seconds on next request.

Property	Value
Cost	Per-token only, nothing when idle
Cold start	Under 2 s on RunInfra Cloud
Idle timeout	5 minutes of no traffic
Best for	Development, bursty traffic, cost-sensitive workloads

Always-on. Replicas stay warm 24/7, zero cold start on any request.

Property	Value
Cost	Per-token rate + flat base fee per warm replica
Cold start	None
Best for	Production APIs with SLAs, latency-critical apps

Calling the endpoint

OpenAI-compatible. Use the OpenAI Python or JavaScript SDK by setting the RunInfra base URL and API key:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.runinfra.ai/v1",
    api_key="YOUR_RUNINFRA_API_KEY",
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.runinfra.ai/v1",
  apiKey: "YOUR_RUNINFRA_API_KEY",
});

const response = await client.chat.completions.create({
  model: "llama-3.3-70b",
  messages: [{ role: "user", content: "Hello" }],
});
console.log(response.choices[0].message.content);

curl https://api.runinfra.ai/v1/chat/completions \
  -H "Authorization: Bearer YOUR_RUNINFRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"llama-3.3-70b","messages":[{"role":"user","content":"Hello"}]}'

Verified clients include the OpenAI Python and JavaScript SDKs, LangChain, LlamaIndex, Vercel AI SDK, Instructor, and direct HTTP for supported endpoints. See OpenAI compatibility for the full contract.

Capacity

Each replica serves up to 30 concurrent requests. Replica budgets by plan:

Plan	Max replicas	Max concurrent requests
Core	32	960
Enterprise	Custom	Custom

Exceeding the budget returns 429 with Retry-After. See Autoscaling to raise caps and tune concurrency.

Playground

Test before or after deploying from the Deploy tab. Send prompts, inspect token counts, compare variants, measure latency, without writing a line of code. After deployment, the playground targets the selected endpoint row, so the latency and output come from the endpoint you are inspecting.

Plan	Playground requests
Free (trial)	100/day
Core	Unlimited
Enterprise	Unlimited

The first playground request after an idle period pays a cold start (up to 2 s on Flex deployments). Subsequent requests are fast.

Manage endpoints

From chat or Deployments:

Stop

Pause the endpoint. No charges while stopped.

Start

Resume from stopped. Cached weights make restarts fast.

Change GPU

Switch tier. The agent warns if re-optimization is recommended.

Worker counts persist across these actions. The worker count requested at deploy time is reapplied on restart, start, and GPU changes: an explicit count on the request always wins, otherwise the deployment’s original sizing applies, otherwise plan defaults. Worker counts are capped at 32 per deployment, and plan or operational caps can be lower.

Known limitations

Request timeout is long but finite. For large max_tokens, stream the response so you don’t hit the timeout.
First deploy of a pipeline pays the full weight warm-up (minutes). Subsequent cold starts reuse Instant Start’s weight cache and are much faster.
Active mode requires a paid Core plan.

Common questions

How do I know when the deployment is ready?

The Deployments dashboard shows provisioning and transitions to active when the replica is serving. You can watch logs from the deployment’s detail page to confirm the model finished loading.

Is there a dry-run or preview mode?

The playground under the Deploy tab is the preview. Send real prompts, inspect quality and latency, compare variants, before you commit to deploying an endpoint that serves external traffic.

How do I roll back to a previous variant?

From the pipeline page, pick any prior optimization variant and redeploy it. Weights are usually still cached from the earlier run, so the rollback is fast.

Next steps

Deployment targets

Managed RunInfra Cloud, your RunPod, self-hosted Modal, or custom GPU.

Autoscaling

Replica budget, concurrency, Flex vs Active knobs.

Instant Start

Cold-start weight caching explained.

Speculation

Draft-model speculative decoding for throughput.

​Request lifecycle

​Where it runs

​Deploy

​Optimized artifact deploys

​Modality verification and API keys

​Export packages

​Deployment modes

​Calling the endpoint

​Capacity

​Playground

​Manage endpoints

Stop

Start

Change GPU

​Known limitations

​Common questions

​Next steps

Deployment targets

Autoscaling

Instant Start

Speculation

Request lifecycle

Where it runs

Deploy

Optimized artifact deploys

Modality verification and API keys

Export packages

Deployment modes

Calling the endpoint

Capacity

Playground

Manage endpoints

Known limitations

Common questions

Next steps