> ## Documentation Index
> Fetch the complete documentation index at: https://runinfra.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Deployments overview

> Deploy optimized RunInfra pipelines as production APIs with supported OpenAI-compatible endpoint shapes. Two modes, fast cold starts, and usage-based billing.

A deployment turns an optimized pipeline into a live HTTP endpoint. You get a URL, a scoped API key, autoscaling replicas, Instant Start cold-start caching, and supported OpenAI-compatible routes where the deployment exposes them.

<Note>
  Deployment requires a paid Core plan (custom amount from $50/mo). Your $10 in free credits let you build, optimize, and test in the playground without deploying.
</Note>

## Request lifecycle

```mermaid theme={"dark"}
flowchart LR
    Client[Your app] -- OpenAI SDK --> Edge[RunInfra Edge]
    Edge -- route + auth --> LB[Load balancer]
    LB -- warm replica --> Replica[GPU replica]
    LB -. no replica .-> InstantStart[Instant Start weight cache]
    InstantStart -- fast path .-> Replica
    Replica -- tokens --> Edge
    Edge -- SSE or JSON --> Client
```

Every request hits the edge, authenticates against your API key, routes to a replica with headroom, and streams tokens back. Cold starts pull from Instant Start; warm requests skip it.

## Where it runs

Four deployment targets, picked when you click Deploy:

| Target                                 | When to pick it                                                                   |
| -------------------------------------- | --------------------------------------------------------------------------------- |
| **Managed RunInfra Cloud** (default)   | Scale-to-zero with Instant Start. Fastest path; RunInfra runs the infra.          |
| **Your RunPod** (bring your own cloud) | Delivery to your RunPod account after RunInfra Modal measurement evidence exists. |
| **Self-hosted Modal**                  | Reserved GPUs on your Modal account, billed to you.                               |
| **Custom GPU**                         | Bare metal, on-prem, regulated industries, existing GPU contracts.                |

See [Deployment targets](/deployments/targets) for the full picking guide.

## Deploy

<Tabs>
  <Tab title="From chat">
    ```text theme={"dark"}
    Deploy this pipeline
    ```

    The agent picks the winning variant, provisions a GPU, and returns the endpoint URL and key.
  </Tab>

  <Tab title="From the Deploy tab">
    Click **Deploy** in the Deploy tab. Pick the variant and the mode. Provisioning takes 1 to 3 minutes on the first deploy.
  </Tab>
</Tabs>

## Optimized artifact deploys

When you deploy an optimized variant, RunInfra Cloud uses the deployable artifact from the selected pipeline version. The deploy request carries the pipeline files, model revision, serving backend, and optimization settings that produced the winning variant.

Managed deployments reject missing or stale artifact snapshots before provisioning. This prevents a deployment from silently falling back to a generic base image and losing the optimization that you selected.

Each custom artifact build returns a deployment receipt with:

| Field            | Meaning                                                     |
| ---------------- | ----------------------------------------------------------- |
| `buildSource`    | Shows whether the image came from a custom artifact build.  |
| `artifactSha256` | Stable identity for the pipeline files used as build input. |
| `contentHash`    | Image build content hash used for cache and audit checks.   |
| `imageTag`       | Container image tag deployed to the serving backend.        |
| `servingBackend` | Backend selected for the deployed model.                    |

The Deploy tab refreshes the live capability contract before provisioning. If a backend or modality is not currently enabled, deployment stops before creating GPU resources and shows the required action.

## Modality verification and API keys

Managed deployments verify through a modality-native gateway canary before they go active. Voice pipeline, speech-to-text, text-to-speech, audio-language, and vision-language deployments verify this way and can mint working API keys immediately after the deployment verifies; previously that immediate key flow covered LLM and embedding deployments only. Minted keys pair with the flat base URL `https://api.runinfra.ai/v1`.

Image generation, rerank, and classification managed deployments remain unavailable. Rerank and classification are delivered through export kits instead; see [Export packages](#export-packages).

## Export packages

The Export tab packages the same selected pipeline version that deployment uses. It now centers on one optimized pipeline package: build it, download it, run the setup guide, and verify before serving. Downloads include these files:

| File                                 | Purpose                                                                         |
| ------------------------------------ | ------------------------------------------------------------------------------- |
| `README.md`                          | Orientation and quick start.                                                    |
| `FETCH_MODEL.sh`                     | Downloads the optimized weights into `./model` from a signed, time-limited URL. |
| `runinfra/manifest.json`             | Provenance, runtime, GPU, optimization, proof level, and portability contract.  |
| `runinfra/verify.sh`                 | Runs the bundle's canary verification against the manifest.                     |
| `runinfra/Dockerfile`                | Builds an image pinned to the runtime used by the selected artifact.            |
| `runinfra/serve.sh`                  | Starts the selected runtime against `./model`.                                  |
| `runinfra/docker-compose.yml`        | Local validation with the same runtime image and model mount.                   |
| `runinfra/benchmarks.json`           | Benchmark projection from the manifest.                                         |
| `runinfra/eval.json`                 | Quality projection from the manifest.                                           |
| `runinfra/canary_inputs/sample.json` | Modality-aware sample request for verification.                                 |
| `runinfra/checksums.txt`             | SHA-256 checksums for the generated bundle files.                               |

Runtime files are generated for the selected backend. vLLM and SGLang launch their Python serving modules, vLLM Omni pins the audio runtime, TensorRT-LLM uses Triton Server, and Diffusers or TEI preserve the base image entrypoint and required environment. Rerank and classification kits use the explicit `tei` export format: the generated serving setup runs text-embeddings-inference, and the kit's serve script answers the kit's own documented endpoints, `/rerank` for rerank kits and `/predict` for classification kits.

The manifest exposes a `proof_level` so downstream teams can see how much measured evidence backs the export:

| Proof level                     | Meaning                                                                   |
| ------------------------------- | ------------------------------------------------------------------------- |
| `benchmarked_and_quality_gated` | Benchmarks and a quality gate are present for the exported artifact.      |
| `benchmarked_only`              | Benchmarks are present, but no quality gate was recorded for this export. |
| `uncharacterized`               | The export is usable, but benchmark or quality evidence is missing.       |

When proof is incomplete, the Export tab and generated manifest show the degraded proof level instead of filling in confident numbers. Missing benchmark, quality, library-version, or out-of-envelope data is rendered as uncharacterized.

Export job status preserves the selected `optimizationVersionId` and runbook `executionId`. Use those ids to confirm the downloaded bundle or Hugging Face push belongs to the accepted optimization run before treating it as production evidence.

The signed weight URL in `FETCH_MODEL.sh` expires. If the link is stale, rebuild the bundle from the Export tab to get a fresh URL.

## Deployment modes

<Tabs>
  <Tab title="Flex (Core)">
    Scale-to-zero. Replicas shed after 5 idle minutes, spin back up in under 2 seconds on next request.

    | Property     | Value                                                 |
    | ------------ | ----------------------------------------------------- |
    | Cost         | Per-token only, nothing when idle                     |
    | Cold start   | Under 2 s on RunInfra Cloud                           |
    | Idle timeout | 5 minutes of no traffic                               |
    | Best for     | Development, bursty traffic, cost-sensitive workloads |
  </Tab>

  <Tab title="Active (Core)">
    Always-on. Replicas stay warm 24/7, zero cold start on any request.

    | Property   | Value                                            |
    | ---------- | ------------------------------------------------ |
    | Cost       | Per-token rate + flat base fee per warm replica  |
    | Cold start | None                                             |
    | Best for   | Production APIs with SLAs, latency-critical apps |
  </Tab>
</Tabs>

## Calling the endpoint

OpenAI-compatible. Use the OpenAI Python or JavaScript SDK by setting the RunInfra base URL and API key:

<CodeGroup>
  ```python Python theme={"dark"}
  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.runinfra.ai/v1",
      api_key="YOUR_RUNINFRA_API_KEY",
  )

  response = client.chat.completions.create(
      model="llama-3.3-70b",
      messages=[{"role": "user", "content": "Hello"}],
  )
  print(response.choices[0].message.content)
  ```

  ```javascript TypeScript theme={"dark"}
  import OpenAI from "openai";

  const client = new OpenAI({
    baseURL: "https://api.runinfra.ai/v1",
    apiKey: "YOUR_RUNINFRA_API_KEY",
  });

  const response = await client.chat.completions.create({
    model: "llama-3.3-70b",
    messages: [{ role: "user", content: "Hello" }],
  });
  console.log(response.choices[0].message.content);
  ```

  ```bash curl theme={"dark"}
  curl https://api.runinfra.ai/v1/chat/completions \
    -H "Authorization: Bearer YOUR_RUNINFRA_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"llama-3.3-70b","messages":[{"role":"user","content":"Hello"}]}'
  ```
</CodeGroup>

Verified clients include the OpenAI Python and JavaScript SDKs, LangChain, LlamaIndex, Vercel AI SDK, Instructor, and direct HTTP for supported endpoints. See [OpenAI compatibility](/tools-sdks/openai-compatibility) for the full contract.

## Capacity

Each replica serves up to 30 concurrent requests. Replica budgets by plan:

| Plan       | Max replicas | Max concurrent requests |
| ---------- | ------------ | ----------------------- |
| Core       | 32           | 960                     |
| Enterprise | Custom       | Custom                  |

Exceeding the budget returns 429 with `Retry-After`. See [Autoscaling](/deployments/autoscaling) to raise caps and tune concurrency.

## Playground

Test before or after deploying from the Deploy tab. Send prompts, inspect token counts, compare variants, measure latency, without writing a line of code.

After deployment, the playground targets the selected endpoint row, so the latency and output come from the endpoint you are inspecting.

| Plan         | Playground requests |
| ------------ | ------------------- |
| Free (trial) | 100/day             |
| Core         | Unlimited           |
| Enterprise   | Unlimited           |

<Info>
  The first playground request after an idle period pays a cold start (up to 2 s on Flex deployments). Subsequent requests are fast.
</Info>

## Manage endpoints

From chat or [Deployments](https://runinfra.ai/deployments):

<Columns cols={3}>
  <Card title="Stop" icon="pause">
    Pause the endpoint. No charges while stopped.
  </Card>

  <Card title="Start" icon="play">
    Resume from stopped. Cached weights make restarts fast.
  </Card>

  <Card title="Change GPU" icon="cpu">
    Switch tier. The agent warns if re-optimization is recommended.
  </Card>
</Columns>

Worker counts persist across these actions. The worker count requested at deploy time is reapplied on restart, start, and GPU changes: an explicit count on the request always wins, otherwise the deployment's original sizing applies, otherwise plan defaults. Worker counts are capped at 32 per deployment, and plan or operational caps can be lower.

## Known limitations

* Request timeout is long but finite. For large `max_tokens`, stream the response so you don't hit the timeout.
* First deploy of a pipeline pays the full weight warm-up (minutes). Subsequent cold starts reuse Instant Start's weight cache and are much faster.
* Active mode requires a paid Core plan.

## Common questions

<AccordionGroup>
  <Accordion title="How do I know when the deployment is ready?">
    The Deployments dashboard shows `provisioning` and transitions to `active` when the replica is serving. You can watch logs from the deployment's detail page to confirm the model finished loading.
  </Accordion>

  <Accordion title="Is there a dry-run or preview mode?">
    The playground under the Deploy tab is the preview. Send real prompts, inspect quality and latency, compare variants, before you commit to deploying an endpoint that serves external traffic.
  </Accordion>

  <Accordion title="How do I roll back to a previous variant?">
    From the pipeline page, pick any prior optimization variant and redeploy it. Weights are usually still cached from the earlier run, so the rollback is fast.
  </Accordion>
</AccordionGroup>

## Next steps

<Columns cols={2}>
  <Card title="Deployment targets" icon="cloud" href="/deployments/targets">
    Managed RunInfra Cloud, your RunPod, self-hosted Modal, or custom GPU.
  </Card>

  <Card title="Autoscaling" icon="gauge" href="/deployments/autoscaling">
    Replica budget, concurrency, Flex vs Active knobs.
  </Card>

  <Card title="Instant Start" icon="zap" href="/deployments/instant-start">
    Cold-start weight caching explained.
  </Card>

  <Card title="Speculation" icon="rocket" href="/deployments/speculation">
    Draft-model speculative decoding for throughput.
  </Card>
</Columns>