> ## Documentation Index
> Fetch the complete documentation index at: https://runinfra.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Load testing optimized deployments

> How to plan high-concurrency tests for each modality and interpret the results.

## Why load test

A model can look healthy at low concurrency and still fail under production traffic. Load testing validates scheduler behavior, queueing, memory pressure, and tail latency before a deployment becomes critical.

High-concurrency tests are useful for three failure modes:

1. **Scheduler saturation**: requests queue once the runtime reaches its in-flight limit.
2. **Connection pressure**: client or gateway connection pools become the bottleneck.
3. **VRAM pressure**: KV cache or activation memory grows until the runtime must reject or restart work.

## Test plan

Start below the expected production peak, then increase concurrency until one metric clearly bends.

| Step     | Purpose                                                                    |
| -------- | -------------------------------------------------------------------------- |
| Warmup   | Load weights, compile kernels where applicable, and avoid cold-start noise |
| Baseline | Measure healthy latency and throughput at expected traffic                 |
| Ramp     | Increase concurrency in stages                                             |
| Stress   | Hold the highest safe stage long enough to observe p95 and p99 behavior    |
| Recovery | Return to baseline and confirm the deployment stabilizes                   |

## What to report

| Metric                     | Why it matters                                        |
| -------------------------- | ----------------------------------------------------- |
| p50, p95, p99 latency      | Shows scheduler health and tail behavior              |
| Successful RPS             | Shows useful throughput under load                    |
| Token or vector throughput | Normalizes work across request shapes                 |
| Peak VRAM                  | Catches memory cliffs before production               |
| OOM threshold              | Identifies the unsafe concurrency region              |
| Success rate               | Reveals dropped requests or backend errors            |
| Runtime settings           | Confirms the optimized launch settings were preserved |

## Failure-mode triage

| Symptom                                             | Likely cause                        | Fix                                                               |
| --------------------------------------------------- | ----------------------------------- | ----------------------------------------------------------------- |
| p99 latency spikes at high concurrency              | Runtime queue depth is too low      | Raise the in-flight request cap until VRAM approaches the ceiling |
| RPS plateaus mid-sweep                              | GPU compute is saturated            | Add replicas, add GPUs, or choose a lower-cost latency target     |
| OOM at higher concurrency                           | KV cache or activations exceed VRAM | Lower max context, lower concurrency, or move to a larger GPU     |
| Runtime settings do not match the optimized variant | Deployment drift                    | Redeploy the selected optimized version                           |
| All requests fail immediately                       | Startup or routing failure          | Inspect deployment health and request IDs                         |

## Modality-specific notes

| Modality         | Load-test focus                                                     |
| ---------------- | ------------------------------------------------------------------- |
| LLM              | First-token latency, output-token throughput, and KV-cache pressure |
| Embeddings       | Batch size, vector throughput, and input length distribution        |
| ASR              | Real-time factor, audio duration mix, and chunk count               |
| TTS              | Time to first audio chunk and stream stability                      |
| Image generation | Per-image latency, image count, size, and scheduler behavior        |
| Vision-language  | Image-heavy prompts, cache hits, and multimodal concurrency         |

Use these results to set deployment SLOs and to decide whether to adjust replicas, GPU tier, context length, batch size, or optimization priority.