Why load test
A model can look healthy at low concurrency and still fail under production traffic. Load testing validates scheduler behavior, queueing, memory pressure, and tail latency before a deployment becomes critical. High-concurrency tests are useful for three failure modes:- Scheduler saturation: requests queue once the runtime reaches its in-flight limit.
- Connection pressure: client or gateway connection pools become the bottleneck.
- VRAM pressure: KV cache or activation memory grows until the runtime must reject or restart work.
Test plan
Start below the expected production peak, then increase concurrency until one metric clearly bends.| Step | Purpose |
|---|---|
| Warmup | Load weights, compile kernels where applicable, and avoid cold-start noise |
| Baseline | Measure healthy latency and throughput at expected traffic |
| Ramp | Increase concurrency in stages |
| Stress | Hold the highest safe stage long enough to observe p95 and p99 behavior |
| Recovery | Return to baseline and confirm the deployment stabilizes |
What to report
| Metric | Why it matters |
|---|---|
| p50, p95, p99 latency | Shows scheduler health and tail behavior |
| Successful RPS | Shows useful throughput under load |
| Token or vector throughput | Normalizes work across request shapes |
| Peak VRAM | Catches memory cliffs before production |
| OOM threshold | Identifies the unsafe concurrency region |
| Success rate | Reveals dropped requests or backend errors |
| Runtime settings | Confirms the optimized launch settings were preserved |
Failure-mode triage
| Symptom | Likely cause | Fix |
|---|---|---|
| p99 latency spikes at high concurrency | Runtime queue depth is too low | Raise the in-flight request cap until VRAM approaches the ceiling |
| RPS plateaus mid-sweep | GPU compute is saturated | Add replicas, add GPUs, or choose a lower-cost latency target |
| OOM at higher concurrency | KV cache or activations exceed VRAM | Lower max context, lower concurrency, or move to a larger GPU |
| Runtime settings do not match the optimized variant | Deployment drift | Redeploy the selected optimized version |
| All requests fail immediately | Startup or routing failure | Inspect deployment health and request IDs |
Modality-specific notes
| Modality | Load-test focus |
|---|---|
| LLM | First-token latency, output-token throughput, and KV-cache pressure |
| Embeddings | Batch size, vector throughput, and input length distribution |
| ASR | Real-time factor, audio duration mix, and chunk count |
| TTS | Time to first audio chunk and stream stability |
| Image generation | Per-image latency, image count, size, and scheduler behavior |
| Vision-language | Image-heavy prompts, cache hits, and multimodal concurrency |