Skip to main content

​
What to monitor

Production teams should monitor both endpoint performance and optimization quality. The key question is simple: did the deployed pipeline keep the latency, throughput, cost, and quality profile that won the optimization run?
SignalWhat it meansAlert when
Runtime driftRuntime launch settings differ from the optimized configurationAny confirmed drift affects production traffic
Technique fallbackA selected optimization technique could not be appliedThe rate rises above normal for a model family or runtime
Quality rollbackA candidate regresses below the configured quality gateRollbacks cluster on the same model, runtime, or technique
Per-phase cost outlierAn optimization phase costs much more than expectedp95 phase cost rises materially above the trailing baseline
Optimization abort rateSessions fail or refund after work startsThe rate spikes across unrelated workspaces
Plan-limit hintsUsers hit plan gates while optimizingThe rate changes enough to affect conversion or support load

​
Per-modality SLO examples

Tune these thresholds for your own traffic and model mix.
ModalitySLOAlert threshold
LLM, 7B-classp95 first token under 200 msAbove 300 ms for 5 minutes
LLM, 70B-classp95 first token under 400 msAbove 600 ms for 5 minutes
LLM throughputMore than 50 RPS per H100-class deploymentBelow 40 RPS for 5 minutes
Embeddingsp95 vector latency under 50 ms at batch 64Above 100 ms for 5 minutes
ASRReal-time factor under 0.05 at batch 16Above 0.10 for 5 minutes
TTSStreaming time to first byte under 200 msAbove 500 ms for 5 minutes
Image generationFLUX Schnell under 1.5 s per imageAbove 2.5 s for 5 minutes
Vision-languageRepeated-image time to first token under 1 sAbove 2 s for 5 minutes

​
Dashboard layout

A compact matrix works well for operations teams: 
                    LLM   Embedding   ASR   TTS   Image-gen   VL
p95 latency          .      .         .     .     .           .
Throughput           .      .         .     .     .           .
Quality              .      .         .     .     .           .
Drift events         .      .         .     .     .           .
Fallback rate        .      .         .     .     .           .
Cost                 .      .         .     .     .           .
Use green for healthy, yellow for approaching the threshold, and red for active breach. Keep one row per metric, one column per modality, and one cell per latest evaluation window.

​
Alert routing

Start with three alert classes:
  • Customer-facing incident: endpoint unavailable, severe latency breach, or repeated upstream failures.
  • Optimization quality risk: drift, fallback spike, or quality rollback cluster.
  • Cost control risk: phase cost outliers or unexpected reservation failures.
Include the deployment ID, model ID, optimization version, request ID where available, and the last known runtime settings in each alert.