Latency
Time to first token measures elapsed time from request arrival until the first generated token becomes available. It includes queueing and setup, while prompt prefill usually supplies most active model computation.
Long first-token delay makes interactive systems feel unresponsive before generation has visibly started. Prefix reuse, prompt length, queueing, and prefill throughput can move this metric independently of decode speed.
Teams record the request boundary and the first streamed token, then inspect the distribution across workloads. They separate queue delay from prefill when diagnosing whether capacity or prompt processing is responsible.
Time to first token measures elapsed time from request arrival until the first generated token becomes available. It includes queueing and setup, while prompt prefill usually supplies most active model computation. Teams record the request boundary and the first streamed token, then inspect the distribution across workloads. They separate queue delay from prefill when diagnosing whether capacity or prompt processing is responsible.
Long first-token delay makes interactive systems feel unresponsive before generation has visibly started. Prefix reuse, prompt length, queueing, and prefill throughput can move this metric independently of decode speed.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
