Serving
A cold start is the delay before an inactive or new replica can accept useful model work. Weight loading, device initialization, graph capture, and cache warming may contribute.
Cold replicas can turn scaling events or failover into long tail delays. Keeping replicas warm reduces exposure but consumes capacity while demand is low.
Prewarming and readiness checks reduce exposed delay but still require honest capacity accounting. The dominant step varies with model size, storage locality, runtime, and hardware.
A cold start is the delay before an inactive or new replica can accept useful model work. Weight loading, device initialization, graph capture, and cache warming may contribute. Prewarming and readiness checks reduce exposed delay but still require honest capacity accounting. The dominant step varies with model size, storage locality, runtime, and hardware.
Cold replicas can turn scaling events or failover into long tail delays. Keeping replicas warm reduces exposure but consumes capacity while demand is low.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
