Latency
CUDA graphs record a repeatable sequence of device operations and replay it with fewer host launches. The captured graph expects stable execution structure and memory addresses.
Replay removes repeated launch work from short decode iterations where host overhead can be visible. Capture consumes setup time and can restrict dynamic shapes or control flow.
Serving systems may capture common shape buckets, pad work, or recapture when shapes change. Unsupported paths remain eager, and graph memory must stay valid across replays.
CUDA graphs record a repeatable sequence of device operations and replay it with fewer host launches. The captured graph expects stable execution structure and memory addresses. Serving systems may capture common shape buckets, pad work, or recapture when shapes change. Unsupported paths remain eager, and graph memory must stay valid across replays.
Replay removes repeated launch work from short decode iterations where host overhead can be visible. Capture consumes setup time and can restrict dynamic shapes or control flow.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
