What does Speculative decoding mean in inference serving?
Speculative decoding uses a cheaper draft process to propose several candidate tokens before the target model verifies them. Exact acceptance methods preserve the target model's output distribution while accepting valid candidates in groups. A serving configuration pairs a target model with a compatible draft method and tracks acceptance alongside latency. Teams test real prompts because sequence behavior and hardware balance determine whether speculation pays off.
