Quantization
FP8 quantization represents selected weights, activations, or cache values with eight-bit floating point formats. Scale factors map the narrower format to the magnitude range needed by each tensor or channel.
Narrower values reduce memory traffic and storage, and supported hardware can execute more low-precision work. Poor scaling or sensitive layers can erase those gains through quality loss or fallback work.
A serving stack chooses which tensors use the narrow format and how scales are produced. Teams compare task scores and serving metrics against the unquantized baseline before accepting the configuration.
FP8 quantization represents selected weights, activations, or cache values with eight-bit floating point formats. Scale factors map the narrower format to the magnitude range needed by each tensor or channel. A serving stack chooses which tensors use the narrow format and how scales are produced. Teams compare task scores and serving metrics against the unquantized baseline before accepting the configuration.
Narrower values reduce memory traffic and storage, and supported hardware can execute more low-precision work. Poor scaling or sensitive layers can erase those gains through quality loss or fallback work.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
