Quantization
Weight-only quantization narrows stored weights while activations remain at higher precision. Weight-activation quantization also narrows activations so supported hardware can use lower-precision compute paths.
Weight-only methods primarily reduce model memory and weight bandwidth with simpler activation handling. Narrowing activations can reduce more traffic and use different compute units, but scaling and outliers become harder.
Teams compare both on the same model, evaluation set, request shapes, and hardware. Kernel availability, memory limits, quality tolerance, and workload shape decide the better configuration.
Weight-only quantization narrows stored weights while activations remain at higher precision. Weight-activation quantization also narrows activations so supported hardware can use lower-precision compute paths. Teams compare both on the same model, evaluation set, request shapes, and hardware. Kernel availability, memory limits, quality tolerance, and workload shape decide the better configuration.
Weight-only methods primarily reduce model memory and weight bandwidth with simpler activation handling. Narrowing activations can reduce more traffic and use different compute units, but scaling and outliers become harder.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
