Hardware
High-bandwidth memory is stacked device memory placed close to an accelerator's compute package. It offers much more bandwidth than typical host memory but remains finite in capacity.
Decode throughput often depends on moving weights and cache state through this memory quickly. Compute rate alone cannot describe serving performance when memory traffic is the limiting work.
Operators budget capacity among weights, KV cache, workspaces, and captured graphs. Precision, batch shape, access pattern, and model architecture determine bandwidth pressure.
High-bandwidth memory is stacked device memory placed close to an accelerator's compute package. It offers much more bandwidth than typical host memory but remains finite in capacity. Operators budget capacity among weights, KV cache, workspaces, and captured graphs. Precision, batch shape, access pattern, and model architecture determine bandwidth pressure.
Decode throughput often depends on moving weights and cache state through this memory quickly. Compute rate alone cannot describe serving performance when memory traffic is the limiting work.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
