Parallelism
Tensor parallelism shards weight matrices and their computation across multiple accelerators within each model layer. Participating devices exchange partial results so the layer produces the same logical output.
Sharding lets a model use combined memory and compute, but communication occurs repeatedly through the network fabric. Slow collectives or small workloads can leave communication cost larger than the local compute saved.
A serving deployment chooses a parallel group size that fits the model and the available interconnect. Operators compare memory headroom and latency because adding devices does not guarantee a faster request.
Tensor parallelism shards weight matrices and their computation across multiple accelerators within each model layer. Participating devices exchange partial results so the layer produces the same logical output. A serving deployment chooses a parallel group size that fits the model and the available interconnect. Operators compare memory headroom and latency because adding devices does not guarantee a faster request.
Sharding lets a model use combined memory and compute, but communication occurs repeatedly through the network fabric. Slow collectives or small workloads can leave communication cost larger than the local compute saved.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
