Hardware
NVLink is a high-bandwidth connection used between supported accelerators in the same system or fabric. Collective operations can move tensor shards and partial results across these links.
Tensor-parallel latency depends on repeated communication as well as local computation. Without direct links, collectives use the available host or network path, whose bandwidth and topology may differ.
Operators map parallel groups onto connected devices and inspect the actual communication topology. They measure collective time because link presence alone does not guarantee balanced or contention-free traffic.
NVLink is a high-bandwidth connection used between supported accelerators in the same system or fabric. Collective operations can move tensor shards and partial results across these links. Operators map parallel groups onto connected devices and inspect the actual communication topology. They measure collective time because link presence alone does not guarantee balanced or contention-free traffic.
Tensor-parallel latency depends on repeated communication as well as local computation. Without direct links, collectives use the available host or network path, whose bandwidth and topology may differ.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
