NCCL
lastmod 2026-11-10created 2018-01-292 edits
NCCL (NVIDIA Collective Communications Library) is NVIDIA's library implementing collective communication among multiple GPUs. It performs operations such as all-reduce, broadcast, reduce, and all-gather over paths optimized for the interconnect, such as PCIe, NVLink, and InfiniBand, and handles both GPUs within a single node and GPUs spanning multiple nodes. The distributed training backends of PyTorch and TensorFlow use this library for gradient synchronization.
Optimized primitives for collective multi-GPU communication.