Originally published on the PyTorch Developer Log.
TL;DR - Python can drive GPUDirect RDMA at line rate without a large
transfer-engine abstraction. rdma4py provides lightweight, backend-specific
bindings for ibverbs, AWS EFA, and NVMe-oF, reaching about 400 gbps in our
benchmarks while preserving direct access to the underlying APIs.
Introduction
Recently, quite a few different “transfer engines” have been developed for fast
weight synchronization between machines using RDMA-capable transports and
libraries such as NVLink, ibverbs, EFA, NVMe-oF, and SPDK. These include projects
such as Mooncake,
NIXL, and
Uniflow.
As with any large project and abstraction, these projects make trade-offs around
specific use cases that might not be optimal for yours.
Originally published on the PyTorch Developer Log.
TL;DR – Modifying the C++ comms layer is a big barrier when researchers want to prototype new collective features. We’ve added Python bindings to torchcomms (#2080) and built two pure-Python backend prototypes — one wrapping NVIDIA’s new nccl4py bindings (#2515) and one built on SymmetricMemory + Triton (#2521) — both passing the core torchcomms integration test suite. Since they plug into torch.distributed, researchers can fork, tweak, and mix them with existing projects like TorchTitan without touching C++.
Originally published on the PyTorch Blog.
Introduction
Torchcomms is a new experimental, lightweight communication API intended for use with PyTorch Distributed (PTD). In addition to the core API, we are open-sourcing NCCLX, a new backend we developed to scale to over 100,000 GPUs.
With our first release of torchcomms, we’re delivering the foundational APIs and backends required for large-scale model training in PyTorch. This initial release focuses on core communication primitives that enable reliable and performant distributed training at scale. Over the next year, we’ll continue to mature the offering—introducing features that make it easier to prototype new collectives, scale seamlessly with built-in fault tolerance, and optimize device-centric communication patterns. Our roadmap is focused on empowering researchers and developers to move faster, test new ideas at scale, and build the next generation of large-scale AI systems.