Optimized Network Architectures for Large Language Model Training with
  Billions of Parameters
This paper challenges the well-established paradigm for building any-to-any
networks for training Large Language Models (LLMs). We show that LLMs exhibit a
unique communication pattern where only small groups of GPUs require
high-bandwidth any-to-any communication within them, to achieve near-optimal
training performance. Across these groups of GPUs, the communication is
insignificant, sparse, and homogeneous. We propose a new network architecture
that closely resembles the communication requirement of LLMs. Our architecture
partitions the cluster into sets of GPUs interconnected with non-blocking
any-to-any high-bandwidth interconnects that we call HB domains. Across the HB
domains, the network only connects GPUs with communication demands. We call
this network a "rail-only" connection, and show that our proposed architecture
reduces the network cost by up to 75% compared to the state-of-the-art
any-to-any Clos networks without compromising the performance of LLM training.
64bf3752af05eb17c71abf36
Weiyang Wang
64bf3752af05eb17c71abf37
Manya Ghobadi
64bf3752af05eb17c71abf38
Kayvon Shakeri
64bf3752af05eb17c71abf39
Ying Zhang
64bf3752af05eb17c71abf3a
Naader Hasani
Optimized Network Architectures for Large Language Model Training with
  Billions of Parameters
This paper challenges the well-established paradigm for building any-to-any
networks for training Large Language Models (LLMs). We show that LLMs exhibit a
unique communication pattern where only small groups of GPUs require
high-bandwidth any-to-any communication within them, to achieve near-optimal
training performance. Across these groups of GPUs, the communication is
insignificant, sparse, and homogeneous. We propose a new network architecture
that closely resembles the communication requirement of LLMs. Our architecture
partitions the cluster into sets of GPUs interconnected with non-blocking
any-to-any high-bandwidth interconnects that we call HB domains. Across the HB
domains, the network only connects GPUs with communication demands. We call
this network a "rail-only" connection, and show that our proposed architecture
reduces the network cost by up to 75% compared to the state-of-the-art
any-to-any Clos networks without compromising the performance of LLM training.
A new network architecture optimizes GPU communication for Large Language Model training by partitioning GPUs into high-bandwidth domains, reducing network costs.
Large Language Models (LLMs) any-to-any networks GPUs high-bandwidth communication HB domains rail-only connection Clos networks