登录
洞察 - Distributed Training of Large Language Models
Distributed Low-Communication Training of Large Language Models with Nesterov Momentum
DiLoCo, a variant of Federated Averaging, enables distributed training of large language models across poorly connected devices by using AdamW as the inner optimizer, Nesterov momentum as the outer optimizer, and a large number of inner optimization steps.
Efficient Distributed Training of Large Language Models Across Heterogeneous GPU Clusters
The Holmes framework enables efficient distributed training of large language models across GPU clusters with heterogeneous network interface cards, outperforming existing frameworks in the heterogeneous NIC environment.
