
DiLoCo, a variant of Federated Averaging, enables distributed training of large language models across poorly connected devices by using AdamW as the inner optimizer, Nesterov momentum as the outer optimizer, and a large number of inner optimization steps.


coremsg

distributed-low-communication-training-of-large-language-models-with-nesterov-momentum


Distributed Low-Communication Training of Large Language Models with Nesterov Momentum


title_rewrite


The Holmes framework enables efficient distributed training of large language models across GPU clusters with heterogeneous network interface cards, outperforming existing frameworks in the heterogeneous NIC environment.


efficient-distributed-training-of-large-language-models-across-heterogeneous-gpu-clusters


Efficient Distributed Training of Large Language Models Across Heterogeneous GPU Clusters
