Preserving Optimal Variance in Decentralized Neural Network Training to Improve Convergence Efficiency

Averaging uncorrelated neural network models in gossip learning systems can lead to a "vanishing variance" problem, causing significant convergence delays. A variance-corrected model averaging algorithm is proposed to eliminate this issue, enabling gossip learning to achieve convergence efficiency comparable to federated learning.