Improving Batch Normalization for Federated Deep Learning

Core Concepts

Batch Normalization (BN) can outperform Group Normalization (GN) in many federated learning (FL) settings, especially when communication frequency is low or non-IID degree is not severe. A simple practice named FIXBN is proposed to mitigate the issues of BN in FL while retaining its benefits.

Abstract

The paper investigates the use of Batch Normalization (BN) and its common alternative, Group Normalization (GN), in federated learning (FL). Through an expanded empirical study, the authors find that BN can outperform GN in many FL settings, except for high-frequency communication and extreme non-IID regimes.
The authors then delve deeper into the issues with BN in FL, including the mismatch of BN statistics across non-IID clients and the deviation of gradients during local training. They propose a simple yet effective practice named FIXBN to address these issues:

In the initial exploration stage, FIXBN follows the standard practice of FEDAVG with BN, enjoying the positive impacts of BN on local training.
In the later calibration stage, FIXBN freezes the BN layer and uses the globally aggregated BN statistics for normalization, mitigating the mismatch of statistics in training and testing, and allowing FEDAVG to recover the centralized gradient under high-frequency settings.

FIXBN is easy to implement, requires no architecture change, and incurs no additional training or communication costs. Extensive experiments show that FIXBN consistently outperforms or matches the performance of BN and GN across a wide range of FL settings, including image classification on CIFAR-10 and Tiny-ImageNet, as well as image segmentation on Cityscapes.
The authors also identify another gap between FEDAVG and centralized training - the lack of maintained local SGD momentum. They show that applying maintained local or global momentum can further improve the performance of FEDAVG with different normalizers.
Overall, the study provides valuable insights and a practical solution for using BN effectively in federated deep learning, serving as a foundation for future research and applications.

Stats

The paper does not contain any key metrics or important figures to support the author's key logics.

Quotes

The paper does not contain any striking quotes supporting the author's key logics.

Key Insights Distilled From

Making Batch Normalization Great in Federated Deep Learning

by Jike Zhong,H... at arxiv.org 04-01-2024

https://arxiv.org/pdf/2303.06530.pdf

Deeper Inquiries

How can the theoretical understanding of Batch Normalization be extended to the federated learning setting?

In the context of federated learning, the theoretical understanding of Batch Normalization (BN) can be extended by considering the unique challenges and characteristics of decentralized data and communication in this setting. One key aspect to explore is the impact of non-IID data distributions across clients on the effectiveness of BN. The theoretical analysis should delve into how the mismatch of mini-batch statistics across non-IID clients affects the convergence and generalization of the federated learning model. Additionally, investigating how the normalization process in BN interacts with the communication constraints and decentralized optimization framework of federated learning would be crucial. By incorporating these factors into the theoretical framework, researchers can gain insights into optimizing BN for federated learning scenarios.

What are the potential drawbacks or limitations of the FIXBN approach, and how can they be addressed?

While FIXBN offers a promising solution to mitigate the negative impacts of Batch Normalization (BN) in federated learning, there are potential drawbacks and limitations to consider. One limitation is the selection of the round (T ⋆) at which to freeze the BN statistics for normalization. The effectiveness of FIXBN could vary based on this choice, and finding an optimal strategy for determining T ⋆ is essential. Additionally, FIXBN may introduce additional complexity to the training process, especially in scenarios where the global accumulated statistics need to be synchronized across clients efficiently. Addressing these limitations would require further research into optimizing the implementation of FIXBN, exploring adaptive strategies for selecting T ⋆, and streamlining the communication and synchronization of global statistics in federated learning environments.

Are there any other normalization techniques beyond BN and GN that could be more suitable for federated learning, and how can they be explored?

Exploring alternative normalization techniques beyond Batch Normalization (BN) and Group Normalization (GN) for federated learning could provide valuable insights into improving model performance in decentralized settings. One potential approach is Layer Normalization (LN), which normalizes activations along the features dimension independently for each sample. LN could be more suitable for federated learning as it does not rely on mini-batch statistics, potentially addressing the challenges posed by non-IID data distributions. Additionally, Instance Normalization (IN) and Weight Normalization (WN) are normalization techniques that could be explored for federated learning, offering different normalization strategies that may be beneficial in decentralized optimization frameworks. By investigating the applicability and performance of these alternative normalization methods in federated learning scenarios, researchers can broaden the understanding of normalization techniques in decentralized settings.