FedPALS: A Novel Approach to Federated Learning That Overcomes Label Shift and Improves Generalization to Target Domains

Core Concepts

FedPALS, a novel model aggregation scheme for federated learning, effectively addresses label shift between client and target data distributions by leveraging knowledge of the target label distribution at the central server, leading to improved model generalization in label-shifted target domains.

Abstract

• Bibliographic Information: Listo Zec, E., Breitholtz, A., & Johansson, F. D. (2024). Overcoming label shift in targeted federated learning. arXiv preprint arXiv:2411.03799.
• Research Objective: This paper introduces FedPALS, a novel model aggregation method for federated learning, designed to overcome the challenges of label shift between client and target data distributions, particularly when the target label distribution is known.
• Methodology: The authors propose FedPALS, which optimizes a convex combination of client models by minimizing the distance between the weighted average of client label distributions and the target label distribution, while also considering the effective sample size to control variance. They theoretically analyze FedPALS, proving its unbiasedness under specific conditions and comparing it to standard federated averaging. The authors conduct extensive experiments on image classification benchmarks, including CIFAR-10, Fashion-MNIST, PACS, and iWildCam, to evaluate FedPALS against established baselines like FedAvg, FedProx, and SCAFFOLD.
• Key Findings: FedPALS consistently outperforms baseline methods in scenarios with significant label shift, demonstrating superior generalization to target domains. The experiments highlight that traditional federated learning methods, particularly FedAvg, struggle in the presence of label shift, especially with sparse client label distributions. The choice of the regularization parameter λ in FedPALS, which balances bias and variance, significantly impacts performance, with lower values generally favoring better alignment with the target distribution.
• Main Conclusions: FedPALS offers a principled and practical solution for mitigating the adverse effects of label distribution mismatch in federated learning. By leveraging knowledge of the target label distribution, FedPALS enables the training of models that generalize effectively to label-shifted target domains, even when no target data is directly available during training.
• Significance: This research significantly contributes to the field of federated learning by addressing the crucial challenge of label shift, a common issue in real-world applications. FedPALS paves the way for more robust and reliable federated learning models that can be deployed in diverse domains with varying data distributions.
• Limitations and Future Research: The authors acknowledge that the optimal choice of the regularization parameter λ in FedPALS can be task-dependent and suggest exploring adaptive strategies for dynamically tuning it during training. Future research could extend FedPALS to address covariate shift, where input distributions differ between clients and the target, potentially by incorporating techniques from unsupervised domain adaptation.

Stats

FedPALS achieves 92.4% mean accuracy on Fashion-MNIST with C=3, compared to 67.1% for FedAvg.
On CIFAR-10 with C=3, FedPALS attains 65.6% mean accuracy, surpassing FedAvg's 44.0%.
In the PACS dataset, FedPALS reaches 86.0% mean accuracy, outperforming FedAvg's 73.4%.

Quotes

"The justification for FedAvg is weaker when clients exhibit systematic data heterogeneity, such as in the case of label shift [Zhao et al., 2018, Woodworth et al., 2020], since the learning objectives of clients differ from the objective optimized by the central server."
"This work aims to improve the generalization of federated learning to target domains under label shift, in settings where the different label distributions of clients and target domains are known to the central server but unknown to the clients."
"Our approach is both well-justified and practical. We prove that the resulting stochastic gradient update behaves, in expectation, as centralized learning in the target domain (Proposition 1), and examine its relation to standard federated averaging (Proposition 3.2)."

Key Insights Distilled From

Overcoming label shift in targeted federated learning

by Edvin Listo ... at arxiv.org 11-07-2024

https://arxiv.org/pdf/2411.03799.pdf