Core Concepts
FedFisher algorithm leverages Fisher information matrices for efficient one-shot federated learning.
Abstract
The content introduces the FedFisher algorithm for one-shot federated learning, addressing drawbacks of standard FL algorithms. It discusses theoretical analysis, practical implementations using diagonal Fisher and K-FAC approximations, and extensive experiments showcasing improved performance over competing baselines.
-
Introduction
- Decentralized data collection and storage drive the need for Federated Learning (FL).
- Standard FL algorithms require multiple rounds of communication, leading to various drawbacks.
-
Proposed Algorithm: FedFisher
- Utilizes Fisher information matrices from local client models for one-shot global model training.
- Theoretical analysis shows error reduction with wider neural networks and increased local training.
-
Theoretical Analysis for Two-layer Over-parameterized Neural Network
- Characterizes sources of error in FedFisher and demonstrates error control with wider models.
-
A Practical Implementation of FedFisher
- Discusses computation efficiency of diagonal Fisher and K-FAC approximations.
- Highlights communication efficiency and compatibility with secure aggregation.
-
Experiments
- Evaluates FedFisher against state-of-the-art baselines across different datasets.
- Shows consistent improvement in performance, especially in multi-round settings and using pre-trained models.
-
Conclusion
- Proposes future work on extending the analysis to deeper neural networks and enhancing privacy guarantees.
Stats
"Extensive experiments on various datasets show consistent improvement over competing baselines."
"FedFisher variants consistently outperform other baselines across varying heterogeneity parameters."
Quotes
"Our contribution lies in showing that for a sufficiently wide model, this distance decreases as O(1/m) where m is the width of the model."
"FedFisher variants offer additional utility in multi-round settings and continue to improve over baselines."