The paper presents a case study to reveal a critical vulnerability in knowledge distillation (KD)-based federated learning (FL) techniques. It shows that while these techniques effectively improve performance under high data heterogeneity, they inadvertently cause higher accuracy degradation under model poisoning attacks, a phenomenon termed "attack amplification".
The authors first provide empirical evidence and theoretical reasoning to explain why KD-based techniques like FedNTD and MOON amplify the impact of model poisoning attacks. They show that the very mechanisms that improve performance in benign conditions also make the models more vulnerable to adversarial attacks.
To address this issue, the authors propose Hybrid Knowledge Distillation for Robust and Accurate FL (HYDRA-FL), a novel technique that applies KD-loss at both the final layer and a shallow layer of the client model via an auxiliary classifier. This hybrid approach reduces the impact of poisoning on the client model by preventing over-reliance on final layer alignment.
The authors adapt HYDRA-FL to FedNTD and MOON, and their extensive experiments across three datasets show that HYDRA-FL significantly boosts accuracy over the baselines in attack settings while maintaining performance in benign settings.
To Another Language
from source content
arxiv.org
Key Insights Distilled From
by Momin Ahmad ... at arxiv.org 10-01-2024
https://arxiv.org/pdf/2409.19912.pdfDeeper Inquiries