
The authors propose FeDeRA, a method that leverages Singular Value Decomposition to initialize the adapter modules in the LoRA technique, in order to improve the performance of parameter-efficient fine-tuning in federated learning settings with highly non-IID data.


coremsg

efficient-fine-tuning-of-language-models-in-federated-learning-using-weight-decomposition


Efficient Fine-tuning of Language Models in Federated Learning Using Weight Decomposition
