
LoRAMoE introduces a novel framework to address the conflict between improving LLM performance on downstream tasks and preventing world knowledge forgetting during SFT.


coremsg

loramoe-alleviating-world-knowledge-forgetting-in-large-language-models-via-moe-style-plugin


LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin


title_rewrite


Large-scale instruction data can damage world knowledge in LLMs, LoRAMoE mitigates this issue while enhancing downstream task performance.


loramoe-addressing-world-knowledge-forgetting-in-large-language-models


LoRAMoE: Addressing World Knowledge Forgetting in Large Language Models



Large-scale increases in instruction data can lead to world knowledge forgetting in LLMs, but LoRAMoE mitigates this issue while enhancing multitasking abilities.
