未被選中的專家也能做出貢獻：通過自我對比釋放 MoE 模型的潛力
本文認為，混合專家模型 (MoE) 中未被選中的專家可以通過自我對比的方式，對模型的輸出做出積極貢獻，從而提升模型的推理能力。
Unleashing the Power of Mixture-of-Experts Models Through Self-Contrast: A Study on Utilizing Unchosen Experts for Improved Performance
Leveraging the contrasting outputs from different routing strategies within a Mixture-of-Experts (MoE) model, specifically contrasting strong activation (e.g., top-2 routing) with weak activation (e.g., rank-k routing), can effectively utilize unchosen experts and significantly enhance the model's reasoning capabilities without substantial computational overhead.
CartesianMoE：透過專家混合模型中的卡氏積路由，促進專家之間的知識共享
CartesianMoE 透過引入卡氏積層和卡氏積路由，實現更有效的專家間知識共享，進而提升語言模型的效能和路由穩健性。
CartesianMoE: Enhancing Knowledge Sharing in Mixture-of-Experts Models for Large Language Models Using Cartesian Product Routing
CartesianMoE, a novel approach for large language models (LLMs), leverages the Cartesian product of sub-expert sets to enhance knowledge sharing among experts in Mixture-of-Experts (MoE) models, leading to improved performance and routing robustness.
AdaMOE：透過空專家實現 MoE 語言模型的token自適應路由
AdaMOE透過引入空專家和自適應路由策略，在保持模型性能的同時，顯著降低了MoE模型的計算成本，提升了模型效率。
널 전문가를 사용한 토큰 적응형 라우팅: Mixture-of-Experts 언어 모델을 위한 AdaMOE
AdaMOE는 MoE 모델에서 각 토큰에 필요한 전문가 수를 자유롭게 조절하여 계산 효율성을 높이고 성능을 향상시키는 새로운 토큰 적응형 라우팅 방법입니다.
Upcycling Pre-trained Dense Language Models to Mixture-of-Experts (MoE) Models for Improved Performance and Efficiency
Upcycling pre-trained dense language models into Mixture-of-Experts (MoE) models is a more efficient method for increasing model capacity and achieving better accuracy compared to continued dense model training, especially with techniques like "virtual group" initialization and weight scaling.
MoE++: Enhancing Mixture-of-Experts Efficiency and Effectiveness with Zero-Computation Experts (Work in Progress)
MoE++, a novel Mixture-of-Experts (MoE) framework, integrates zero-computation experts and gating residuals to significantly improve both the efficiency and effectiveness of large language models.
Dynamic Mixture of Experts (DYNMOE): Auto-Tuning Experts and Their Activation in Transformer Models for Enhanced Efficiency
DYNMOE, a novel training technique for Mixture-of-Experts (MoE) models, eliminates the need for manual hyperparameter tuning by automatically determining the optimal number of experts and the number of activated experts per token during training, leading to enhanced efficiency and competitive performance across various machine learning tasks.
© 2024 by Linnk AI