Enhancing Model Capacity and Fine-Grained Understanding with Multi-Head Mixture-of-Experts

Multi-Head Mixture-of-Experts (MH-MoE) employs a multi-head mechanism to split each input token into multiple sub-tokens, assign them to diverse experts in parallel, and seamlessly reintegrate them, enabling denser expert activation and finer-grained understanding.