Core Concepts
This paper introduces the Multi-Agent Sparse Training (MAST) framework, which enables efficient training of deep multi-agent reinforcement learning models using ultra-sparse neural networks while maintaining high performance.
Abstract
The paper proposes the Multi-Agent Sparse Training (MAST) framework to address the computational challenges in training deep multi-agent reinforcement learning (MARL) models. Deep MARL relies on neural networks with numerous parameters, leading to substantial computational overhead, especially as the number of agents grows.
The key innovations in MAST are:
- Hybrid TD(λ) targets combined with the Soft Mellowmax operator to mitigate estimation errors arising from network sparsity and reduce overestimation bias.
- A dual replay buffer mechanism to enhance the distribution of training samples and reduce policy inconsistency errors due to sparsification.
- Gradient-based topology evolution to exclusively train multiple MARL agents using sparse networks.
The comprehensive experimental evaluation on the StarCraft Multi-Agent Challenge (SMAC) benchmark demonstrates that MAST can achieve model compression ranging from 5x to 20x with less than 3% performance degradation. Additionally, MAST reduces the Floating Point Operations (FLOPs) required for both training and inference by up to 20x, significantly outperforming other baseline methods.
Stats
The paper reports the following key metrics:
Model sparsity levels ranging from 85% to 95%
Model size reductions of 5x to 20x compared to dense models
FLOPs reduction of up to 20x for both training and inference
Quotes
"MAST introduces innovative solutions to enhance the accuracy of value learning in ultra-sparse models by concurrently refining training data targets and distributions."
"Extensive experiments validate MAST's effectiveness in sparse training, achieving model compression ratios of 5× to 20× with minimal performance degradation and up to a remarkable 20× reduction in FLOPs for both training and inference."