The paper introduces a new bi-level projection method that can efficiently enforce structured sparsity, particularly the ℓ1,∞ norm, in neural networks. The key idea is to split the projection into two simpler steps: first aggregating the columns using the q-norm, then projecting the aggregated vector onto the p-norm ball.
The authors show that this bi-level approach has a time complexity of O(nm) for a matrix in Rn×m, compared to O(nm log(nm)) for the best existing ℓ1,∞ projection algorithm. They also generalize the bi-level approach to a multi-level projection, which can achieve an exponential parallel speedup.
Experiments show that the bi-level ℓ1,∞ projection is 2.5 times faster than the state-of-the-art method while providing the same accuracy and better sparsity in neural network applications. The authors also demonstrate the application of their bi-level and multi-level projections to other structured sparsity norms like ℓ1,1 and ℓ1,2.
To Another Language
from source content
arxiv.org
Thông tin chi tiết chính được chắt lọc từ
by Guillaume Pe... lúc arxiv.org 05-06-2024
https://arxiv.org/pdf/2405.02086.pdfYêu cầu sâu hơn