The content discusses an efficient and flexible method for reducing the size of moderate-size deep neural networks using the concept of condensation. Key points:
Neural networks have been extensively applied in scientific fields, but their scale is generally moderate to ensure fast inference during application. Reducing the size of neural networks is important for enabling efficient deployment in resource-constrained environments.
Theoretical work has shown that under strong nonlinearity, neurons in the same layer of a neural network tend to exhibit a "condensation" phenomenon, where their parameter vectors align. This suggests the presence of redundant neurons that can be merged.
The authors propose a condensation reduction algorithm that can be applied to both fully connected networks and convolutional networks. The method involves identifying and merging neurons that have condensed, creating a smaller subnetwork with similar performance.
Experiments on combustion simulation and CIFAR10 image classification tasks demonstrate the effectiveness of the condensation reduction method. In the combustion task, the neural network size was reduced to 41.7% of the original while maintaining prediction accuracy. In CIFAR10, the network size was reduced to 11.5% of the original with only a slight drop in classification accuracy.
The condensation reduction method is shown to be efficient and broadly applicable, making it a promising approach for reducing the size of neural networks in scientific and resource-constrained applications.
To Another Language
from source content
arxiv.org
Key Insights Distilled From
by Tianyi Chen,... at arxiv.org 05-03-2024
https://arxiv.org/pdf/2405.01041.pdfDeeper Inquiries