Core Concepts
The authors propose an iterative pruning strategy with an importance-score metric to deactivate unimportant connections in deep neural networks, achieving significant parameter compression while maintaining comparable accuracy. Their approach aims to simplify networks by finding the smallest number of connections necessary for a given task.
Abstract
The study introduces a novel pruning algorithm based on neural activity to reduce overparameterization in deep neural networks. By deactivating unimportant connections, the algorithm achieves substantial parameter compression while preserving accuracy. The research focuses on simplifying network architectures by identifying and removing unnecessary connections, leading to more efficient models with reduced computational complexity.
Key points include:
- Proposal of an iterative pruning strategy with an importance-score metric.
- Aim to find the smallest number of connections needed for a task.
- Achieving significant parameter compression without sacrificing accuracy.
- Focus on simplifying network architectures and reducing computational complexity.
Stats
We achieve comparable performance for LeNet architectures on MNIST.
Significantly higher parameter compression than state-of-the-art algorithms for VGG and ResNet architectures.
Compressions of more than 50 times for VGG architectures on CIFAR-10 and Tiny-ImageNet datasets.