Efficient Recurrent Language Models with Joint Activity Sparsity and Connectivity Sparsity

Jointly applying activity sparsity and connectivity sparsity in recurrent neural networks can significantly reduce computational requirements without proportional drop in task performance for language modeling.