A Geometric Approach to Analyzing Deep Neural Networks with Non-Differentiable Activation Functions

This paper extends a geometric framework for studying deep neural networks to handle piecewise differentiable activation functions like ReLU, convolutional layers, residual blocks, and recurrent networks. It introduces random walk-based algorithms to explore the equivalence classes of the input and representation manifolds, which can have varying dimensions due to the non-differentiability.