Centrala begrepp
Modifying the backpropagation step to create a more balanced vector field for optimization can significantly improve the learning of complex physics-based tasks compared to the regular gradient.
Sammanfattning
The paper addresses the challenges of optimizing long unrolled sequences in training setups that combine neural networks and physics simulators. The key issue is the unbalanced gradient field resulting from repeatedly applying the same operator, leading to exploding and vanishing gradients.
The authors propose modifying the backpropagation step to create an alternative vector field for optimization. Specifically, they stop the feedback propagation through the neural network inputs while keeping the full temporal trajectory through the physics simulator. This results in a more balanced gradient flow, as physical systems typically have inherent constraints on the rate of change.
However, this modification also introduces rotation in the vector field, which can cause issues for optimization algorithms like Adam. To address this, the authors introduce a combined update that selectively uses the original gradient direction near minima while leveraging the balanced updates elsewhere.
The authors evaluate their method on three control tasks of increasing complexity: a guidance-by-repulsion model, a cart pole swing-up, and a quantum control problem. The results show that the proposed combined update consistently outperforms the regular gradient, especially as the tasks become more challenging. This demonstrates the benefits of the authors' approach in effectively learning complex physics-based behaviors.
Statistik
The cart pole system has 1 to 4 poles.
The quantum control task has target states of 2, 3, and 4.
Citat
"Of all the vector fields surrounding the minima of recurrent learning setups, the gradient field with its exploding and vanishing updates appears a poor choice for optimization, offering little beyond efficient computability."
"Backpropagating only along this path improves the situation of the gradient sizes as most physics simulators come with a well-behaved gradient flow; the key lies in separately considering the partial derivatives for x and c."
"Any modifications of the backpropagation pass decouple it from the corresponding forward pass and can destroy the rotation-free character of the gradient field, an unfamiliar situation in optimization."