Symmetry breaking is crucial for improving end-to-end learning performance in phase retrieval problems.