Contrastive self-supervised learning can provably recover the underlying true transition dynamics, enabling efficient exploration in reinforcement learning.