Learning a Markov abstract state representation is crucial for improving deep reinforcement learning efficiency.