We develop DISCOVER, which enables RL agents to solve substantially more challenging tasks than previous exploration strategies in RL.