We introduce a computationally efficient method for visual policy learning using differentiable simulation.