We propose Differential RL, a physics-informed framework that reformulates RL as a differential control problem. Its algorithm, dfPO, achieves pointwise convergence and outperforms standard RL in low-data scientific computing tasks.