We construct generalized Lyapunov certificates by augmenting value functions with residual terms to certify the stability of control policies from optimal control or reinforcement learning.