We propose Pseudo-Zeroth-Order (PseuZO), a framework that estimates Jacobian matrices via model output differentiation and applies EMA for variance reduction, with theoretical convergence guarantees and empirical validation.