We introduce a computationally efficient kernel learning method that employs adversarial feature perturbations instead of input perturbations, achieving better optimization, adaptive regularization, and strong theoretical and empirical results.