We propose a data removal method using factor decorrelation and loss perturbation to boost model robustness and accuracy, achieving strong performance on five benchmarks even under significant distribution shifts.