Alias-Free ViT: Fractional Shift Invariance via Linear Attention

Transformers have emerged as a competitive alternative to convnets in vision tasks, yet they lack the architectural inductive bias of convnets, which may hinder their potential performance. Specifically, Vision Transformers (ViTs) are not translation-invariant and are more sensitive to minor image translations than standard convnets.

Previous studies have shown, however, that convnets are also not perfectly shift-invariant, due to aliasing in downsampling and nonlinear layers. Consequently, anti-aliasing approaches have been proposed to certify convnets translation robustness.

Building on this line of work, we propose an Alias-Free ViT, which combines two main components.

Our model maintains competitive performance in image classification and outperforms similar-sized models in terms of robustness to adversarial translations.