Register and CLS tokens induce a decoupling of local and global features in large ViTs

Recent work has shown that the attention maps of the widely popular DINOv2model exhibit artifacts, which hurt both model interpretability and performanceon dense image tasks.

These artifacts emerge due to the model repurposing patchtokens with redundant local information for the storage of global image information. To address this problem, additional register tokens have been incorporated in whichthe model can store such information instead.

We carefully examine the influenceof these register tokens on the relationship between global and local image features, showing that while register tokens yield cleaner attention maps, these maps donot accurately reflect the integration of local image information in large models. Instead, global information is dominated by information extracted from registertokens, leading to a disconnect between local and global features. Inspired by thesefindings, we show that the CLS token itself leads to a very similar phenomenonin models without explicit register tokens.

Our work shows that care must be takenwhen interpreting attention maps of large ViTs. Further, by clearly attributingthe faulty behavior to register and CLS tokens, we show a path towards moreinterpretable vision models.