We propose CATransformer, the first carbon-aware co-optimization framework for Transformer-based models and hardware accelerators.