We show that slightly increasing transformers' depth with the input length increases their expressive power under standard complexity conjectures.