We study the expression and limitations of transformers in performing in-context learning of functions governed by Markovian dynamics.