We propose CARMANIA, a context-aware regularization method for nucleotide sequence analysis that enforces Markovian properties in Transformers, improving long-range dependencies, convergence speed, and out-of-domain generalization.