We extend DeltaNet by using products of householders as state-transition matrices allowing us to trade-off expressivity and computational complexity.

We present a unified theory for the study of RNN expressivity, with novel results on several popular architectures, and insights on the relationship between linear and non-linear RNNs.