We identify the connection between the output of the encoder and the ensuing dense layers as the main underlying factor limiting scaling capabilities in deep RL