John Kirchenbauer

We fit scaling laws for large language models with varying width-to-depth ratios and parameter counts.

We show that recurrent-depth transformers can be scaled to be effective language models, with particularly strong gains through additional compute for reasoning tasks.

We collect an 8 TB of public domain and openly licensed text and use it to pre-train a performant 7B-parameter LLM.