Qiang Huang

Full Professor, Harbin Institute of Technology

1 paper at NeurIPS 2025

Homepage· OpenReview· Semantic Scholar· Google Scholar

Poster Session 1

1 paper

Wednesday, December 3, 2025 · 11:00 AM → 2:00 PM

Exhibit Hall C,D,E

A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone

#3719 Spotlight · Jitai Hao, Qiang Huang, Hao Liu, Xinyan Xiao, Zhaochun Ren, Jun Yu

We "clone" large LLMs into small SLMs by training only low-rank projection matrices for weights and making all student activations identical to the teacher's. This yields comparable SLM performance with 1000x fewer training tokens.