preconditioner

2 papers across 1 session

Poster Session 1

Wednesday, December 3, 2025 · 11:00 AM → 2:00 PM

Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner

#903 Spotlight · Runa Eschenhagen, Aaron Defazio, Tsung-Hsien Lee, Richard Turner, Hao-Jun Shi

We investigate the role of learning rate grafting and the staleness of the preconditioner in Shampoo by decoupling the updates of the eigenvalues and eigenbasis of its preconditioner.

PaZO: Preconditioned Accelerated Zeroth-Order Optimization for Fine-Tuning LLMs

#705 · Hanzhen Zhao, Ding Shihong, Cong Fang, Zhouchen Lin