We investigate the role of learning rate grafting and the staleness of the preconditioner in Shampoo by decoupling the updates of the eigenvalues and eigenbasis of its preconditioner.

We propose a novel step size strategy for the extragradient method to solve root-finding and min-max optimization problems under relaxed Lipschitz assumptions

We study the expression and limitations of transformers in performing in-context learning of functions governed by Markovian dynamics.

We provide sharp convergence analysis on sketched adaptive distributed deep learning that only depends on the intrinsic dimension of loss Hessian, instead of the full dimensionality..

We propose a novel adversarial purification technique based on optimization that regards the attack landscape.

We prove divergence results for gradient flows for deep neural networks with analytic activation and polynomial target functions.

A quasi-second order optimizer with first-order efficiency by using Kalman Filter

This work introduces learn2mix, a new training strategy that adaptively adjusts class proportions in batches to accelerate neural network convergence in resource-constrained environments.

A novel suite benchmark problems for data-driven engineering design (beams, airfoils, heat conduction, etc.) under a standardized API.

This work proposes a new framework to solve parametric optimization problems with guarantees on feasibility and convergence.

We develop an optimizer ASGO that can provably exploit the low-rank gradients and block-wise diagonal Hessians in training.

We observe that Adam’s performance in training transformers degrades differently under different types of random rotations of the objective function, highlighting the need for new, basis-dependent theory to fully understand Adam’s success.

We propose CATransformer, the first carbon-aware co-optimization framework for Transformer-based models and hardware accelerators.

We introduce dQP, a solver-agnostic, modular framework for differentiable quadratic programming (QP), enabling plug-and-play differentiation with any solver and demonstrating advantages in large-scale sparse problems.

We prove that ALS with overparameterized rank $k=r^2$ achieves global convergence for decomposing rank $r$ tensors

We introduce a bicycle design benchmark that evaluates multiphysics performance, constraint satisfaction, and adherence to human preferences, and we benchmark LLMs, tabular generative models, and design optimization algorithms side by side.

DCcluster-Opt introduces a high-fidelity benchmark for sustainable workload scheduling in geo-distributed data centers, integrating real-world data and physics-based models to evaluate multi-objective optimization strategies via a Gym environment.

We introduce 3BASiL-TM, a highly efficient one-shot post-training method for Sparse plus Low-Rank decomposition of LLMs that reduces the WikiText2 perplexity gap to dense model by over $30\%$ compared to prior methods.

MLPs contain "channels to infinity" where pairs of neurons evolve to form gated linear units with diverging output weights, creating regions that appear like flat minima but actually have slowly decreasing loss value

we theoretically study when and how unlabeled data can help in multi-objective learning

We design a mean-field explicit differentiation method for incentive design in large-scale games with exchangeable agents.

We show that adaptive optimizers like RMSProp lead to fairer minima than SGD, both theoretically and empirically.

We introduce the Fair Minimum Labeling problem for designing temporally efficient and fair activation plans, prove tight hardness bounds, and present approximation algorithms with strong empirical results on fair multi-source learning.

Our proposed orthogonality and variance losses improve performance in downstream fine-tuning of Mixture-of-Experts models by enhancing expert specificity, addressing expert homogenization caused by load balancing, while maintaining load balance.

We solve noisy linear inverse problems 10-50x faster than existing diffusion methods and with exact recovery of measurement constraints

Adam with equal betas works well, and its form can be simplified for further insights

We introduce ALE-bench, a new benchmark for evaluating AI systems on score-based algorithmic programming contests.

We propose a data removal method using factor decorrelation and loss perturbation to boost model robustness and accuracy, achieving strong performance on five benchmarks even under significant distribution shifts.

Model-free optimization methods typically use only current cost samples (e.g., one per iteration) by discarding all the past cost samples. We introduce a simple yet memory mechanism to maintain and use them until new data become available.

We propose the first deep menu-based method for single-bidder combinatorial auctions that is dominant-strategy incentive compatible and revenue-optimizing, by adapting continuous flow matching for optimization problems.

We present an algorithm for test-time scaling of SDE-based diffusion models by searching for noise trajectories which optimize arbitrary rewards, empirically matching/exceeding MCTS performance.

On a linear bigram model, we show that the heavy-tailed distribution found in text requires training for a number of iteration that scales linearly with the vocabulary size with gradient descent but only a square-root scaling with sign descent.

We prove that the gradient descent map is non-singular for any neural network using piecewise analytic activation functions.

RefLoRA improves upon LoRA by dynamically selecting the optimal low-rank factorization per step. This leads to faster and stabler convergence, as well as superior performance on various NLP tasks, while incurring negligible computational overhead.

We prove that the interaction of parameter symmetry and equivariance constraints can create critical points and minima in the loss landscape.

We derive well-known learning rules from an objective that casts learning rules as policies for navigating uncertain loss landscapes

We show that neural operators/neural emulators/neural simulators can outperform the numerical method that produced their training data.

We introduce Adaptive Constrained Equivariance, a homotopy-inspired constrained optimization apprach for training equivariant neural networks.

We propoed the first application of RLVR to directly enhance LLMs' proficiency in optimization modeling: SI-RL, achieves state-of-the-art on diverse public benchmarks

Nonlinear systems whose future behavior is not overly sensitive to small perturbations can be efficiently parallelized; whereas unpredictable dynamical systems cannot be efficiently parallelized.

LC-Opt is a high-fidelity RL benchmarking environment for optimizing liquid cooling in HPC systems, enabling scalable, explainable, and energy-efficient control with RL and LLMs, while promoting optimal data center solutions.