Read the original document by opening this link in a new tab.
Table of Contents
Abstract
1 Introduction
1.1 Shampoo for matrices
1.2 Related work
2 Background and technical tools
2.1 Online convex optimization
2.2 Adaptive regularization in online optimization
2.3 Kronecker products
2.4 Matrix inequalities
3 Analysis of Shampoo for matrices
Summary
Shampoo: Preconditioned Stochastic Tensor Optimization is a paper that introduces a new structure-aware preconditioning algorithm called Shampoo for stochastic optimization over tensor spaces. The algorithm maintains a set of preconditioning matrices, each operating on a single dimension and contracting over the remaining dimensions. The paper discusses the motivation behind Shampoo, its implementation in Python, and its performance in comparison to other optimizers. Shampoo is shown to converge faster than commonly used optimizers while maintaining a comparable runtime per step. The analysis of Shampoo for matrices provides insights into its effectiveness in online convex optimization scenarios.