A Survey of Optimization Methods from a Machine Learning Perspective

By Shiliang Sun et al

Published on Oct. 23, 2019

Read the original document by opening this link in a new tab.

Table of Contents

Abstract
I. Introduction
II. Machine Learning Formulated as Optimization
A. Optimization Problems in Supervised Learning
B. Optimization Problems in Semi-supervised Learning
C. Optimization Problems in Unsupervised Learning
D. Optimization Problems in Reinforcement Learning
E. Optimization for Machine Learning
III. Fundamental Optimization Methods and Progresses

Summary

Machine learning develops rapidly, with optimization being a crucial part of it. This paper provides a systematic overview of optimization methods in machine learning, discussing the challenges faced and the progress made. It covers various optimization categories such as first-order, high-order, and derivative-free methods. The importance of optimization in deep neural networks, reinforcement learning, variational inference, and other machine learning fields is emphasized. The paper also explores the impact of optimization on different machine learning applications, presenting challenges and open problems for further research.