Tensor Decomposition for Model Reduction in Neural Networks: A Review

By X. Liu et al.

Read the original document by opening this link in a new tab.

Table of Contents

I. Introduction
II. Tensor Decomposition Methods
A. Truncated Singular Value Decomposition
B. Tensor Train Decomposition
C. Canonical Polyadic Decomposition
D. Tucker Decomposition
E. Tensor Ring Decomposition
F. Block Term Decomposition
G. Hierarchical Tucker Decomposition
III. Tensorizing Convolutional Neural Networks

Summary

The document reviews tensor decomposition methods used for model reduction in neural networks, focusing on applications in CNNs, RNNs, and Transformers. It covers various decomposition techniques such as Truncated SVD, Tensor Train, CP Decomposition, Tucker Decomposition, Tensor Ring, Block Term, and Hierarchical Tucker. The methods are explained with examples and their application in reducing model parameters and improving efficiency in neural networks.