Read the original document by opening this link in a new tab.
Table of Contents
Abstract
1 Introduction
1.1 Random Matrix Theory in Deep Learning
1.2 New Approach to Random Matrix Theory
1.3 The Tensor Programs Framework
1.4 Tensor Programs Master Theorems
Summary of Our Contributions
Summary
In this paper, the Free Independence Principle (FIP) is introduced, showing that (pre-)activations of a randomly initialized NN become independent from the weights as the NN's widths tend to infinity. The paper presents a Master Theorem for any Tensor Program and gives new proofs of the semicircle and Marchenko-Pastur laws. The application of FIP in computing the Jacobian singular value distribution of a randomly initialized NN is rigorously demonstrated. The paper also discusses the Tensor Programs framework, which unifies different lines of research in deep learning and provides a scalable approach to analyzing neural networks. Overall, the paper highlights the versatility and power of the Tensor Programs technique.