Tensor Programs II: Neural Tangent Kernel for Any Architecture

By Greg Yang et al.

Published on Nov. 30, 2020

Read the original document by opening this link in a new tab.

Table of Contents

Abstract
1 Introduction
2 Background
3 Related Works
4 Warmup: Neural Tangent Kernel for a Multi-Layer Perceptron
5 NTK for Any Architecture? The Issues and the Proposal
6 Strategy for Computing the Infinite-Width NTK

Summary

We prove that a randomly initialized neural network of any architecture has its Tangent Kernel (NTK) converge to a deterministic limit, as the network widths tend to infinity. In this paper, we show that the NTK for any randomly initialized neural network of standard architecture converges almost surely to a deterministic limit. The results here imply a universal Neural Network-Tangent Kernel correspondence.