The Boundary of Neural Network Trainability is Fractal

By Jascha Sohl-Dickstein et al

Read the original document by opening this link in a new tab.

Table of Contents

I. INTRODUCTION
II. EXPERIMENTS
A. Network and data
B. Training
C. Visualization and analysis
D. Experimental conditions
III. DISCUSSION
A. Elaborate functions in high dimensional spaces
B. Non-homogeneity of boundary
C. Stochastic training
D. Higher dimensional fractals
E. Meta-loss landscapes are difficult to navigate
F. Fractals are beautiful and relaxing
ACKNOWLEDGMENTS
REFERENCES

Summary

This paper explores the boundary between neural network hyperparameters that result in stable and divergent training. It visualizes the fractal nature of this boundary across various experimental conditions, including different network nonlinearities, training methods, and hyperparameter configurations. The study reveals the intricate relationship between fractal generation and neural network training, highlighting the sensitivity of training outcomes to hyperparameter variations. The experiments demonstrate that the boundary between trainable and untrainable neural network hyperparameters exhibits fractal properties, offering insights into the complex dynamics of neural network optimization.