Limitations of Autoregressive Models and Their Alternatives

By Chu-Cheng Lin et al

Read the original document by opening this link in a new tab.

Table of Contents

1 Introduction
2 Background
3 Autoregressive ECCP models (ELNCP models) have reduced capacity
3.1 ELN and ELNCP models
3.2 ELNCP models cannot exactly capture all EC (or ECCP) distributions
3.3 ELNCP models cannot even capture all EC (or ECCP) supports or rankings

Summary

This document discusses the limitations of autoregressive models in language processing and explores alternative models such as energy-based models and latent-variable autoregressive models. It highlights the challenges faced by autoregressive models in computing probabilities efficiently and the need for alternative approaches to overcome these limitations. The paper delves into the complexities of sequence modeling in NLP and the implications of different model families on computational efficiency and expressive power.