Read the original document by opening this link in a new tab.
Table of Contents
1. Abstract
2. Introduction
3. Large-scale Language Modeling
4. Problem Statement
5. Robust Language Modeling
6. Algorithm
Summary
Language models are generally trained on data spanning a wide range of topics but might be applied to an a priori unknown target distribution. In this paper, the authors propose an approach called topic conditional value at risk (topic CVaR) to improve robustness of language models. They demonstrate through theoretical analysis and experiments that their approach reduces perplexity and achieves improved robustness against subpopulation shifts.