When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories

By Alex Mallen et al.

Published on July 2, 2023

Read the original document by opening this link in a new tab.

Table of Contents

1 Introduction
2 Related Work
3 Evaluation Setup
4 Memorization Depends on Popularity and Relationship Type

Summary

Large language models (LMs) have shown impressive performance on various NLP tasks, but struggle with tasks requiring rich world knowledge. This paper investigates LMs' strengths and limitations in memorizing factual knowledge. The authors conduct large-scale knowledge probing experiments using two open-domain entity-centric QA datasets. They find that LMs struggle with less popular factual knowledge and propose a new method for retrieval augmentation to improve performance. The study evaluates different LMs with varying sizes and analyzes how subject entity popularity and relationship types influence memorization. Scaling up models mainly improves memorization of popular knowledge, while non-parametric memories help with less popular entities. The paper provides insights into when to trust LMs' outputs and the potential of retrieval-augmented LMs for future research.