The Reversal Curse: LLMs Trained on “A is B” Fail to Learn “B is A”

By Lukas Berglund et al

Published on Sept. 21, 2023

Read the original document by opening this link in a new tab.

Table of Contents

1. Abstract
2. Introduction
3. Contributions: Evidence for the Reversal Curse
4. Finetune on Synthetic Facts
5. Evaluate in Both Orders
6. Name to Description
7. Description to Name

Summary

The document discusses the Reversal Curse in large language models (LLMs) where models fail to generalize from 'A is B' to 'B is A'. It presents evidence through finetuning experiments on synthetic data and highlights the failure of logical deduction in LLMs. The Reversal Curse is shown to affect practical generalization in state-of-the-art models. Various attempts to mitigate the Reversal Curse are explored but prove unsuccessful.