The Power of Scale for Parameter-Efficient Prompt Tuning

By Brian Lester et al

Published on Sept. 2, 2021

Read the original document by opening this link in a new tab.

Table of Contents

Abstract
1 Introduction
2 Prompt Tuning
2.1 Design Decisions
2.2 Unlearning Span Corruption
3 Results
3.1 Closing the Gap

Summary

In this work, the authors explore 'prompt tuning,' a mechanism for learning soft prompts to condition frozen language models for downstream tasks. The method outperforms few-shot learning and matches model tuning performance as model size increases. The approach is cost-effective and allows reuse of frozen models for multiple tasks. The paper compares prompt tuning with other approaches like prefix tuning and prompt design, showing competitive results. Key contributions include demonstrating the competitiveness of prompt tuning with model tuning, showing improved quality and robustness with scale, and proposing prompt ensembling. The paper presents detailed experiments and results, highlighting the importance of language model capacity in the success of prompt tuning.