Long LoRA: Efficient Fine-Tuning of Long-Context Large Language Models

By Yukang Chen et al

Published on March 8, 2024

Read the original document by opening this link in a new tab.

Table of Contents

ABSTRACT
1 INTRODUCTION
2 RELATED WORK
3 LONG LORA
3.1 BACKGROUND
3.2 SHIFTED SPARSE ATTENTION
3.3 IMPROVED LORA FOR LONG CONTEXT

Summary

Long LoRA presents an efficient fine-tuning approach that extends the context sizes of pre-trained large language models (LLMs) with limited computation cost. The paper introduces Shifted Sparse Attention (S2-Attn) as an efficient substitute for standard self-attention to enable context extension. The proposed method demonstrates strong empirical results on various tasks with Llama2 models. It combines improved LoRA with S2-Attn to achieve efficient fine-tuning and context extension. The work addresses the computational challenges of training LLMs with long context sizes, providing an effective and efficient solution for researchers.