Week 9: LoRA (Low-Rank Adaptation)

Fine-tuning massive models on consumer GPUs.

Overview

Full fine-tuning updates all weights. LoRA freezes the main weights and trains only tiny adapter matrices A and B. W' = W + BA.

Theory & Concepts

Rank Decomposition

If W is 1000x1000, it has 1M params. If we use rank r=4, A is 4x1000 and B is 1000x4. Total = 8000 params. That's a 99% reduction!

Concept Check

Make sure you grasp the core equations before moving to code.

Interactive Lab: Matrix Decomposition

Adjust the Rank 'r' to see how the adapter matrices change size relative to the fixed weight W.

W (Fixed)
+
B
x
A

Rank r = 10

💡 Experiment: Try different values to see how the system reacts in real-time.

Code Implementation

Core Logic

Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.

Python
class LoRALinear(nn.Module):
    def __init__(self, in_ft, out_ft, r=4):
        super().__init__()
        self.linear = nn.Linear(in_ft, out_ft) # Frozen
        self.lora_A = nn.Parameter(torch.randn(r, in_ft))
        self.lora_B = nn.Parameter(torch.zeros(out_ft, r))
        
    def forward(self, x):
        return self.linear(x) + (x @ self.lora_A.T @ self.lora_B.T)

Weekly Project

Implement LoRA

Create a wrapper that converts Linear layers to LoRA layers.

Step-by-Step Implementation

  • Build LoRALinear.
  • Replace layers in a dummy model.
  • Verify only A and B have grads.
🚀 Challenge: Can you optimize this further? Check the exercises.py file for bonus tasks.