Week 9: LoRA (Low-Rank Adaptation)
Fine-tuning massive models on consumer GPUs.
Overview
Full fine-tuning updates all weights. LoRA freezes the main weights and trains only tiny adapter matrices A and B. W' = W + BA.
Theory & Concepts
Rank Decomposition
If W is 1000x1000, it has 1M params. If we use rank r=4, A is 4x1000 and B is 1000x4. Total = 8000 params. That's a 99% reduction!
Concept Check
Make sure you grasp the core equations before moving to code.
Interactive Lab: Matrix Decomposition
Adjust the Rank 'r' to see how the adapter matrices change size relative to the fixed weight W.
W (Fixed)
+
B
x
A
Rank r = 10
💡 Experiment: Try different values to see how the system reacts in real-time.
Code Implementation
Core Logic
Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.
Python
class LoRALinear(nn.Module):
def __init__(self, in_ft, out_ft, r=4):
super().__init__()
self.linear = nn.Linear(in_ft, out_ft) # Frozen
self.lora_A = nn.Parameter(torch.randn(r, in_ft))
self.lora_B = nn.Parameter(torch.zeros(out_ft, r))
def forward(self, x):
return self.linear(x) + (x @ self.lora_A.T @ self.lora_B.T)
Weekly Project
Implement LoRA
Create a wrapper that converts Linear layers to LoRA layers.
Step-by-Step Implementation
- Build
LoRALinear. - Replace layers in a dummy model.
- Verify only A and B have grads.
🚀 Challenge: Can you optimize this further? Check the
exercises.py file for bonus tasks.