Week 3: LLM Variants & Tokenization
Understanding the data pipeline and architectures of GPT-2, LLaMA, and more.
Overview
Large Language Models are not just "big transformers". They rely on sophisticated tokenization, positional embeddings (RoPE, ALiBi), and decoding strategies. This week focues on the input pipeline and the decoder-only architecture typical of GPT models.
Theory & Concepts
Byte-Pair Encoding (BPE)
Computers don't read words. They deal with numbers. Simple character-level selection is too verbose, and word-level is too sparse.
BPE starts with characters and iteratively merges the most frequent pairs. ('u', 'n') -> 'un'. Eventually ('un', 'related') -> 'unrelated'.
Decoder-Only Architecture (GPT)
Unlike BERT (Encoder), GPT is a specific type of Transformer that uses Masked Self-Attention. It cannot look at future tokens.
Concept Check
Make sure you grasp the core equations before moving to code.
Interactive Lab: Visual Tokenizer
See how text gets chopped into tokens. Notice how common words are single tokens, while rare words are split.
Code Implementation
Core Logic
Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.
# Generating text with a causal mask
def generate(model, context, max_new_tokens):
for _ in range(max_new_tokens):
# Crop context to context_window
context_cond = context[:, -block_size:]
# Get predictions
logits, _ = model(context_cond)
# Focus only on the last time step
logits = logits[:, -1, :]
# Apply softmax to get probabilities
probs = F.softmax(logits, dim=-1)
# Sample from the distribution
idx_next = torch.multinomial(probs, num_samples=1)
# Append to running sequence
context = torch.cat((context, idx_next), dim=1)
return context
Weekly Project
Mini-GPT
Load GPT-2 weights and implement a text generation loop with Temperature and Top-K sampling.
Step-by-Step Implementation
- Load
gpt2using HuggingFace Transformers. - Explore the
vocab.jsonto understand BPE. - Implement the
generate()function manually (nomodel.generate). - Add Temperature scaling to control randomness.
exercises.py file for bonus tasks.