Week 3: LLM Variants & Tokenization

Understanding the data pipeline and architectures of GPT-2, LLaMA, and more.

Overview

Large Language Models are not just "big transformers". They rely on sophisticated tokenization, positional embeddings (RoPE, ALiBi), and decoding strategies. This week focues on the input pipeline and the decoder-only architecture typical of GPT models.

Theory & Concepts

Byte-Pair Encoding (BPE)

Computers don't read words. They deal with numbers. Simple character-level selection is too verbose, and word-level is too sparse.

BPE starts with characters and iteratively merges the most frequent pairs. ('u', 'n') -> 'un'. Eventually ('un', 'related') -> 'unrelated'.

Decoder-Only Architecture (GPT)

Unlike BERT (Encoder), GPT is a specific type of Transformer that uses Masked Self-Attention. It cannot look at future tokens.

mask = torch.tril(torch.ones(seq_len, seq_len))

Concept Check

Make sure you grasp the core equations before moving to code.

Interactive Lab: Visual Tokenizer

See how text gets chopped into tokens. Notice how common words are single tokens, while rare words are split.

💡 Experiment: Try different values to see how the system reacts in real-time.

Code Implementation

Core Logic

Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.

Python
# Generating text with a causal mask
def generate(model, context, max_new_tokens):
    for _ in range(max_new_tokens):
        # Crop context to context_window
        context_cond = context[:, -block_size:]
        
        # Get predictions
        logits, _ = model(context_cond)
        
        # Focus only on the last time step
        logits = logits[:, -1, :]
        
        # Apply softmax to get probabilities
        probs = F.softmax(logits, dim=-1)
        
        # Sample from the distribution
        idx_next = torch.multinomial(probs, num_samples=1)
        
        # Append to running sequence
        context = torch.cat((context, idx_next), dim=1)
    return context

Weekly Project

Mini-GPT

Load GPT-2 weights and implement a text generation loop with Temperature and Top-K sampling.

Step-by-Step Implementation

  • Load gpt2 using HuggingFace Transformers.
  • Explore the vocab.json to understand BPE.
  • Implement the generate() function manually (no model.generate).
  • Add Temperature scaling to control randomness.
🚀 Challenge: Can you optimize this further? Check the exercises.py file for bonus tasks.