Week 2: The Transformer Architecture

Attention Is All You Need: The architecture that changed modern AI forever.

Overview

Before Transformers, NLP relied on RNNs and LSTMs, which processed data sequentially (slow) and struggled with long-range dependencies. The Transformer (2017) revolutionized this by introducing Self-Attention and Parallelization.

In this week, we dissect the architecture block by block: Positional Encoding, Multi-Head Attention, and Feed-Forward Networks.

Theory & Concepts

The Core Mechanism: Self-Attention

Self-attention allows the model to look at other words in the sentence to build a better representation of the current word. For example, in "The animal didn't cross the street because it was too tired", attention links "it" to "animal".

The Equation

Attention(Q, K, V) = softmax( (QK^T) / √d_k ) V
  • Query (Q): What we are looking for.
  • Key (K): What the token offers.
  • Value (V): The actual information content.

Positional Encoding

Since the Transformer has no recurrence, it has no inherent notion of order. We must inject position information.

PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Concept Check

Make sure you grasp the core equations before moving to code.

Interactive Lab: Multi-Head Attention

Visualize how different heads focus on different parts of the sentence. Click 'Simulate' to see new patterns.


💡 Experiment: Try different values to see how the system reacts in real-time.

Code Implementation

Core Logic

Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.

Python
class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads):
        super().__init__()
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads
        
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)
        
    def forward(self, x):
        batch, seq_len, _ = x.shape
        
        # 1. Linear Projections & Split Heads
        Q = self.W_q(x).view(batch, seq_len, self.num_heads, self.d_k).transpose(1, 2)
        K = self.W_k(x).view(batch, seq_len, self.num_heads, self.d_k).transpose(1, 2)
        V = self.W_v(x).view(batch, seq_len, self.num_heads, self.d_k).transpose(1, 2)
        
        # 2. Scaled Dot-Product Attention
        scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_k)
        attn = torch.softmax(scores, dim=-1)
        output = torch.matmul(attn, V)
        
        # 3. Concatenate & Final Linear
        output = output.transpose(1, 2).contiguous().view(batch, seq_len, self.d_model)
        return self.W_o(output)

Weekly Project

Build a Transformer Block

You will implement a reusable Transformer Encoder Layer containing Multi-Head Attention and a FeedForward Network.

Step-by-Step Implementation

  • Implement PositionalEncoding class using sine/cosine functions.
  • Implement MultiHeadAttention with correct tensor reshaping.
  • Build the FeedForward block (Linear -> ReLU -> Linear).
  • Assemble them into a TransformerBlock with LayerNorm and Residual Connections.
🚀 Challenge: Can you optimize this further? Check the exercises.py file for bonus tasks.