Week 2: The Transformer Architecture
Attention Is All You Need: The architecture that changed modern AI forever.
Overview
Before Transformers, NLP relied on RNNs and LSTMs, which processed data sequentially (slow) and struggled with long-range dependencies. The Transformer (2017) revolutionized this by introducing Self-Attention and Parallelization.
In this week, we dissect the architecture block by block: Positional Encoding, Multi-Head Attention, and Feed-Forward Networks.
Theory & Concepts
The Core Mechanism: Self-Attention
Self-attention allows the model to look at other words in the sentence to build a better representation of the current word. For example, in "The animal didn't cross the street because it was too tired", attention links "it" to "animal".
The Equation
- Query (Q): What we are looking for.
- Key (K): What the token offers.
- Value (V): The actual information content.
Positional Encoding
Since the Transformer has no recurrence, it has no inherent notion of order. We must inject position information.
Concept Check
Make sure you grasp the core equations before moving to code.
Interactive Lab: Multi-Head Attention
Visualize how different heads focus on different parts of the sentence. Click 'Simulate' to see new patterns.
Code Implementation
Core Logic
Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.
class MultiHeadAttention(nn.Module):
def __init__(self, d_model, num_heads):
super().__init__()
self.d_model = d_model
self.num_heads = num_heads
self.d_k = d_model // num_heads
self.W_q = nn.Linear(d_model, d_model)
self.W_k = nn.Linear(d_model, d_model)
self.W_v = nn.Linear(d_model, d_model)
self.W_o = nn.Linear(d_model, d_model)
def forward(self, x):
batch, seq_len, _ = x.shape
# 1. Linear Projections & Split Heads
Q = self.W_q(x).view(batch, seq_len, self.num_heads, self.d_k).transpose(1, 2)
K = self.W_k(x).view(batch, seq_len, self.num_heads, self.d_k).transpose(1, 2)
V = self.W_v(x).view(batch, seq_len, self.num_heads, self.d_k).transpose(1, 2)
# 2. Scaled Dot-Product Attention
scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_k)
attn = torch.softmax(scores, dim=-1)
output = torch.matmul(attn, V)
# 3. Concatenate & Final Linear
output = output.transpose(1, 2).contiguous().view(batch, seq_len, self.d_model)
return self.W_o(output)
Weekly Project
Build a Transformer Block
You will implement a reusable Transformer Encoder Layer containing Multi-Head Attention and a FeedForward Network.
Step-by-Step Implementation
- Implement
PositionalEncodingclass using sine/cosine functions. - Implement
MultiHeadAttentionwith correct tensor reshaping. - Build the
FeedForwardblock (Linear -> ReLU -> Linear). - Assemble them into a
TransformerBlockwith LayerNorm and Residual Connections.
exercises.py file for bonus tasks.