Week 6: The U-Net
The backbone of modern generative image models.
Overview
The U-Net architecture, originally designed for medical segmentation, is perfect for diffusion because it preserves spatial size. It uses downsampling to capture context and upsampling to localize details, linked by Skip Connections.
Theory & Concepts
Architecture Shape
The "U" shape comes from:
- Encoder (Down): Convolutions + MaxPool. Increases channels, decreases size.
- Bottleneck: Deep semantic features.
- Decoder (Up): Transpose Convolutions. Decreases channels, increases size.
Skip Connections
We concatenate feature maps from the encoder directly to the decoder. This allows gradients to flow easily and provides high-res details to the upsampling path.
Concept Check
Make sure you grasp the core equations before moving to code.
Interactive Lab: Skip Connections
Hover over the encoder blocks (Left) to see where the information flows in the decoder (Right).
Code Implementation
Core Logic
Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.
class UNet(nn.Module):
def __init__(self, in_channels, out_channels):
super().__init__()
self.down1 = DoubleConv(in_channels, 64)
self.down2 = DoubleConv(64, 128)
self.bot = DoubleConv(128, 256)
self.up1 = Up(256, 128)
self.up2 = Up(128, 64)
self.out = nn.Conv2d(64, out_channels, 1)
def forward(self, x):
x1 = self.down1(x)
x2 = self.down2(x1) # Maxpool inside
bot = self.bot(x2)
x = self.up1(bot, x2) # Skip connection x2
x = self.up2(x, x1) # Skip connection x1
return self.out(x)
Weekly Project
Build a Simple U-Net
Construct a functional U-Net for noise prediction.
Step-by-Step Implementation
- Implement the
DoubleConvblock. - Implement
Down(Pool) andUp(Upsample + Concat) blocks. - Verify output shapes match input shapes.
- (Bonus) Add Time Embeddings for Diffusion.
exercises.py file for bonus tasks.