Week 6: The U-Net

The backbone of modern generative image models.

Overview

The U-Net architecture, originally designed for medical segmentation, is perfect for diffusion because it preserves spatial size. It uses downsampling to capture context and upsampling to localize details, linked by Skip Connections.

Theory & Concepts

Architecture Shape

The "U" shape comes from:

  • Encoder (Down): Convolutions + MaxPool. Increases channels, decreases size.
  • Bottleneck: Deep semantic features.
  • Decoder (Up): Transpose Convolutions. Decreases channels, increases size.

Skip Connections

We concatenate feature maps from the encoder directly to the decoder. This allows gradients to flow easily and provides high-res details to the upsampling path.

Concept Check

Make sure you grasp the core equations before moving to code.

Interactive Lab: Skip Connections

Hover over the encoder blocks (Left) to see where the information flows in the decoder (Right).

💡 Experiment: Try different values to see how the system reacts in real-time.

Code Implementation

Core Logic

Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.

Python
class UNet(nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.down1 = DoubleConv(in_channels, 64)
        self.down2 = DoubleConv(64, 128)
        self.bot = DoubleConv(128, 256)
        self.up1 = Up(256, 128)
        self.up2 = Up(128, 64)
        self.out = nn.Conv2d(64, out_channels, 1)

    def forward(self, x):
        x1 = self.down1(x)
        x2 = self.down2(x1) # Maxpool inside
        bot = self.bot(x2)
        x = self.up1(bot, x2) # Skip connection x2
        x = self.up2(x, x1)   # Skip connection x1
        return self.out(x)

Weekly Project

Build a Simple U-Net

Construct a functional U-Net for noise prediction.

Step-by-Step Implementation

  • Implement the DoubleConv block.
  • Implement Down (Pool) and Up (Upsample + Concat) blocks.
  • Verify output shapes match input shapes.
  • (Bonus) Add Time Embeddings for Diffusion.
🚀 Challenge: Can you optimize this further? Check the exercises.py file for bonus tasks.