Week 7: Latent Diffusion
Stable Diffusion: Making it fast by working in compressed space.
Overview
Pixel-space diffusion is slow. Latent Diffusion Models (LDMs) use a Variational Autoencoder (VAE) to compress images such that a 512x512 image becomes a 64x64 latent tensor. We then diffuse this small tensor.
Theory & Concepts
Perceptual Compression
The VAE removes high-frequency details that aren't perceptually important, while keeping the semantic content.
Conditioning
How do we guide generation? We use Cross-Attention in the U-Net. Text embeddings (from CLIP) are injected into the U-Net layers to steer the denoising.
Concept Check
Make sure you grasp the core equations before moving to code.
Interactive Lab: Latent Compression
Slide to compress the image into Latent Space and back. Notice the loss of detail.
Original (Pixel Space)
Code Implementation
Core Logic
Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.
# Training Loop Sketch
# 1. Encode image to latent
z = vae.encode(image).latent_dist.sample() * 0.18215
# 2. Add noise
noise = torch.randn_like(z)
z_noisy = scheduler.add_noise(z, noise, t)
# 3. Predict noise (conditioned on text)
pred_noise = unet(z_noisy, t, encoder_hidden_states=prompts)
# 4. Loss
loss = F.mse_loss(pred_noise, noise)
Weekly Project
Latent Pipeline Mockup
Simulate the LDM pipeline using mock components.
Step-by-Step Implementation
- Create the `LatentDiffusion` class.
- Mock the VAE encoder/decoder.
- Mock the CLIP text encoder.
- Trace the tensor shapes through the entire pipeline.
exercises.py file for bonus tasks.